Re: accumulo upgrade from 2.0.1 to 2.1.1
On Sat, Oct 14, 2023 at 9:41 AM Vincent Russell wrote: > > Christopher, > > I don't remember configuring anything with regards to commons-vfs. How do > I disable this? There's some VFS stuff enabled by default, for backwards compatibility. There isn't an explicit way to disable it. It's being phased out, so that it's entirely optional, via a pluggable mechanism, if somebody wants to use it. But I've never seen the default stuff cause any errors like what you're seeing if you're not using it to specify any non-local paths in your classpath properties. > > After continuously hitting refresh on either monitor they just continuously > say the manager is down. > > I"ll dig some more on Monday. > > Thanks for the response. > > -Vincent > > On Fri, Oct 13, 2023 at 5:39 PM Christopher wrote: > > > I am not sure what's up with the VFS error. I generally recommend > > against using commons-vfs, because of the lack of any known good > > versions, and the fact that most of what it is used for can be done > > better in other ways. We have taken steps in recent versions to make > > sure that the classloader stuff is more pluggable, so people do not > > have to rely on commons-vfs, if they don't want to use it. One thing > > that's concerning is that VFS is using some temp file in /tmp, which > > could be wiped at any time by the OS, so if you do have need of VFS, > > you may want to consider customizing it to use a different directory, > > that isn't going to be wiped periodically by the OS. I believe the > > relevant Accumulo property is: general.vfs.cache.dir > > > > The manager is down message is expected on initial startup if the > > monitor starts faster than the manager, but refreshing the page should > > make it work eventually if the manager is actually running. If that > > doesn't happen eventually, I'm not sure what could be happening > > without further troubleshooting. > > > > On Thu, Oct 12, 2023 at 12:36 PM Vincent Russell > > wrote: > > > > > > Christopher, > > > > > > I have run through an upgrade with a bit more data than I did previously > > > and monitors with high availability just don't seem as consistent as they > > > were with 2.0.1. > > > > > > After upgrading both accumulo monitors are reporting "Manager is Down" > > and > > > I was only able to achieve that after restarting both of the monitors and > > > managers multiple times. > > > > > > The only ERROR log that I see in the manager logs are (at startup): > > > > > > java.nio.File.NoSuchFileException: /tmp/accumulo-vfs-cache-410@hostname > > > > > > at > > > > > org.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.close(AccumuloVFSClassloader.java:453) > > > > > > I see this on both managers. > > > > > > Any ideas? > > > > > > Thanks, > > > > > > On Mon, Sep 25, 2023 at 12:21 PM Christopher > > wrote: > > > > > > > I'm not sure about the monitor behavior. I've never had a reason to > > > > run more than one, and I don't think we have any well-defined behavior > > > > for trying to run more than one (I could see an argument for both). It > > > > would be weird if one incorrectly reported "Master[now Manager] is > > > > down" because another monitor was running, though. I would expect the > > > > 2.1 behavior you described over that 2.0 behavior. > > > > > > > > On Mon, Sep 25, 2023 at 11:59 AM Vincent Russell > > > > wrote: > > > > > > > > > > Thank you Christopher, > > > > > > > > > > I did neglect to update the log4j2 files. > > > > > > > > > > One thing that is weird is that the monitors appear to work > > differently > > > > > with high availability. > > > > > > > > > > With 2.0.1 the accumulo monitors both work and you can hit them both > > (one > > > > > just says 'Master is down'; however, with 2.1.1, it appears that > > only one > > > > > works.Is this expected? > > > > > > > > > > Thank you, > > > > > > > > > > On Thu, Sep 14, 2023 at 4:28 PM Christopher > > wrote: > > > > > > > > > > > I do not know about the error message you saw. > > StatusConsoleListener > > > > > > is not an Accumulo class, but it looks like it's a log4j one. My > > best > > > > > > guess is that during your upgrade, you did not migrate your log4j > > > > > > config files over to be based on the log4j2 ones we have as an > > example > > > > > > in the 2.1 tarball. It's possible that there's some log4j > > > > > > configuration issue that's preventing you from seeing the reason > > why > > > > > > the Monitor failed to start. > > > > > > > > > > > > To troubleshoot, I would compare your log4j config with what we > > ship > > > > > > in the 2.1 tarball and also check to see if you've done any log4j > > > > > > config overrides in the system properties or environment variables > > > > > > that control log4j. > > > > > > > > > > > > On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell > > > > > > wrote: > > > > > > > > > > > > > > Thank you Christopher, > > > > > > > > > > > > > > Don't worry. I definitely planned on testing an upgrade on a > >
Re: accumulo upgrade from 2.0.1 to 2.1.1
Christopher, I don't remember configuring anything with regards to commons-vfs. How do I disable this? After continuously hitting refresh on either monitor they just continuously say the manager is down. I"ll dig some more on Monday. Thanks for the response. -Vincent On Fri, Oct 13, 2023 at 5:39 PM Christopher wrote: > I am not sure what's up with the VFS error. I generally recommend > against using commons-vfs, because of the lack of any known good > versions, and the fact that most of what it is used for can be done > better in other ways. We have taken steps in recent versions to make > sure that the classloader stuff is more pluggable, so people do not > have to rely on commons-vfs, if they don't want to use it. One thing > that's concerning is that VFS is using some temp file in /tmp, which > could be wiped at any time by the OS, so if you do have need of VFS, > you may want to consider customizing it to use a different directory, > that isn't going to be wiped periodically by the OS. I believe the > relevant Accumulo property is: general.vfs.cache.dir > > The manager is down message is expected on initial startup if the > monitor starts faster than the manager, but refreshing the page should > make it work eventually if the manager is actually running. If that > doesn't happen eventually, I'm not sure what could be happening > without further troubleshooting. > > On Thu, Oct 12, 2023 at 12:36 PM Vincent Russell > wrote: > > > > Christopher, > > > > I have run through an upgrade with a bit more data than I did previously > > and monitors with high availability just don't seem as consistent as they > > were with 2.0.1. > > > > After upgrading both accumulo monitors are reporting "Manager is Down" > and > > I was only able to achieve that after restarting both of the monitors and > > managers multiple times. > > > > The only ERROR log that I see in the manager logs are (at startup): > > > > java.nio.File.NoSuchFileException: /tmp/accumulo-vfs-cache-410@hostname > > > > at > > > org.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.close(AccumuloVFSClassloader.java:453) > > > > I see this on both managers. > > > > Any ideas? > > > > Thanks, > > > > On Mon, Sep 25, 2023 at 12:21 PM Christopher > wrote: > > > > > I'm not sure about the monitor behavior. I've never had a reason to > > > run more than one, and I don't think we have any well-defined behavior > > > for trying to run more than one (I could see an argument for both). It > > > would be weird if one incorrectly reported "Master[now Manager] is > > > down" because another monitor was running, though. I would expect the > > > 2.1 behavior you described over that 2.0 behavior. > > > > > > On Mon, Sep 25, 2023 at 11:59 AM Vincent Russell > > > wrote: > > > > > > > > Thank you Christopher, > > > > > > > > I did neglect to update the log4j2 files. > > > > > > > > One thing that is weird is that the monitors appear to work > differently > > > > with high availability. > > > > > > > > With 2.0.1 the accumulo monitors both work and you can hit them both > (one > > > > just says 'Master is down'; however, with 2.1.1, it appears that > only one > > > > works.Is this expected? > > > > > > > > Thank you, > > > > > > > > On Thu, Sep 14, 2023 at 4:28 PM Christopher > wrote: > > > > > > > > > I do not know about the error message you saw. > StatusConsoleListener > > > > > is not an Accumulo class, but it looks like it's a log4j one. My > best > > > > > guess is that during your upgrade, you did not migrate your log4j > > > > > config files over to be based on the log4j2 ones we have as an > example > > > > > in the 2.1 tarball. It's possible that there's some log4j > > > > > configuration issue that's preventing you from seeing the reason > why > > > > > the Monitor failed to start. > > > > > > > > > > To troubleshoot, I would compare your log4j config with what we > ship > > > > > in the 2.1 tarball and also check to see if you've done any log4j > > > > > config overrides in the system properties or environment variables > > > > > that control log4j. > > > > > > > > > > On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell > > > > > wrote: > > > > > > > > > > > > Thank you Christopher, > > > > > > > > > > > > Don't worry. I definitely planned on testing an upgrade on a > test > > > > > > cluster. :) > > > > > > > > > > > > I just upgraded a test cluster and everything appeared to go > smoothly > > > > > > except for one thing. > > > > > > > > > > > > I am able to login to the accumulo shell and I can see the table > > > that I > > > > > > created before the upgrade exists and I can scan that table. > > > > > > > > > > > > Before the upgrade the monitor was running on port 9995, but now > > > nothing > > > > > > appears to be running on that port.Nothing in the monitor log > > > > > exception: > > > > > > > > > > > > ERROR StatusConsoleListener Unable to send HTTP in appender > > > [MonitorLog] > > > > > > IllegalARgumentException:
Re: accumulo upgrade from 2.0.1 to 2.1.1
I am not sure what's up with the VFS error. I generally recommend against using commons-vfs, because of the lack of any known good versions, and the fact that most of what it is used for can be done better in other ways. We have taken steps in recent versions to make sure that the classloader stuff is more pluggable, so people do not have to rely on commons-vfs, if they don't want to use it. One thing that's concerning is that VFS is using some temp file in /tmp, which could be wiped at any time by the OS, so if you do have need of VFS, you may want to consider customizing it to use a different directory, that isn't going to be wiped periodically by the OS. I believe the relevant Accumulo property is: general.vfs.cache.dir The manager is down message is expected on initial startup if the monitor starts faster than the manager, but refreshing the page should make it work eventually if the manager is actually running. If that doesn't happen eventually, I'm not sure what could be happening without further troubleshooting. On Thu, Oct 12, 2023 at 12:36 PM Vincent Russell wrote: > > Christopher, > > I have run through an upgrade with a bit more data than I did previously > and monitors with high availability just don't seem as consistent as they > were with 2.0.1. > > After upgrading both accumulo monitors are reporting "Manager is Down" and > I was only able to achieve that after restarting both of the monitors and > managers multiple times. > > The only ERROR log that I see in the manager logs are (at startup): > > java.nio.File.NoSuchFileException: /tmp/accumulo-vfs-cache-410@hostname > > at > org.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.close(AccumuloVFSClassloader.java:453) > > I see this on both managers. > > Any ideas? > > Thanks, > > On Mon, Sep 25, 2023 at 12:21 PM Christopher wrote: > > > I'm not sure about the monitor behavior. I've never had a reason to > > run more than one, and I don't think we have any well-defined behavior > > for trying to run more than one (I could see an argument for both). It > > would be weird if one incorrectly reported "Master[now Manager] is > > down" because another monitor was running, though. I would expect the > > 2.1 behavior you described over that 2.0 behavior. > > > > On Mon, Sep 25, 2023 at 11:59 AM Vincent Russell > > wrote: > > > > > > Thank you Christopher, > > > > > > I did neglect to update the log4j2 files. > > > > > > One thing that is weird is that the monitors appear to work differently > > > with high availability. > > > > > > With 2.0.1 the accumulo monitors both work and you can hit them both (one > > > just says 'Master is down'; however, with 2.1.1, it appears that only one > > > works.Is this expected? > > > > > > Thank you, > > > > > > On Thu, Sep 14, 2023 at 4:28 PM Christopher wrote: > > > > > > > I do not know about the error message you saw. StatusConsoleListener > > > > is not an Accumulo class, but it looks like it's a log4j one. My best > > > > guess is that during your upgrade, you did not migrate your log4j > > > > config files over to be based on the log4j2 ones we have as an example > > > > in the 2.1 tarball. It's possible that there's some log4j > > > > configuration issue that's preventing you from seeing the reason why > > > > the Monitor failed to start. > > > > > > > > To troubleshoot, I would compare your log4j config with what we ship > > > > in the 2.1 tarball and also check to see if you've done any log4j > > > > config overrides in the system properties or environment variables > > > > that control log4j. > > > > > > > > On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell > > > > wrote: > > > > > > > > > > Thank you Christopher, > > > > > > > > > > Don't worry. I definitely planned on testing an upgrade on a test > > > > > cluster. :) > > > > > > > > > > I just upgraded a test cluster and everything appeared to go smoothly > > > > > except for one thing. > > > > > > > > > > I am able to login to the accumulo shell and I can see the table > > that I > > > > > created before the upgrade exists and I can scan that table. > > > > > > > > > > Before the upgrade the monitor was running on port 9995, but now > > nothing > > > > > appears to be running on that port.Nothing in the monitor log > > > > exception: > > > > > > > > > > ERROR StatusConsoleListener Unable to send HTTP in appender > > [MonitorLog] > > > > > IllegalARgumentException: invalid URI schema <> > > > > > > > > > > > > > > > This seems not great, but I am not sure if it's related. > > > > > > > > > > It doesn't look like the monitor is running on another port. Any > > > > > suggestions for debugging this? > > > > > > > > > > Thanks, > > > > > Vincent > > > > > > > > > > > > > > > > > > > > On Wed, Aug 30, 2023 at 7:09 PM Christopher > > wrote: > > > > > > > > > > > My understanding is that the upgrade of ZK is pretty easy... but I > > > > > > would consult the ZooKeeper community for advice on that, since > > they > > > > > > are
Re: accumulo upgrade from 2.0.1 to 2.1.1
Christopher, I have run through an upgrade with a bit more data than I did previously and monitors with high availability just don't seem as consistent as they were with 2.0.1. After upgrading both accumulo monitors are reporting "Manager is Down" and I was only able to achieve that after restarting both of the monitors and managers multiple times. The only ERROR log that I see in the manager logs are (at startup): java.nio.File.NoSuchFileException: /tmp/accumulo-vfs-cache-410@hostname at org.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.close(AccumuloVFSClassloader.java:453) I see this on both managers. Any ideas? Thanks, On Mon, Sep 25, 2023 at 12:21 PM Christopher wrote: > I'm not sure about the monitor behavior. I've never had a reason to > run more than one, and I don't think we have any well-defined behavior > for trying to run more than one (I could see an argument for both). It > would be weird if one incorrectly reported "Master[now Manager] is > down" because another monitor was running, though. I would expect the > 2.1 behavior you described over that 2.0 behavior. > > On Mon, Sep 25, 2023 at 11:59 AM Vincent Russell > wrote: > > > > Thank you Christopher, > > > > I did neglect to update the log4j2 files. > > > > One thing that is weird is that the monitors appear to work differently > > with high availability. > > > > With 2.0.1 the accumulo monitors both work and you can hit them both (one > > just says 'Master is down'; however, with 2.1.1, it appears that only one > > works.Is this expected? > > > > Thank you, > > > > On Thu, Sep 14, 2023 at 4:28 PM Christopher wrote: > > > > > I do not know about the error message you saw. StatusConsoleListener > > > is not an Accumulo class, but it looks like it's a log4j one. My best > > > guess is that during your upgrade, you did not migrate your log4j > > > config files over to be based on the log4j2 ones we have as an example > > > in the 2.1 tarball. It's possible that there's some log4j > > > configuration issue that's preventing you from seeing the reason why > > > the Monitor failed to start. > > > > > > To troubleshoot, I would compare your log4j config with what we ship > > > in the 2.1 tarball and also check to see if you've done any log4j > > > config overrides in the system properties or environment variables > > > that control log4j. > > > > > > On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell > > > wrote: > > > > > > > > Thank you Christopher, > > > > > > > > Don't worry. I definitely planned on testing an upgrade on a test > > > > cluster. :) > > > > > > > > I just upgraded a test cluster and everything appeared to go smoothly > > > > except for one thing. > > > > > > > > I am able to login to the accumulo shell and I can see the table > that I > > > > created before the upgrade exists and I can scan that table. > > > > > > > > Before the upgrade the monitor was running on port 9995, but now > nothing > > > > appears to be running on that port.Nothing in the monitor log > > > exception: > > > > > > > > ERROR StatusConsoleListener Unable to send HTTP in appender > [MonitorLog] > > > > IllegalARgumentException: invalid URI schema <> > > > > > > > > > > > > This seems not great, but I am not sure if it's related. > > > > > > > > It doesn't look like the monitor is running on another port. Any > > > > suggestions for debugging this? > > > > > > > > Thanks, > > > > Vincent > > > > > > > > > > > > > > > > On Wed, Aug 30, 2023 at 7:09 PM Christopher > wrote: > > > > > > > > > My understanding is that the upgrade of ZK is pretty easy... but I > > > > > would consult the ZooKeeper community for advice on that, since > they > > > > > are the experts. For Accumulo's part, I believe 2.0 worked fine > with > > > > > newer versions of ZK (3.5+), so you should be able to update ZK > first > > > > > if you didn't want to do them at the same time. There may be some > > > > > things you might need to do to ensure your classpath is set up > > > > > correctly for Accumulo if using it with 2.0, since 2.0 might > assume ZK > > > > > 3.4 jar locations. I don't recall the specifics, but I remember it > > > > > wasn't that bad. fluo-uno had used newer ZK versions with 2.0 for a > > > > > while. The necessary config changes to support that might even > still > > > > > be scripted in its repo. > > > > > > > > > > I'm really not familiar with Hadoop upgrades, but as far as > Accumulo > > > > > is concerned, it would probably treat both 3.3.1 and 3.3.5 as > > > > > essentially equivalent. I don't think anything substantially > changed > > > > > that would cause Accumulo to even notice a difference. Seems like > it > > > > > would just be a quick shutdown, update the classpath, restart, for > > > > > Accumulo. For Hadoop, you may have some upgrade tasks... I'd check > > > > > with the Hadoop community first. You could probably upgrade Hadoop > > > > > before or after you upgrade Accumulo for this one. I don't think > it's > > > >
Re: accumulo upgrade from 2.0.1 to 2.1.1
I'm not sure about the monitor behavior. I've never had a reason to run more than one, and I don't think we have any well-defined behavior for trying to run more than one (I could see an argument for both). It would be weird if one incorrectly reported "Master[now Manager] is down" because another monitor was running, though. I would expect the 2.1 behavior you described over that 2.0 behavior. On Mon, Sep 25, 2023 at 11:59 AM Vincent Russell wrote: > > Thank you Christopher, > > I did neglect to update the log4j2 files. > > One thing that is weird is that the monitors appear to work differently > with high availability. > > With 2.0.1 the accumulo monitors both work and you can hit them both (one > just says 'Master is down'; however, with 2.1.1, it appears that only one > works.Is this expected? > > Thank you, > > On Thu, Sep 14, 2023 at 4:28 PM Christopher wrote: > > > I do not know about the error message you saw. StatusConsoleListener > > is not an Accumulo class, but it looks like it's a log4j one. My best > > guess is that during your upgrade, you did not migrate your log4j > > config files over to be based on the log4j2 ones we have as an example > > in the 2.1 tarball. It's possible that there's some log4j > > configuration issue that's preventing you from seeing the reason why > > the Monitor failed to start. > > > > To troubleshoot, I would compare your log4j config with what we ship > > in the 2.1 tarball and also check to see if you've done any log4j > > config overrides in the system properties or environment variables > > that control log4j. > > > > On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell > > wrote: > > > > > > Thank you Christopher, > > > > > > Don't worry. I definitely planned on testing an upgrade on a test > > > cluster. :) > > > > > > I just upgraded a test cluster and everything appeared to go smoothly > > > except for one thing. > > > > > > I am able to login to the accumulo shell and I can see the table that I > > > created before the upgrade exists and I can scan that table. > > > > > > Before the upgrade the monitor was running on port 9995, but now nothing > > > appears to be running on that port.Nothing in the monitor log > > exception: > > > > > > ERROR StatusConsoleListener Unable to send HTTP in appender [MonitorLog] > > > IllegalARgumentException: invalid URI schema <> > > > > > > > > > This seems not great, but I am not sure if it's related. > > > > > > It doesn't look like the monitor is running on another port. Any > > > suggestions for debugging this? > > > > > > Thanks, > > > Vincent > > > > > > > > > > > > On Wed, Aug 30, 2023 at 7:09 PM Christopher wrote: > > > > > > > My understanding is that the upgrade of ZK is pretty easy... but I > > > > would consult the ZooKeeper community for advice on that, since they > > > > are the experts. For Accumulo's part, I believe 2.0 worked fine with > > > > newer versions of ZK (3.5+), so you should be able to update ZK first > > > > if you didn't want to do them at the same time. There may be some > > > > things you might need to do to ensure your classpath is set up > > > > correctly for Accumulo if using it with 2.0, since 2.0 might assume ZK > > > > 3.4 jar locations. I don't recall the specifics, but I remember it > > > > wasn't that bad. fluo-uno had used newer ZK versions with 2.0 for a > > > > while. The necessary config changes to support that might even still > > > > be scripted in its repo. > > > > > > > > I'm really not familiar with Hadoop upgrades, but as far as Accumulo > > > > is concerned, it would probably treat both 3.3.1 and 3.3.5 as > > > > essentially equivalent. I don't think anything substantially changed > > > > that would cause Accumulo to even notice a difference. Seems like it > > > > would just be a quick shutdown, update the classpath, restart, for > > > > Accumulo. For Hadoop, you may have some upgrade tasks... I'd check > > > > with the Hadoop community first. You could probably upgrade Hadoop > > > > before or after you upgrade Accumulo for this one. I don't think it's > > > > going to matter much. > > > > > > > > If you're concerned about problems you might encounter, I would run > > > > through the upgrades on a small test instance first, just to practice. > > > > If you run into a specific problem doing that, we could probably offer > > > > more specific insight, having probably seen it before. > > > > > > > > > > > > On Wed, Aug 30, 2023 at 4:50 PM Vincent Russell > > > > wrote: > > > > > > > > > > Thank you Christopher. > > > > > > > > > > Does there exist some documentation for upgrading zookeeper from > > 3.4.14 > > > > to > > > > > 3.8.1? Is there a preferred upgrade path? > > > > > > > > > > Also...Is there any documentation on how this affects hadoop when > > using > > > > > high availability? We are using hadoop 3.3.1. I want to upgrade to > > > > > hadoop 3.3.5, but I didn't know if that should be done as part of > > > > upgrading > > > > > accumulo or
Re: accumulo upgrade from 2.0.1 to 2.1.1
Thank you Christopher, I did neglect to update the log4j2 files. One thing that is weird is that the monitors appear to work differently with high availability. With 2.0.1 the accumulo monitors both work and you can hit them both (one just says 'Master is down'; however, with 2.1.1, it appears that only one works.Is this expected? Thank you, On Thu, Sep 14, 2023 at 4:28 PM Christopher wrote: > I do not know about the error message you saw. StatusConsoleListener > is not an Accumulo class, but it looks like it's a log4j one. My best > guess is that during your upgrade, you did not migrate your log4j > config files over to be based on the log4j2 ones we have as an example > in the 2.1 tarball. It's possible that there's some log4j > configuration issue that's preventing you from seeing the reason why > the Monitor failed to start. > > To troubleshoot, I would compare your log4j config with what we ship > in the 2.1 tarball and also check to see if you've done any log4j > config overrides in the system properties or environment variables > that control log4j. > > On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell > wrote: > > > > Thank you Christopher, > > > > Don't worry. I definitely planned on testing an upgrade on a test > > cluster. :) > > > > I just upgraded a test cluster and everything appeared to go smoothly > > except for one thing. > > > > I am able to login to the accumulo shell and I can see the table that I > > created before the upgrade exists and I can scan that table. > > > > Before the upgrade the monitor was running on port 9995, but now nothing > > appears to be running on that port.Nothing in the monitor log > exception: > > > > ERROR StatusConsoleListener Unable to send HTTP in appender [MonitorLog] > > IllegalARgumentException: invalid URI schema <> > > > > > > This seems not great, but I am not sure if it's related. > > > > It doesn't look like the monitor is running on another port. Any > > suggestions for debugging this? > > > > Thanks, > > Vincent > > > > > > > > On Wed, Aug 30, 2023 at 7:09 PM Christopher wrote: > > > > > My understanding is that the upgrade of ZK is pretty easy... but I > > > would consult the ZooKeeper community for advice on that, since they > > > are the experts. For Accumulo's part, I believe 2.0 worked fine with > > > newer versions of ZK (3.5+), so you should be able to update ZK first > > > if you didn't want to do them at the same time. There may be some > > > things you might need to do to ensure your classpath is set up > > > correctly for Accumulo if using it with 2.0, since 2.0 might assume ZK > > > 3.4 jar locations. I don't recall the specifics, but I remember it > > > wasn't that bad. fluo-uno had used newer ZK versions with 2.0 for a > > > while. The necessary config changes to support that might even still > > > be scripted in its repo. > > > > > > I'm really not familiar with Hadoop upgrades, but as far as Accumulo > > > is concerned, it would probably treat both 3.3.1 and 3.3.5 as > > > essentially equivalent. I don't think anything substantially changed > > > that would cause Accumulo to even notice a difference. Seems like it > > > would just be a quick shutdown, update the classpath, restart, for > > > Accumulo. For Hadoop, you may have some upgrade tasks... I'd check > > > with the Hadoop community first. You could probably upgrade Hadoop > > > before or after you upgrade Accumulo for this one. I don't think it's > > > going to matter much. > > > > > > If you're concerned about problems you might encounter, I would run > > > through the upgrades on a small test instance first, just to practice. > > > If you run into a specific problem doing that, we could probably offer > > > more specific insight, having probably seen it before. > > > > > > > > > On Wed, Aug 30, 2023 at 4:50 PM Vincent Russell > > > wrote: > > > > > > > > Thank you Christopher. > > > > > > > > Does there exist some documentation for upgrading zookeeper from > 3.4.14 > > > to > > > > 3.8.1? Is there a preferred upgrade path? > > > > > > > > Also...Is there any documentation on how this affects hadoop when > using > > > > high availability? We are using hadoop 3.3.1. I want to upgrade to > > > > hadoop 3.3.5, but I didn't know if that should be done as part of > > > upgrading > > > > accumulo or later. > > > > > > > > Any experience folks have would be greatly appreciated. > > > > > > > > Thank you, > > > > Vincent > > > > > > > > On Thu, Aug 24, 2023 at 12:58 PM Christopher > > > wrote: > > > > > > > > > The correct link seems to be > > > > > > > > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/zookeeper#zookeeper-acls > > > > > > > > > > It looks like both `bin/accumulo dump-zoo` and `bin/accumulo-util > > > > > dump-zoo` will do the same thing. It was originally a disconnected > > > > > utility that existed for convenience, but was incorporated into our > > > > > regular tooling recently as part of an effort to make all these >
Re: accumulo upgrade from 2.0.1 to 2.1.1
I do not know about the error message you saw. StatusConsoleListener is not an Accumulo class, but it looks like it's a log4j one. My best guess is that during your upgrade, you did not migrate your log4j config files over to be based on the log4j2 ones we have as an example in the 2.1 tarball. It's possible that there's some log4j configuration issue that's preventing you from seeing the reason why the Monitor failed to start. To troubleshoot, I would compare your log4j config with what we ship in the 2.1 tarball and also check to see if you've done any log4j config overrides in the system properties or environment variables that control log4j. On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell wrote: > > Thank you Christopher, > > Don't worry. I definitely planned on testing an upgrade on a test > cluster. :) > > I just upgraded a test cluster and everything appeared to go smoothly > except for one thing. > > I am able to login to the accumulo shell and I can see the table that I > created before the upgrade exists and I can scan that table. > > Before the upgrade the monitor was running on port 9995, but now nothing > appears to be running on that port.Nothing in the monitor log exception: > > ERROR StatusConsoleListener Unable to send HTTP in appender [MonitorLog] > IllegalARgumentException: invalid URI schema <> > > > This seems not great, but I am not sure if it's related. > > It doesn't look like the monitor is running on another port. Any > suggestions for debugging this? > > Thanks, > Vincent > > > > On Wed, Aug 30, 2023 at 7:09 PM Christopher wrote: > > > My understanding is that the upgrade of ZK is pretty easy... but I > > would consult the ZooKeeper community for advice on that, since they > > are the experts. For Accumulo's part, I believe 2.0 worked fine with > > newer versions of ZK (3.5+), so you should be able to update ZK first > > if you didn't want to do them at the same time. There may be some > > things you might need to do to ensure your classpath is set up > > correctly for Accumulo if using it with 2.0, since 2.0 might assume ZK > > 3.4 jar locations. I don't recall the specifics, but I remember it > > wasn't that bad. fluo-uno had used newer ZK versions with 2.0 for a > > while. The necessary config changes to support that might even still > > be scripted in its repo. > > > > I'm really not familiar with Hadoop upgrades, but as far as Accumulo > > is concerned, it would probably treat both 3.3.1 and 3.3.5 as > > essentially equivalent. I don't think anything substantially changed > > that would cause Accumulo to even notice a difference. Seems like it > > would just be a quick shutdown, update the classpath, restart, for > > Accumulo. For Hadoop, you may have some upgrade tasks... I'd check > > with the Hadoop community first. You could probably upgrade Hadoop > > before or after you upgrade Accumulo for this one. I don't think it's > > going to matter much. > > > > If you're concerned about problems you might encounter, I would run > > through the upgrades on a small test instance first, just to practice. > > If you run into a specific problem doing that, we could probably offer > > more specific insight, having probably seen it before. > > > > > > On Wed, Aug 30, 2023 at 4:50 PM Vincent Russell > > wrote: > > > > > > Thank you Christopher. > > > > > > Does there exist some documentation for upgrading zookeeper from 3.4.14 > > to > > > 3.8.1? Is there a preferred upgrade path? > > > > > > Also...Is there any documentation on how this affects hadoop when using > > > high availability? We are using hadoop 3.3.1. I want to upgrade to > > > hadoop 3.3.5, but I didn't know if that should be done as part of > > upgrading > > > accumulo or later. > > > > > > Any experience folks have would be greatly appreciated. > > > > > > Thank you, > > > Vincent > > > > > > On Thu, Aug 24, 2023 at 12:58 PM Christopher > > wrote: > > > > > > > The correct link seems to be > > > > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/zookeeper#zookeeper-acls > > > > > > > > It looks like both `bin/accumulo dump-zoo` and `bin/accumulo-util > > > > dump-zoo` will do the same thing. It was originally a disconnected > > > > utility that existed for convenience, but was incorporated into our > > > > regular tooling recently as part of an effort to make all these > > > > utilities more easily discoverable and executable. I don't think we > > > > realized that it already had a convenient and documented entry point. > > > > We were just looking for utilities with a "main" method. Either should > > > > work fine. They call the same code. > > > > > > > > On Thu, Aug 24, 2023 at 11:04 AM Vincent Russell > > > > wrote: > > > > > > > > > > Also the link to: > > > > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/ZooKeeper#ACLs > > > > > > > > > > is not resolving. > > > > > > > > > > What needs to be done at this step? > > > > > > > > > > > > > > > Thanks, > > > > >
Re: accumulo upgrade from 2.0.1 to 2.1.1
Thank you Christopher, Don't worry. I definitely planned on testing an upgrade on a test cluster. :) I just upgraded a test cluster and everything appeared to go smoothly except for one thing. I am able to login to the accumulo shell and I can see the table that I created before the upgrade exists and I can scan that table. Before the upgrade the monitor was running on port 9995, but now nothing appears to be running on that port.Nothing in the monitor log exception: ERROR StatusConsoleListener Unable to send HTTP in appender [MonitorLog] IllegalARgumentException: invalid URI schema <> This seems not great, but I am not sure if it's related. It doesn't look like the monitor is running on another port. Any suggestions for debugging this? Thanks, Vincent On Wed, Aug 30, 2023 at 7:09 PM Christopher wrote: > My understanding is that the upgrade of ZK is pretty easy... but I > would consult the ZooKeeper community for advice on that, since they > are the experts. For Accumulo's part, I believe 2.0 worked fine with > newer versions of ZK (3.5+), so you should be able to update ZK first > if you didn't want to do them at the same time. There may be some > things you might need to do to ensure your classpath is set up > correctly for Accumulo if using it with 2.0, since 2.0 might assume ZK > 3.4 jar locations. I don't recall the specifics, but I remember it > wasn't that bad. fluo-uno had used newer ZK versions with 2.0 for a > while. The necessary config changes to support that might even still > be scripted in its repo. > > I'm really not familiar with Hadoop upgrades, but as far as Accumulo > is concerned, it would probably treat both 3.3.1 and 3.3.5 as > essentially equivalent. I don't think anything substantially changed > that would cause Accumulo to even notice a difference. Seems like it > would just be a quick shutdown, update the classpath, restart, for > Accumulo. For Hadoop, you may have some upgrade tasks... I'd check > with the Hadoop community first. You could probably upgrade Hadoop > before or after you upgrade Accumulo for this one. I don't think it's > going to matter much. > > If you're concerned about problems you might encounter, I would run > through the upgrades on a small test instance first, just to practice. > If you run into a specific problem doing that, we could probably offer > more specific insight, having probably seen it before. > > > On Wed, Aug 30, 2023 at 4:50 PM Vincent Russell > wrote: > > > > Thank you Christopher. > > > > Does there exist some documentation for upgrading zookeeper from 3.4.14 > to > > 3.8.1? Is there a preferred upgrade path? > > > > Also...Is there any documentation on how this affects hadoop when using > > high availability? We are using hadoop 3.3.1. I want to upgrade to > > hadoop 3.3.5, but I didn't know if that should be done as part of > upgrading > > accumulo or later. > > > > Any experience folks have would be greatly appreciated. > > > > Thank you, > > Vincent > > > > On Thu, Aug 24, 2023 at 12:58 PM Christopher > wrote: > > > > > The correct link seems to be > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/zookeeper#zookeeper-acls > > > > > > It looks like both `bin/accumulo dump-zoo` and `bin/accumulo-util > > > dump-zoo` will do the same thing. It was originally a disconnected > > > utility that existed for convenience, but was incorporated into our > > > regular tooling recently as part of an effort to make all these > > > utilities more easily discoverable and executable. I don't think we > > > realized that it already had a convenient and documented entry point. > > > We were just looking for utilities with a "main" method. Either should > > > work fine. They call the same code. > > > > > > On Thu, Aug 24, 2023 at 11:04 AM Vincent Russell > > > wrote: > > > > > > > > Also the link to: > > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/ZooKeeper#ACLs > > > > > > > > is not resolving. > > > > > > > > What needs to be done at this step? > > > > > > > > > > > > Thanks, > > > > Vincent > > > > > > > > > > > > On Thu, Aug 24, 2023 at 10:36 AM Vincent Russell < > > > vincent.russ...@gmail.com> > > > > wrote: > > > > > > > > > Hello, > > > > > > > > > > I'm practicing going to the upgrade instructions from 2.0.1 to > 2.1.1 > > > and I > > > > > wanted to confirm that the zoo-dump command is run via the > > > accumulo-util > > > > > command and not via accumulo. > > > > > > > > > > The instructions say: > > > > > > > > > > $ACCUMULO_HOME/bin/accumulo dump-zoo --xml --root /accumulo | tee > > > > > PATH_TO_SNAPSHOT > > > > > > > > > > Thank you, > > > > > Vincent > > > > > > > > >
Re: accumulo upgrade from 2.0.1 to 2.1.1
My understanding is that the upgrade of ZK is pretty easy... but I would consult the ZooKeeper community for advice on that, since they are the experts. For Accumulo's part, I believe 2.0 worked fine with newer versions of ZK (3.5+), so you should be able to update ZK first if you didn't want to do them at the same time. There may be some things you might need to do to ensure your classpath is set up correctly for Accumulo if using it with 2.0, since 2.0 might assume ZK 3.4 jar locations. I don't recall the specifics, but I remember it wasn't that bad. fluo-uno had used newer ZK versions with 2.0 for a while. The necessary config changes to support that might even still be scripted in its repo. I'm really not familiar with Hadoop upgrades, but as far as Accumulo is concerned, it would probably treat both 3.3.1 and 3.3.5 as essentially equivalent. I don't think anything substantially changed that would cause Accumulo to even notice a difference. Seems like it would just be a quick shutdown, update the classpath, restart, for Accumulo. For Hadoop, you may have some upgrade tasks... I'd check with the Hadoop community first. You could probably upgrade Hadoop before or after you upgrade Accumulo for this one. I don't think it's going to matter much. If you're concerned about problems you might encounter, I would run through the upgrades on a small test instance first, just to practice. If you run into a specific problem doing that, we could probably offer more specific insight, having probably seen it before. On Wed, Aug 30, 2023 at 4:50 PM Vincent Russell wrote: > > Thank you Christopher. > > Does there exist some documentation for upgrading zookeeper from 3.4.14 to > 3.8.1? Is there a preferred upgrade path? > > Also...Is there any documentation on how this affects hadoop when using > high availability? We are using hadoop 3.3.1. I want to upgrade to > hadoop 3.3.5, but I didn't know if that should be done as part of upgrading > accumulo or later. > > Any experience folks have would be greatly appreciated. > > Thank you, > Vincent > > On Thu, Aug 24, 2023 at 12:58 PM Christopher wrote: > > > The correct link seems to be > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/zookeeper#zookeeper-acls > > > > It looks like both `bin/accumulo dump-zoo` and `bin/accumulo-util > > dump-zoo` will do the same thing. It was originally a disconnected > > utility that existed for convenience, but was incorporated into our > > regular tooling recently as part of an effort to make all these > > utilities more easily discoverable and executable. I don't think we > > realized that it already had a convenient and documented entry point. > > We were just looking for utilities with a "main" method. Either should > > work fine. They call the same code. > > > > On Thu, Aug 24, 2023 at 11:04 AM Vincent Russell > > wrote: > > > > > > Also the link to: > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/ZooKeeper#ACLs > > > > > > is not resolving. > > > > > > What needs to be done at this step? > > > > > > > > > Thanks, > > > Vincent > > > > > > > > > On Thu, Aug 24, 2023 at 10:36 AM Vincent Russell < > > vincent.russ...@gmail.com> > > > wrote: > > > > > > > Hello, > > > > > > > > I'm practicing going to the upgrade instructions from 2.0.1 to 2.1.1 > > and I > > > > wanted to confirm that the zoo-dump command is run via the > > accumulo-util > > > > command and not via accumulo. > > > > > > > > The instructions say: > > > > > > > > $ACCUMULO_HOME/bin/accumulo dump-zoo --xml --root /accumulo | tee > > > > PATH_TO_SNAPSHOT > > > > > > > > Thank you, > > > > Vincent > > > > > >
Re: accumulo upgrade from 2.0.1 to 2.1.1
Thank you Christopher. Does there exist some documentation for upgrading zookeeper from 3.4.14 to 3.8.1? Is there a preferred upgrade path? Also...Is there any documentation on how this affects hadoop when using high availability? We are using hadoop 3.3.1. I want to upgrade to hadoop 3.3.5, but I didn't know if that should be done as part of upgrading accumulo or later. Any experience folks have would be greatly appreciated. Thank you, Vincent On Thu, Aug 24, 2023 at 12:58 PM Christopher wrote: > The correct link seems to be > > https://accumulo.apache.org/docs/2.x/troubleshooting/zookeeper#zookeeper-acls > > It looks like both `bin/accumulo dump-zoo` and `bin/accumulo-util > dump-zoo` will do the same thing. It was originally a disconnected > utility that existed for convenience, but was incorporated into our > regular tooling recently as part of an effort to make all these > utilities more easily discoverable and executable. I don't think we > realized that it already had a convenient and documented entry point. > We were just looking for utilities with a "main" method. Either should > work fine. They call the same code. > > On Thu, Aug 24, 2023 at 11:04 AM Vincent Russell > wrote: > > > > Also the link to: > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/ZooKeeper#ACLs > > > > is not resolving. > > > > What needs to be done at this step? > > > > > > Thanks, > > Vincent > > > > > > On Thu, Aug 24, 2023 at 10:36 AM Vincent Russell < > vincent.russ...@gmail.com> > > wrote: > > > > > Hello, > > > > > > I'm practicing going to the upgrade instructions from 2.0.1 to 2.1.1 > and I > > > wanted to confirm that the zoo-dump command is run via the > accumulo-util > > > command and not via accumulo. > > > > > > The instructions say: > > > > > > $ACCUMULO_HOME/bin/accumulo dump-zoo --xml --root /accumulo | tee > > > PATH_TO_SNAPSHOT > > > > > > Thank you, > > > Vincent > > > >
Re: accumulo upgrade from 2.0.1 to 2.1.1
The correct link seems to be https://accumulo.apache.org/docs/2.x/troubleshooting/zookeeper#zookeeper-acls It looks like both `bin/accumulo dump-zoo` and `bin/accumulo-util dump-zoo` will do the same thing. It was originally a disconnected utility that existed for convenience, but was incorporated into our regular tooling recently as part of an effort to make all these utilities more easily discoverable and executable. I don't think we realized that it already had a convenient and documented entry point. We were just looking for utilities with a "main" method. Either should work fine. They call the same code. On Thu, Aug 24, 2023 at 11:04 AM Vincent Russell wrote: > > Also the link to: > > https://accumulo.apache.org/docs/2.x/troubleshooting/ZooKeeper#ACLs > > is not resolving. > > What needs to be done at this step? > > > Thanks, > Vincent > > > On Thu, Aug 24, 2023 at 10:36 AM Vincent Russell > wrote: > > > Hello, > > > > I'm practicing going to the upgrade instructions from 2.0.1 to 2.1.1 and I > > wanted to confirm that the zoo-dump command is run via the accumulo-util > > command and not via accumulo. > > > > The instructions say: > > > > $ACCUMULO_HOME/bin/accumulo dump-zoo --xml --root /accumulo | tee > > PATH_TO_SNAPSHOT > > > > Thank you, > > Vincent > >
Re: accumulo upgrade from 2.0.1 to 2.1.1
Also the link to: https://accumulo.apache.org/docs/2.x/troubleshooting/ZooKeeper#ACLs is not resolving. What needs to be done at this step? Thanks, Vincent On Thu, Aug 24, 2023 at 10:36 AM Vincent Russell wrote: > Hello, > > I'm practicing going to the upgrade instructions from 2.0.1 to 2.1.1 and I > wanted to confirm that the zoo-dump command is run via the accumulo-util > command and not via accumulo. > > The instructions say: > > $ACCUMULO_HOME/bin/accumulo dump-zoo --xml --root /accumulo | tee > PATH_TO_SNAPSHOT > > Thank you, > Vincent >
