Christopher, I have run through an upgrade with a bit more data than I did previously and monitors with high availability just don't seem as consistent as they were with 2.0.1.
After upgrading both accumulo monitors are reporting "Manager is Down" and I was only able to achieve that after restarting both of the monitors and managers multiple times. The only ERROR log that I see in the manager logs are (at startup): java.nio.File.NoSuchFileException: /tmp/accumulo-vfs-cache-410@hostname .... at org.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.close(AccumuloVFSClassloader.java:453) I see this on both managers. Any ideas? Thanks, On Mon, Sep 25, 2023 at 12:21 PM Christopher <ctubb...@apache.org> wrote: > I'm not sure about the monitor behavior. I've never had a reason to > run more than one, and I don't think we have any well-defined behavior > for trying to run more than one (I could see an argument for both). It > would be weird if one incorrectly reported "Master[now Manager] is > down" because another monitor was running, though. I would expect the > 2.1 behavior you described over that 2.0 behavior. > > On Mon, Sep 25, 2023 at 11:59 AM Vincent Russell > <vincent.russ...@gmail.com> wrote: > > > > Thank you Christopher, > > > > I did neglect to update the log4j2 files. > > > > One thing that is weird is that the monitors appear to work differently > > with high availability. > > > > With 2.0.1 the accumulo monitors both work and you can hit them both (one > > just says 'Master is down'; however, with 2.1.1, it appears that only one > > works. Is this expected? > > > > Thank you, > > > > On Thu, Sep 14, 2023 at 4:28 PM Christopher <ctubb...@apache.org> wrote: > > > > > I do not know about the error message you saw. StatusConsoleListener > > > is not an Accumulo class, but it looks like it's a log4j one. My best > > > guess is that during your upgrade, you did not migrate your log4j > > > config files over to be based on the log4j2 ones we have as an example > > > in the 2.1 tarball. It's possible that there's some log4j > > > configuration issue that's preventing you from seeing the reason why > > > the Monitor failed to start. > > > > > > To troubleshoot, I would compare your log4j config with what we ship > > > in the 2.1 tarball and also check to see if you've done any log4j > > > config overrides in the system properties or environment variables > > > that control log4j. > > > > > > On Tue, Sep 12, 2023 at 8:10 PM Vincent Russell > > > <vincent.russ...@gmail.com> wrote: > > > > > > > > Thank you Christopher, > > > > > > > > Don't worry. I definitely planned on testing an upgrade on a test > > > > cluster. :) > > > > > > > > I just upgraded a test cluster and everything appeared to go smoothly > > > > except for one thing. > > > > > > > > I am able to login to the accumulo shell and I can see the table > that I > > > > created before the upgrade exists and I can scan that table. > > > > > > > > Before the upgrade the monitor was running on port 9995, but now > nothing > > > > appears to be running on that port. Nothing in the monitor log > > > exception: > > > > > > > > ERROR StatusConsoleListener Unable to send HTTP in appender > [MonitorLog] > > > > IllegalARgumentException: invalid URI schema <<hostname>> > > > > > > > > > > > > This seems not great, but I am not sure if it's related. > > > > > > > > It doesn't look like the monitor is running on another port. Any > > > > suggestions for debugging this? > > > > > > > > Thanks, > > > > Vincent > > > > > > > > > > > > > > > > On Wed, Aug 30, 2023 at 7:09 PM Christopher <ctubb...@apache.org> > wrote: > > > > > > > > > My understanding is that the upgrade of ZK is pretty easy... but I > > > > > would consult the ZooKeeper community for advice on that, since > they > > > > > are the experts. For Accumulo's part, I believe 2.0 worked fine > with > > > > > newer versions of ZK (3.5+), so you should be able to update ZK > first > > > > > if you didn't want to do them at the same time. There may be some > > > > > things you might need to do to ensure your classpath is set up > > > > > correctly for Accumulo if using it with 2.0, since 2.0 might > assume ZK > > > > > 3.4 jar locations. I don't recall the specifics, but I remember it > > > > > wasn't that bad. fluo-uno had used newer ZK versions with 2.0 for a > > > > > while. The necessary config changes to support that might even > still > > > > > be scripted in its repo. > > > > > > > > > > I'm really not familiar with Hadoop upgrades, but as far as > Accumulo > > > > > is concerned, it would probably treat both 3.3.1 and 3.3.5 as > > > > > essentially equivalent. I don't think anything substantially > changed > > > > > that would cause Accumulo to even notice a difference. Seems like > it > > > > > would just be a quick shutdown, update the classpath, restart, for > > > > > Accumulo. For Hadoop, you may have some upgrade tasks... I'd check > > > > > with the Hadoop community first. You could probably upgrade Hadoop > > > > > before or after you upgrade Accumulo for this one. I don't think > it's > > > > > going to matter much. > > > > > > > > > > If you're concerned about problems you might encounter, I would run > > > > > through the upgrades on a small test instance first, just to > practice. > > > > > If you run into a specific problem doing that, we could probably > offer > > > > > more specific insight, having probably seen it before. > > > > > > > > > > > > > > > On Wed, Aug 30, 2023 at 4:50 PM Vincent Russell > > > > > <vincent.russ...@gmail.com> wrote: > > > > > > > > > > > > Thank you Christopher. > > > > > > > > > > > > Does there exist some documentation for upgrading zookeeper from > > > 3.4.14 > > > > > to > > > > > > 3.8.1? Is there a preferred upgrade path? > > > > > > > > > > > > Also...Is there any documentation on how this affects hadoop when > > > using > > > > > > high availability? We are using hadoop 3.3.1. I want to > upgrade to > > > > > > hadoop 3.3.5, but I didn't know if that should be done as part of > > > > > upgrading > > > > > > accumulo or later. > > > > > > > > > > > > Any experience folks have would be greatly appreciated. > > > > > > > > > > > > Thank you, > > > > > > Vincent > > > > > > > > > > > > On Thu, Aug 24, 2023 at 12:58 PM Christopher < > ctubb...@apache.org> > > > > > wrote: > > > > > > > > > > > > > The correct link seems to be > > > > > > > > > > > > > > > > > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/zookeeper#zookeeper-acls > > > > > > > > > > > > > > It looks like both `bin/accumulo dump-zoo` and > `bin/accumulo-util > > > > > > > dump-zoo` will do the same thing. It was originally a > disconnected > > > > > > > utility that existed for convenience, but was incorporated > into our > > > > > > > regular tooling recently as part of an effort to make all these > > > > > > > utilities more easily discoverable and executable. I don't > think we > > > > > > > realized that it already had a convenient and documented entry > > > point. > > > > > > > We were just looking for utilities with a "main" method. Either > > > should > > > > > > > work fine. They call the same code. > > > > > > > > > > > > > > On Thu, Aug 24, 2023 at 11:04 AM Vincent Russell > > > > > > > <vincent.russ...@gmail.com> wrote: > > > > > > > > > > > > > > > > Also the link to: > > > > > > > > > > > > > > > > > > > https://accumulo.apache.org/docs/2.x/troubleshooting/ZooKeeper#ACLs > > > > > > > > > > > > > > > > is not resolving. > > > > > > > > > > > > > > > > What needs to be done at this step? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Vincent > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 24, 2023 at 10:36 AM Vincent Russell < > > > > > > > vincent.russ...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > > > > > I'm practicing going to the upgrade instructions from > 2.0.1 to > > > > > 2.1.1 > > > > > > > and I > > > > > > > > > wanted to confirm that the zoo-dump command is run via the > > > > > > > accumulo-util > > > > > > > > > command and not via accumulo. > > > > > > > > > > > > > > > > > > The instructions say: > > > > > > > > > > > > > > > > > > $ACCUMULO_HOME/bin/accumulo dump-zoo --xml --root > /accumulo | > > > tee > > > > > > > > > PATH_TO_SNAPSHOT > > > > > > > > > > > > > > > > > > Thank you, > > > > > > > > > Vincent > > > > > > > > > > > > > > > > > > > > > > > > >
