ctubbsii commented on issue #1689: URL: https://github.com/apache/accumulo/issues/1689#issuecomment-695053928
Thanks, that helps a lot. > 16:51:02,398 - WARN loadTablet message from a master that does not hold the master lock Hmm. Did a master get into a bad state first and fail over to a secondary? This seems weird. I wonder if the master was having issues also. > 19:18:01,855 - vfs.AccumuloReloadingVFSClassLoader ERROR: Invalid URI escape sequence What's this? This looks like it was triggered by a bad VFS classloader configuration. Maybe that'd be a good place to guard against. It'd be very hard to protect code against a bad classloader... because it could be doing bad things anywhere. But, it should be possible to protect the classloader from bad configuration / user error. > 16:53:22,105 - ERROR: memory manager failed null. > 16:53:22,105 - First memory manager failed. I don't recognize either of these two messages, and cannot find anything in the 1.9.3 code that looks like these messages. Are you sure you aren't seeing this error in some fork of Accumulo with custom code in these areas? > ``` > [time] [tserver.TableServerResourceManager] TabletServerResourceManager ERROR: Memory manager failed null > java.lang.NullPointerException > at org.apache.accumulo.core.client.impl.Tables.exists(Tables.java:148) > at org.apache.accumulo.server.tabletserver.LargestFirstMemoryManager.tableExists(LargestFirstMemoryManager.java:153) > ``` This certainly looks like ZooKeeper returned null for the list of children on that entry. However, I can see no way that would be possible unless the ZooKeeper client threads were somehow also in a bad state. If that's the case, then while you saw this error occur in the memory manager... it could actually occur pretty much anywhere. I wonder: were the ZooKeeper jars being loaded via the reloading VFS classloader? If so, that might be a ready explanation for how the ZooKeeper code failed here as a consequence of a bug or failure condition in that classloader, and indicate that the underlying issue is that classloader, and that this stack trace might be a red herring. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
