EdColeman opened a new issue #1689: URL: https://github.com/apache/accumulo/issues/1689
The symptom is that the system is encountering an IOException (incorrect data check) where the zlib decompressor cannot uncompress a file. The file would have been created during a previous compaction. The tserver seems to be in a bad state, and rather than writing corrupt files, it would be preferable if the condition(s) can't be corrected, then stop the server. On processing the corrupted files: The IOException is being thrown on lines 348 or 385 of Compactor.compactLocalityGroup. (exceptions occur on both) https://github.com/apache/accumulo/blob/7a2d12eaf785f924f555733a54f40828cdb2414f/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Compactor.java#L348 or https://github.com/apache/accumulo/blob/7a2d12eaf785f924f555733a54f40828cdb2414f/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/Compactor.java#L385 The file(s) can be partially processed with rfile-info. You can examine the file metadata, but if you do something that reads the entire file (like -keyStats) then it fails with an exception reported in line 815 of RFile LocalityGroupReader. When the files are being compacted it looks like the tserver was in an unhealthy state. Potential contributors: 1) The dynamic class loader kept continually rebuilding. AccumuloReloadingVFSClassLoader run loop just keeps executing - the message at line 83 appears frequently in the log. https://github.com/apache/accumulo/blob/7a2d12eaf785f924f555733a54f40828cdb2414f/start/src/main/java/org/apache/accumulo/start/classloader/vfs/AccumuloReloadingVFSClassLoader.java#L83 2) The tablet servers are throwing a null pointer exception that is being logged by https://github.com/apache/accumulo/blob/7a2d12eaf785f924f555733a54f40828cdb2414f/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServerResourceManager.java#L452 There may be other issues, but these stand out in the logs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
