Apache9 commented on PR #7552: URL: https://github.com/apache/hbase/pull/7552#issuecomment-4530862677
> @Apache9 My sincere apologies for delayed response. Here are the details : > > > but it is still a bit strange to me that why we close a store file reader and then we still try to select for compaction? > > So here we are actually seeing that the file is sent for compaction (DateTieredCompaction) and simultaneously when the file is marked as compacted, the reader is closed > > _newCompactedfiles.forEach(HStoreFile::markCompactedAway);_ > > Here we are marking them compacted while a parallel thread is marking all the marked files closed. Whereas in the next step we have : > > _compactedfiles = ImmutableList.sortedCopyOf(storeFileComparator, Iterables.concat(compactedfiles, newCompactedfiles));_ > > Here the sorting happens based on timestamp (_reader.getMaximumTimestamp()_) where we are seeing the NPE. > > The root cause of the NPE is a race condition between store file reader closure and compaction bookkeeping. During compaction, the system performs two logically separate steps: > > * Compaction bookkeeping > The StoreFileManager (e.g. DefaultStoreFileManager) removes compacted store files from its internal active file set via addCompactionResults(...). > * Reader cleanup > The compacted HStoreFile’s reader is eventually closed (via closeStoreFile(...)), either synchronously or asynchronously. > > Before this fix, these two steps were not strictly ordered. As a result, it was possible for a store file’s reader to be closed while the file was still visible to compaction logic. This only affects DateTieredCompaction or is a general problem for all compaction algorithm? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
