keith-turner commented on issue #2667: URL: https://github.com/apache/accumulo/issues/2667#issuecomment-1115139592
I think I have found one possible cause to this. Before doing the consistency check on close the following code will wait for there the be zero non-external compactions by looking at the `runningJobs` set. https://github.com/apache/accumulo/blob/b3ac225d43bacf86bb280be48de750c409f45e66/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java#L1504-L1505 The following code is called when a compaction completes and it removes itself from `runningJobs` and then calls `selectFiles()` https://github.com/apache/accumulo/blob/b3ac225d43bacf86bb280be48de750c409f45e66/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableImpl.java#L1220-L1233 The call to `selectFiles()` eventaully calls the the following code which updates the metadata table and then updates the tablets in memory last compact ID. The following code can run concurrently with the close consistency check. https://github.com/apache/accumulo/blob/b3ac225d43bacf86bb280be48de750c409f45e66/server/tserver/src/main/java/org/apache/accumulo/tserver/tablet/CompactableUtils.java#L439-L444 Looking at the error message `Closed tablet 5<;r13c3b lastCompactID is inconsistent with metadata : 25 != 23` * 25 is from metadata table * 23 is from tablet instance variable lastCompactID Which is consistent with the metadata table being updated first followed by the lastCompactID instance variable. I think this code with the race conditions only executes when an empty tablet is forced to compact, so all it does is write the compact id to the metadata table to signify its done w/o any files changes. There was a forced compaction in the manager logs a bit before this split that closed the tablet. I am going to look into this some more and see if I can figure out if there is a problem with reordering the code to avoid the race condition. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
