mjwall commented on pull request #2293: URL: https://github.com/apache/accumulo/pull/2293#issuecomment-931666584
> > I am worried my assumptions are wrong here, but assuming they are correct the logic in ensureAllTablesChecked is this. > > The logic in ensureAllTablesChecked looks great and I really like the variable names. As for the assumptions, I was thinking of this a bit differently. Not in terms of the delete markers, but more in terms of what we should see in the table section of the metadata table when scanning for file references. > > 1. If a table id exists in zookeeper before and after scanning the metadata table for references then that table id should be seen in the metadata table when scanning for file references. > 2. If table id exists in zookeeper before xor after scanning the metadata table for references then that table id may be seen in the metadata table when scanning for file references. > > This PR is adding a sanity check for case 1 above. For case 2 we are not adding any sanity checks, the question is are there any checks that could be added for that case? > > I think there is a special case of case 2 where we could be a bit more strict. If we see no table ids in ZK before and after scanning the metadata table, but we do see table id references in the metadata table and delete markers. We may want to avoid doing any GC in this case as this indicate that reading from ZK is problematic. Seeing zero table ids in ZK could also be caused by there being in zero tables in Accumulo, however in this case we would not expect to see any table id references in the metadata table. @keith-turner thanks so much for looking at this. You are right, I was only concerned about #1. Failing to remove candidates still in use cause deletion of needed files My first thought about scenario 2 was this. If we scan things in metadata that weren't in the before or after ids, we run the risk of over removing candidates, which results in those candidates not be deleted. But thinking about it further, I realize you might be getting at this. For whatever reason, something unexpected happened during our scan of the metadata table. Therefore, we should except and start the GC process again. Is that accurate? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
