keith-turner commented on code in PR #2792:
URL: https://github.com/apache/accumulo/pull/2792#discussion_r969790741
##########
server/gc/src/main/java/org/apache/accumulo/gc/GarbageCollectionAlgorithm.java:
##########
@@ -216,6 +225,46 @@ private long
removeBlipCandidates(GarbageCollectionEnvironment gce,
return blipCount;
}
+ @VisibleForTesting
+ /**
+ * Double check no tables were missed during GC
+ */
+ protected void ensureAllTablesChecked(Set<TableId> tableIdsBefore,
Set<TableId> tableIdsSeen,
+ Set<TableId> tableIdsAfter) {
+
+ // if a table was added or deleted during this run, it is acceptable to not
+ // have seen those tables ids when scanning the metadata table. So get the
intersection
+ final Set<TableId> tableIdsMustHaveSeen = new HashSet<>(tableIdsBefore);
+ tableIdsMustHaveSeen.retainAll(tableIdsAfter);
+
+ if (tableIdsMustHaveSeen.isEmpty() && !tableIdsSeen.isEmpty()) {
+ if (!tableIdsBefore.isEmpty() && tableIdsAfter.isEmpty()) {
+ throw new RuntimeException("ZK returned no table ids after scanning
for references,"
+ + " maybe all the tables were deleted");
+ } else {
+ // we saw no table ids in ZK but did in the metadata table. This is
unexpected.
+ throw new RuntimeException(
+ "Saw no table ids in ZK but did see table ids in metadata table: "
+ tableIdsSeen);
+ }
+ }
Review Comment:
There are few possible interleavings that could make different empty sets
possible under normal circumstances.
The following would result in tablesIdsBefore and tableIdAfter being empty
while tableIdsSeen is not empty.
1. tableIdsBefore is read from ZK and is empty
2. a new table is created
3. tableIdsSeen is read from metadata table and is not empty
4. the table single is deleted
5. tableIdsAfter is read from ZK and is empty
The following will result in only tableIdsBefore being empty
1. tableIdsBefore is read from ZK and is empty
2. a new table is created
3. tableIdsSeen is read from metadata table and is not empty
4. tableIdsAfter is read from ZK and is not empty
The following will result in only tableIdsAfter being empty
1. a new table is created
2. tableIdsBefore is read from ZK and is not empty
4. tableIdsSeen is read from metadata table and is not empty
5. the table single is deleted
6. tableIdsAfter is read from ZK and is empty
Given that there are multiple transient situations that lead to different
empty sets I think we can have one error message like the following. Its long,
but I didn't attempt to wrap as I typed it up in GH.
```suggestion
if (tableIdsMustHaveSeen.isEmpty() && !tableIdsSeen.isEmpty()) {
throw new RuntimeException("Garbage collection will not proceed
because table ids were seen in the metadata table and none were seen Zookeeper.
This can have two causes. First, total number of tables going to/from zero
during a GC cycle will cause this. Second, it could be caused by corruption of
the metadata table and/or Zookeeper. Only the second cause is problematic, but
there is not way to distinguish between the two causes so this GC cycle will
not proceed. The first cause should be transient and one would not expect to
see this message repeated in subsequent GC cycles. ");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
