keith-turner commented on a change in pull request #2320:
URL: https://github.com/apache/accumulo/pull/2320#discussion_r761332628
##########
File path:
server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java
##########
@@ -794,6 +802,58 @@ public void run() {
}
}, 0, 5000, TimeUnit.MILLISECONDS);
+ int tabletCheckFrequency = 30 + random.nextInt(31); // random 30-60 minute
delay
+ // Periodically check that metadata of tablets matches what is held in
memory
+
ThreadPools.createGeneralScheduledExecutorService(aconf).scheduleWithFixedDelay(()
-> {
+ final SortedMap<KeyExtent,Tablet> onlineTabletsSnapshot =
onlineTablets.snapshot();
+
+ final SortedSet<KeyExtent> userTablets = new TreeSet<>();
+ final SortedSet<KeyExtent> nonUserTablets = new TreeSet<>();
+
+ // Create subsets of tablets based on DataLevel: one set who's DataLevel
is USER and another
+ // containing the remaining tablets (those who's DataLevel is ROOT or
METADATA).
+ // This needs to happen so we can use .readTablets() on the
DataLevel.USER tablets in order
+ // to reduce RPCs.
+ // TODO: Push this partitioning, based on DataLevel, to ample - accumulo
issue #2373
+ onlineTabletsSnapshot.forEach((ke, tablet) -> {
+ if (Ample.DataLevel.of(ke.tableId()) == Ample.DataLevel.USER) {
+ userTablets.add(ke);
+ } else {
+ nonUserTablets.add(ke);
+ }
+ });
+
+ Map<KeyExtent,Long> updateCounts = new HashMap<>();
+
+ // gather updateCounts for each tablet
+ onlineTabletsSnapshot.forEach((ke, tablet) -> {
+ updateCounts.put(ke, tablet.getUpdateCount());
+ });
+
+ List<TabletMetadata> tmdList;
+
+ // gather metadata for all tablets with DataLevel.USER using
readTablets()
+ try (TabletsMetadata tabletsMetadata =
getContext().getAmple().readTablets()
+ .forTablets(userTablets).fetch(FILES, LOGS, ECOMP,
PREV_ROW).build()) {
+ tmdList = tabletsMetadata.stream().collect(Collectors.toList());
+ }
+
+ // gather metadata for all tablets with DataLevel.ROOT or METADATA using
readTablet()
+ nonUserTablets.forEach(extent -> {
+ TabletMetadata tabletMetadata =
+ getContext().getAmple().readTablet(extent, FILES, LOGS, ECOMP,
PREV_ROW);
+ tmdList.add(tabletMetadata);
Review comment:
> Using streams seemed straight forward enough. Addressed in b334df9
That looks really nice. Looking at the change, wondering if we could just
skip creating the list and replace the for each loop over the list with
`Stream.concat(...).foreach(tabletMetadata -> {...})` As long as the code in
the foreach loop over the list is not throwing checked exceptions, it would
probably work out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
