keith-turner commented on code in PR #4551:
URL: https://github.com/apache/accumulo/pull/4551#discussion_r1597505373
##########
server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java:
##########
@@ -246,9 +246,30 @@ public ScanServer(ScanServerOpts opts, String[] args) {
LOG.warn(
"Tablet metadata caching less than one minute, may cause excessive
scans on metadata table.");
}
- tabletMetadataCache =
- Caffeine.newBuilder().expireAfterWrite(cacheExpiration,
TimeUnit.MILLISECONDS)
-
.scheduler(Scheduler.systemScheduler()).recordStats().build(tabletMetadataLoader);
+
+ // Get the cache refresh percentage property
+ // Value must be less than 100% as 100 or over would effectively disable
it
+ double cacheRefreshPercentage =
+
getConfiguration().getFraction(Property.SSERV_CACHED_TABLET_METADATA_REFRESH_PERCENT);
+ Preconditions.checkArgument(cacheRefreshPercentage < cacheExpiration,
+ "Tablet metadata cache refresh percentage is '%s' but must be less
than 1",
+ cacheRefreshPercentage);
+
+ var builder = Caffeine.newBuilder().expireAfterWrite(cacheExpiration,
TimeUnit.MILLISECONDS)
Review Comment:
> The idea is to aggregate individual refreshes over a time window, perform
the bulk operation, and complete the respective CompletableFuture with the
value for the given key.
I see. So we could do something like have single thread that continually
pull refresh work off a queue, then does an Accumulo batch scan to read all
data, and then completes futures related to work it pulled off the queue. The
reload pluging to caffeine could push uncompleted futures onto the queue.
##########
server/tserver/src/main/java/org/apache/accumulo/tserver/ScanServer.java:
##########
@@ -246,9 +246,30 @@ public ScanServer(ScanServerOpts opts, String[] args) {
LOG.warn(
"Tablet metadata caching less than one minute, may cause excessive
scans on metadata table.");
}
- tabletMetadataCache =
- Caffeine.newBuilder().expireAfterWrite(cacheExpiration,
TimeUnit.MILLISECONDS)
-
.scheduler(Scheduler.systemScheduler()).recordStats().build(tabletMetadataLoader);
+
+ // Get the cache refresh percentage property
+ // Value must be less than 100% as 100 or over would effectively disable
it
+ double cacheRefreshPercentage =
+
getConfiguration().getFraction(Property.SSERV_CACHED_TABLET_METADATA_REFRESH_PERCENT);
+ Preconditions.checkArgument(cacheRefreshPercentage < cacheExpiration,
+ "Tablet metadata cache refresh percentage is '%s' but must be less
than 1",
+ cacheRefreshPercentage);
+
+ var builder = Caffeine.newBuilder().expireAfterWrite(cacheExpiration,
TimeUnit.MILLISECONDS)
Review Comment:
> The idea is to aggregate individual refreshes over a time window, perform
the bulk operation, and complete the respective CompletableFuture with the
value for the given key.
I see. So we could do something like have single thread that continually
pulls refresh work off a queue, then does an Accumulo batch scan to read all
data, and then completes futures related to work it pulled off the queue. The
reload pluging to caffeine could push uncompleted futures onto the queue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
