virajjasani commented on code in PR #5893:
URL: https://github.com/apache/hadoop/pull/5893#discussion_r1314086566
##########
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##########
@@ -444,6 +444,7 @@ private void deleteBlockFileAndEvictCache(Entry
elementToPurge) {
entryListSize--;
prefetchingStatistics.blockRemovedFromFileCache();
blocks.remove(elementToPurge.blockNumber);
+ prefetchingStatistics.blockEvictedFromFileCache();
Review Comment:
They will be equal, until the input stream is closed. When closed, all
blocks get removed, which only increments `blockRemovedFromFileCache` but not
`blockEvictedFromFileCache`
##########
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingLruEviction.java:
##########
@@ -172,6 +174,8 @@ public void testSeeksWithLruEviction() throws Throwable {
LambdaTestUtils.eventually(TIMEOUT_MILLIS, INTERVAL_MILLIS, () -> {
LOG.info("IO stats: {}", ioStats);
verifyStatisticGaugeValue(ioStats, STREAM_READ_BLOCKS_IN_FILE_CACHE, 0);
+ assertThatStatisticCounter(ioStats,
+ STREAM_EVICT_BLOCKS_FROM_FILE_CACHE).isGreaterThanOrEqualTo(5);
Review Comment:
there is bit of an async behavior that will likely make the test flaky even
with eventually because the order in which lazy seek is going to trigger block
cache is not guaranteed and that can still sometimes hurt the exact number that
we are trying to match.
i really wish we could do this but it's still lazy seek and verifying the
exact order of seeks leading to cache load is not guaranteed every time (some
of the problems are also being discussed on unbuffer PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
