spuru9 commented on code in PR #29008:
URL: https://github.com/apache/flink/pull/29008#discussion_r4500517369
##########
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java:
##########
@@ -229,23 +328,46 @@ void scanArchives(
continue;
}
+ if (cachedRetainedJobs > 0
+ && fileOrderedIndexOnModifiedTime >
cachedRetainedJobs) {
+ // the remote archive stays, only evict what an
earlier scan cached
+ if
(cachedArchivesPerRefreshDirectory.get(refreshDir).contains(archiveId)) {
+ archivesToEvictLocally
+ .computeIfAbsent(refreshDir, ignored ->
new HashSet<>())
+ .add(archiveId);
+ }
+ continue;
+ }
+
if (fetch) {
- fetchArchive(refreshDir, archiveId, archivePath,
archiveLoadMode, events);
+ synchronized (localStateLock) {
Review Comment:
minor: localStateLock is now taken by every job request
(acquire/release/mark), but it's also held across slow I/O: the scan's archive
read in fetchArchive, deleting an evicted job's files, and rebuilding the
overview. In minikube an in-window job's /config went from under 60 ms to
however long the lock was held, about 250 ms while a 40k-file job was evicted
and up to 700 ms while a scan read new archives, and the scan case also hits
LAZY setups that never set cached-retained-jobs, so "when unset, behavior is
unchanged" isn't quite true anymore. Could the I/O move outside the lock and
keep only the bookkeeping under it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
