argoyal2212 commented on code in PR #29008:
URL: https://github.com/apache/flink/pull/29008#discussion_r4264147271
##########
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServer.java:
##########
@@ -291,6 +291,8 @@ public HistoryServer(
config.get(HISTORY_SERVER_LAZY_FETCH_EXECUTOR_COMMON_POOL_SIZE);
int lazyFetchExecutorIndividualPoolSize =
config.get(HISTORY_SERVER_LAZY_FETCH_EXECUTOR_INDIVIDUAL_POOL_SIZE);
+ boolean retainRemoteBeyondLocalLimit =
+
config.get(HistoryServerOptions.HISTORY_SERVER_RETAIN_REMOTE_BEYOND_LOCAL_LIMIT);
Review Comment:
Done. Both option values are now logged at startup in HistoryServer.
##########
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java:
##########
@@ -240,9 +273,16 @@ void scanArchives(
&& processExpiredArchiveDeletion) {
events.addAll(cleanupExpiredArchives(archivesToRemove));
}
- // clean remote and local
if (!archivesBeyondRetainedLimit.isEmpty()) {
-
events.addAll(cleanupArchivesBeyondRetainedLimit(archivesBeyondRetainedLimit));
+ if (retainRemoteBeyondLocalLimit) {
+ // clean local only; the remote archive is left in place
and remains
+ // fetchable on demand (e.g. via LAZY archive load mode).
+ events.addAll(
+
cleanupLocalArchivesBeyondRetainedLimit(archivesBeyondRetainedLimit));
Review Comment:
Fixed. Remote retention (count and TTL) is now checked first and deletes as
before, so TTL-expired archives are never kept. Only archives that pass it can
be evicted locally. The boolean flag is gone.
##########
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServer.java:
##########
@@ -301,7 +306,8 @@ public HistoryServer(
archiveStorage,
archiveMetaInfoCache,
lazyFetchExecutorCommonPoolSize,
- lazyFetchExecutorIndividualPoolSize);
+ lazyFetchExecutorIndividualPoolSize,
+ retainRemoteBeyondLocalLimit);
applicationArchiveFetcher =
new HistoryServerApplicationArchiveFetcher<>(
Review Comment:
Right, this only covers legacy job archives. I've said so in the option
description and docs. Application support would be a separate change.
##########
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java:
##########
@@ -223,9 +257,17 @@ void scanArchives(
fileOrderedIndexOnModifiedTime++;
if (!retainedStrategy.shouldRetain(archive,
fileOrderedIndexOnModifiedTime)) {
- archivesBeyondRetainedLimit
- .computeIfAbsent(refreshDir, ignored -> new
HashSet<>())
- .add(archivePath);
+ if (retainedStrategy.isExpiredByTtl(archive)) {
+ // TTL expiry always applies remote deletion,
regardless of
+ // retainRemoteBeyondLocalLimit, which only
concerns the count limit.
+ archivesExpiredByTtl
+ .computeIfAbsent(refreshDir, ignored ->
new HashSet<>())
+ .add(archivePath);
+ } else {
+ archivesBeyondRetainedLimit
+ .computeIfAbsent(refreshDir, ignored ->
new HashSet<>())
+ .add(archivePath);
+ }
continue;
}
Review Comment:
Fixed. Local eviction now only touches ids an earlier scan cached, and
recently viewed jobs are skipped, so repeated scans do nothing.
##########
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java:
##########
@@ -178,6 +178,34 @@ public class HistoryServerOptions {
.text(LEGACY_NOTE_MESSAGE)
.build());
+ /**
+ * If this option is enabled, job archives that fall outside {@link
+ * #HISTORY_SERVER_RETAINED_JOBS} are no longer processed/refreshed
locally, but are kept in the
+ * remote archive directory instead of being deleted. They remain
reachable on demand (e.g. by
+ * directly requesting {@code /jobs/<jobId>} in {@link
HistoryServerArchiveLoadMode#LAZY}
+ * mode).
+ */
+ public static final ConfigOption<Boolean>
HISTORY_SERVER_RETAIN_REMOTE_BEYOND_LOCAL_LIMIT =
+ key("historyserver.archive.retain-remote-beyond-local-limit")
Review Comment:
Switched to the voted keys, historyserver.archive.cached-retained-jobs and
historyserver.archive.num-cached-most-recently-viewed-jobs, and dropped the
custom one. Setting cached-retained-jobs is what decouples local from remote,
as in the FLIP. I also added the most recently viewed cache here. I'll check
#26878 for overlap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
