zentol commented on a change in pull request #12636:
URL: https://github.com/apache/flink/pull/12636#discussion_r440115913
##########
File path:
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
##########
@@ -295,6 +315,19 @@ public void run() {
}
}
+ private List<ArchiveEvent> cleanupJobsBeyondSizeLimit(Set<Path>
jobArchivesToRemove) {
+ Set<String> jobIdsToRemoveFromOverview = new
HashSet<>();
+ for (Path archive : jobArchivesToRemove) {
+
jobIdsToRemoveFromOverview.add(archive.getName());
+ try {
+ archive.getFileSystem().delete(archive,
false);
+ } catch (IOException ioe) {
+ LOG.error("Error while deleting old
archive " + archive, ioe);
+ }
+ }
+ return cleanupExpiredJobs(jobIdsToRemoveFromOverview);
Review comment:
This is redundant; cleanupExpiredJobs already handles this case since
we're skipping `jobsToRemove.remove(jobID);` at L235.
##########
File path:
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java
##########
@@ -99,6 +99,11 @@
.withDescription("Enable HTTPs access to the
HistoryServer web frontend. This is applicable only when the" +
" global SSL flag security.ssl.enabled is set
to true.");
+ public static final ConfigOption<Integer> HISTORY_SERVER_MAX_SIZE =
+ key("historyserver.max.history.size")
+ .defaultValue(50)
Review comment:
We can't enable this by default since this could result in us deleting
externally kept data all of a sudden.
##########
File path:
flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/history/HistoryServerArchiveFetcher.java
##########
@@ -205,6 +212,10 @@ public void run() {
if (jobArchives == null) {
continue;
}
+
+ Arrays.sort(jobArchives,
Comparator.comparingLong(FileStatus::getModificationTime).reversed());
Review comment:
It should be documented that the `maxHistorySize` is enforced per
archive directory, and not globally.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
