argoyal2212 opened a new pull request, #29008:
URL: https://github.com/apache/flink/pull/29008
## What is the purpose of the change
`HistoryServerOptions.HISTORY_SERVER_RETAINED_JOBS` currently gates both (a)
which archives the `HistoryServer` polls/processes locally, and (b) which
archives are deleted from the remote archive directory. This means operators
cannot keep a large/unbounded remote job-archive history while only actively
refreshing a small recent window locally — trimming the local processing window
also deletes the "older" jobs remotely.
This closes that gap, which is what FLIP-505 originally set out to solve,
before FLIP-584/585 (FLINK-39911, FLINK-40097) landed a more general pluggable
`ArchiveStorage` backend + on-demand lazy-fetch mechanism that already covers
FLIP-505's other goals (per-job on-demand fetch, avoiding local disk/inode
exhaustion via a pluggable storage backend). This PR adds the one remaining
piece: decoupling remote retention from the local processing limit.
## Brief change log
- Added `historyserver.archive.retain-remote-beyond-local-limit` (boolean,
default `false`, fully backward compatible).
- `HistoryServerArchiveFetcher.scanArchives()`: when enabled, archives
beyond `historyserver.archive.retained-jobs` are cleaned up locally only (new
`cleanupLocalArchivesBeyondRetainedLimit`), leaving the remote archive intact,
instead of calling `cleanupArchivesBeyondRetainedLimit` (which deletes both
local and remote).
- Such archives remain reachable on demand via the existing
`lazyFetchArchiveProactively` path when `historyserver.archive.load.mode=LAZY`.
- `HistoryServer` reads and wires the new option into the job-archive
fetcher (applications are unaffected — this only applies to job archives,
matching FLIP-505's original scope).
- Regenerated
`docs/layouts/shortcodes/generated/history_server_configuration.html`.
## Verifying this change
This change added tests and can be verified as follows:
- Added
`HistoryServerArchiveFetcherTest#testArchivesBeyondRetainedLimitAreDeletedFromRemoteByDefault`
and `#testArchivesBeyondRetainedLimitAreKeptRemotelyWhenConfigured`, covering
both storage backends (File/RocksDB), asserting: default behavior is unchanged
(remote archive deleted beyond limit); with the flag enabled, the remote
archive persists, is not locally cached, and is still fetchable on demand.
- All 24 tests in `HistoryServerArchiveFetcherTest` pass locally.
## Does this pull request potentially affect one of the following parts?
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes, one new `@PublicEvolving` `ConfigOption`
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? yes
- If yes, how is the feature documented? JavaDocs + generated
configuration docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
