[
https://issues.apache.org/jira/browse/FLINK-28172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan closed FLINK-28172.
-------------------------------------
Fix Version/s: 1.16.0
Resolution: Fixed
Merged as 135130379e592d9ae752bc4eea20b6fb222f8c15 into master (1.16).
Thanks [~Feifan Wang] for the contribution!
> Scatter dstl files into separate directories by job id
> ------------------------------------------------------
>
> Key: FLINK-28172
> URL: https://issues.apache.org/jira/browse/FLINK-28172
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.15.0
> Reporter: Feifan Wang
> Assignee: Feifan Wang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.0
>
>
> In the current implementation of {_}FsStateChangelogStorage{_}, dstl files
> from all jobs are put into the same directory (configured via
> {_}dstl.dfs.base-path{_}). Everything is fine if it's a filesystem like
> S3.But if it is a file system like hadoop, there will be some problems.
> First, there may be an upper limit to the number of files in a single
> directory. Increasing this threshold will greatly reduce the performance of
> the distributed file system.
> Second, dstl file management becomes difficult because the user cannot tell
> which job the dstl file belongs to, especially when the retained checkpoint
> is turned on.
> h3. Propose
> # create a subdirectory named with the job id under the _dstl.dfs.base-path_
> directory when the job starts
> # all dstl files upload to the subdirectory
> ( Going a step further, we can even create two levels of subdirectories under
> the _dstl.dfs.base-path_ directory, like _base-path/\{jobId}/dstl ._ This
> way, if the user configures the same dstl.dfs.base-path as
> state.checkpoints.dir, all files needed for job recovery will be in the same
> directory and well organized. )
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
