Biao Geng created FLINK-37518:
---------------------------------
Summary: Optimize number of files created by flink history server
Key: FLINK-37518
URL: https://issues.apache.org/jira/browse/FLINK-37518
Project: Flink
Issue Type: Improvement
Reporter: Biao Geng
Attachments: image-2025-03-20-11-41-19-610.png
When using flink session cluster to run large-scale batch jobs, we may use
flink history server to record the finished job status. However, it can easily
use up all inodes of the filesystem as it will create lots of small files. For
example, as the following picture shows, for a job with parallelism=640, each
stage will have 640 records. If we run lots of jobs like running TPC-DS tests,
the newly created small files would drain the filesystem's inode.
!image-2025-03-20-11-41-19-610.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
