pia park created FLINK-41198:
--------------------------------
Summary: Add a configurable option for RocksDB
max_manifest_file_size
Key: FLINK-41198
URL: https://issues.apache.org/jira/browse/FLINK-41198
Project: Flink
Issue Type: Improvement
Components: Runtime / State Backends
Reporter: pia park
With incremental checkpoints, only SST files are shared across checkpoints; the
MANIFEST is uploaded in full every time. Since RocksDB only rolls the MANIFEST
over at max_manifest_file_size (1 GB by default), the checkpoint size of
small-state jobs keeps growing until the job is restarted.
In one of our jobs, the MANIFEST made up almost all of the checkpoint size
after a week without restart (~10 MB vs ~80 KB of SST files per instance).
This can currently only be tuned with a custom RocksDBOptionsFactory. This
ticket proposes a new option,
{*}state.backend.rocksdb.manifest.max-file-size{*}, with no default value so
that the current behavior is unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
