[
https://issues.apache.org/jira/browse/FLINK-41198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pia park updated FLINK-41198:
-----------------------------
Description:
With incremental checkpoints, only SST files are shared across checkpoints; the
MANIFEST is uploaded in full every time. Since RocksDB only rolls the MANIFEST
over at max_manifest_file_size (1 GB by default), the checkpoint size of
small-state jobs keeps growing until the job is restarted.
In one of our jobs (240 RocksDB instances, 7 days without restart), the
MANIFEST was ~10.4 MB per instance against ~80 KB of SST files, making up
almost all of a ~2.3 GB checkpoint. Without a restart, it is only bounded by
the 1 GB default per MANIFEST, i.e. ~240 GB per checkpoint for this job.
This can currently only be tuned with a custom RocksDBOptionsFactory. This
ticket proposes a new option,
{*}state.backend.rocksdb.manifest.max-file-size{*}, with no default value so
that the current behavior is unchanged.
I'd like to work on this and already have an implementation ready. Could
someone assign this ticket to me?
was:
With incremental checkpoints, only SST files are shared across checkpoints; the
MANIFEST is uploaded in full every time. Since RocksDB only rolls the MANIFEST
over at max_manifest_file_size (1 GB by default), the checkpoint size of
small-state jobs keeps growing until the job is restarted.
In one of our jobs, the MANIFEST made up almost all of the checkpoint size
after a week without restart (~10 MB vs ~80 KB of SST files per instance).
This can currently only be tuned with a custom RocksDBOptionsFactory. This
ticket proposes a new option,
{*}state.backend.rocksdb.manifest.max-file-size{*}, with no default value so
that the current behavior is unchanged.
> Add a configurable option for RocksDB max_manifest_file_size
> ------------------------------------------------------------
>
> Key: FLINK-41198
> URL: https://issues.apache.org/jira/browse/FLINK-41198
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Reporter: pia park
> Priority: Major
>
> With incremental checkpoints, only SST files are shared across checkpoints;
> the MANIFEST is uploaded in full every time. Since RocksDB only rolls the
> MANIFEST over at max_manifest_file_size (1 GB by default), the checkpoint
> size of small-state jobs keeps growing until the job is restarted.
> In one of our jobs (240 RocksDB instances, 7 days without restart), the
> MANIFEST was ~10.4 MB per instance against ~80 KB of SST files, making up
> almost all of a ~2.3 GB checkpoint. Without a restart, it is only bounded by
> the 1 GB default per MANIFEST, i.e. ~240 GB per checkpoint for this job.
> This can currently only be tuned with a custom RocksDBOptionsFactory. This
> ticket proposes a new option,
> {*}state.backend.rocksdb.manifest.max-file-size{*}, with no default value so
> that the current behavior is unchanged.
> I'd like to work on this and already have an implementation ready. Could
> someone assign this ticket to me?
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
