[
https://issues.apache.org/jira/browse/FLINK-36679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ferenc Csaky closed FLINK-36679.
--------------------------------
Assignee: Mingliang Liu
Resolution: Fixed
[{{edfd7f9}}|https://github.com/apache/flink/commit/edfd7f99755d490de44c3821810ab2c9f181fee3]
in master
> Add a metric to track checkpoint _metadata size
> -----------------------------------------------
>
> Key: FLINK-36679
> URL: https://issues.apache.org/jira/browse/FLINK-36679
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Metrics
> Affects Versions: 1.18.1, 1.20.0, 1.19.1
> Reporter: Mingliang Liu
> Assignee: Mingliang Liu
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.2.0
>
>
> Currently we expose multiple metrics for the checkpoint size. One specific
> interesting data point is the {{_metadata}} file size, which can also be
> added as a metric. The {{_metadata}} file has multiple types of data to store
> including operator states, coordinator states and properties. Its size should
> be scoped to a reasonable range, otherwise job may take too long to restore
> from checkpoints and/or fail to start when its size exceeding RPC frame limit.
> However, we saw multiple times the {{_metadata}} file bloats up to 100MB~3GB
> causing job slow and/or fail to start. In user mail list (for e.g.
> [[1]|https://lists.apache.org/thread/dttjs2v412xd7slrrx94837ch8wjfo11],
> [[2]|https://lists.apache.org/thread/yj66dnbs7xrmbspdltq3yfptccm25llt]) and
> FLINK-32658 community reported similar problems.
> Tracking the metadata size can be helpful for operations. Currently we have
> following use cases:
> * The first case was primarily informational. When a job fails to deploy
> because the payload exceeds the Akka frame size, it’s often unclear what
> caused the issue. By monitoring the metrics of the checkpoint _metadata file,
> the problem becomes more apparent. As a workaround to restore the job, users
> typically increase the Akka frame size config, or redeploy it from an empty
> state and rewind the Kafka broker’s committed offsets for the consumer group.
> * The second case involved a suspected bug in Kafka source v1, which occurs
> when the operator UID changes. In this scenario, the union state of the old
> operator grows exponentially with each savepoint, eventually resulting in a
> multiple GB _metadata file that exceeds the Akka frame size. We recommended
> that users use the State Processor API to rewrite the checkpoint file,
> omitting the problematic operator state for the obsolete operator. Although
> we haven’t identified the root cause, we advised users to maintain
> deterministic operator UIDs. In the long term, we plan to migrate to Kafka
> source v2 (FLIP-27).
> * The final case involved a _metadata file that was growing in size, but
> still contained valid data. However, the file became too large to allow for
> quick job restarts or rescaling. We advised the user to improve the
> serialization of the operator state if possible, or to minimize heavy usage
> of operator state to avoid such issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
