[
https://issues.apache.org/jira/browse/FLINK-36679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mingliang Liu updated FLINK-36679:
----------------------------------
Description:
Currently we expose multiple metrics for the checkpoint size. One specific
interesting data point is the {{_metadata}} file size, which can also be added
as a metric. The {{_metadata}} file has multiple types of data to store
including operator states, coordinator states and properties. Its size should
be scoped to a reasonable range, otherwise job may take too long to restore
from checkpoints and/or fail to start when its size exceeding RPC frame limit.
However, we saw multiple times the {{_metadata}} file bloats up to 100MB~3GB
causing job slow and/or fail to start. In user mail list (for e.g.
[[1]|https://lists.apache.org/thread/dttjs2v412xd7slrrx94837ch8wjfo11],
[[2]|https://lists.apache.org/thread/yj66dnbs7xrmbspdltq3yfptccm25llt]) and
FLINK-32658 community reported similar problems.
Tracking the metadata size can be helpful for operations. Currently we have
following use cases:
* The first case was primarily informational. When a job fails to deploy
because the payload exceeds the Akka frame size, it’s often unclear what caused
the issue. By monitoring the metrics of the checkpoint _metadata file, the
problem becomes more apparent. As a workaround to restore the job, users
typically increase the Akka frame size config, or redeploy it from an empty
state and rewind the Kafka broker’s committed offsets for the consumer group.
* The second case involved a suspected bug in Kafka source v1, which occurs
when the operator UID changes. In this scenario, the union state of the old
operator grows exponentially with each savepoint, eventually resulting in a
multiple GB _metadata file that exceeds the Akka frame size. We recommended
that users use the State Processor API to rewrite the checkpoint file, omitting
the problematic operator state for the obsolete operator. Although we haven’t
identified the root cause, we advised users to maintain deterministic operator
UIDs. In the long term, we plan to migrate to Kafka source v2 (FLIP-27).
* The final case involved a _metadata file that was growing in size, but still
contained valid data. However, the file became too large to allow for quick job
restarts or rescaling. We advised the user to improve the serialization of the
operator state if possible, or to minimize heavy usage of operator state to
avoid such issues.
was:
Currently we expose multiple metrics for the checkpoint size. One specific
interesting data point is the {{_metadata}} file size, which can also be added
as a metric. The {{_metadata}} file has multiple types of data to store
including operator states, coordinator states and properties. Its size should
be scoped to a reasonable range, otherwise job may take too long to restore
from checkpoints and/or fail to start when its size exceeding RPC frame limit.
However, we saw multiple times the {{_metadata}} file bloats up to 100MB~3GB
causing job slow and/or fail to start. In user mail list (for e.g.
[[1]|https://lists.apache.org/thread/dttjs2v412xd7slrrx94837ch8wjfo11],
[[2]|https://lists.apache.org/thread/yj66dnbs7xrmbspdltq3yfptccm25llt]) and
FLINK-32658 community reported similar problems.
Tracking the metadata size can be helpful for operations.
> Add a metric to track checkpoint _metadata size
> -----------------------------------------------
>
> Key: FLINK-36679
> URL: https://issues.apache.org/jira/browse/FLINK-36679
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Affects Versions: 1.18.1, 1.20.0, 1.19.1
> Reporter: Mingliang Liu
> Priority: Major
> Labels: pull-request-available
>
> Currently we expose multiple metrics for the checkpoint size. One specific
> interesting data point is the {{_metadata}} file size, which can also be
> added as a metric. The {{_metadata}} file has multiple types of data to store
> including operator states, coordinator states and properties. Its size should
> be scoped to a reasonable range, otherwise job may take too long to restore
> from checkpoints and/or fail to start when its size exceeding RPC frame limit.
> However, we saw multiple times the {{_metadata}} file bloats up to 100MB~3GB
> causing job slow and/or fail to start. In user mail list (for e.g.
> [[1]|https://lists.apache.org/thread/dttjs2v412xd7slrrx94837ch8wjfo11],
> [[2]|https://lists.apache.org/thread/yj66dnbs7xrmbspdltq3yfptccm25llt]) and
> FLINK-32658 community reported similar problems.
> Tracking the metadata size can be helpful for operations. Currently we have
> following use cases:
> * The first case was primarily informational. When a job fails to deploy
> because the payload exceeds the Akka frame size, it’s often unclear what
> caused the issue. By monitoring the metrics of the checkpoint _metadata file,
> the problem becomes more apparent. As a workaround to restore the job, users
> typically increase the Akka frame size config, or redeploy it from an empty
> state and rewind the Kafka broker’s committed offsets for the consumer group.
> * The second case involved a suspected bug in Kafka source v1, which occurs
> when the operator UID changes. In this scenario, the union state of the old
> operator grows exponentially with each savepoint, eventually resulting in a
> multiple GB _metadata file that exceeds the Akka frame size. We recommended
> that users use the State Processor API to rewrite the checkpoint file,
> omitting the problematic operator state for the obsolete operator. Although
> we haven’t identified the root cause, we advised users to maintain
> deterministic operator UIDs. In the long term, we plan to migrate to Kafka
> source v2 (FLIP-27).
> * The final case involved a _metadata file that was growing in size, but
> still contained valid data. However, the file became too large to allow for
> quick job restarts or rescaling. We advised the user to improve the
> serialization of the operator state if possible, or to minimize heavy usage
> of operator state to avoid such issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
