[
https://issues.apache.org/jira/browse/FLINK-31809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fan Hong updated FLINK-31809:
-----------------------------
Summary: Improve the efficiency of ListStateWithCache#snapshotState (was:
Improve efficiency of ListStateWithCache#snapshotState)
> Improve the efficiency of ListStateWithCache#snapshotState
> ----------------------------------------------------------
>
> Key: FLINK-31809
> URL: https://issues.apache.org/jira/browse/FLINK-31809
> Project: Flink
> Issue Type: Improvement
> Components: Library / Machine Learning
> Reporter: Fan Hong
> Priority: Major
>
> In the current implementation of {{{}ListStateWithCache{}}}, the
> {{snapshotState}} function writes the full data to the file system every
> time, even if the stored data has not changed since initialization. This can
> result in high IO costs, especially when working with large data sets.
> Additionally, this method is called in the same thread as operators, which
> can negatively impact job efficiency.
> Furthermore, when using local file systems, the full data is also written to
> Flink state storage, which doubles the costs.
> To address these issues, an incremental snapshot approach should be
> considered to reduce IO and network costs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
