[
https://issues.apache.org/jira/browse/FLINK-39213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18063138#comment-18063138
]
Yanquan Lv commented on FLINK-39213:
------------------------------------
I would like to take it.
> Avoid creating an empty commit when no data is written in Paimon Sink
> ---------------------------------------------------------------------
>
> Key: FLINK-39213
> URL: https://issues.apache.org/jira/browse/FLINK-39213
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.2.0, cdc-3.3.0, cdc-3.2.1, cdc-3.4.0, cdc-3.5.0
> Reporter: Yanquan Lv
> Priority: Major
>
> Currently, we might see such commit content:
> {code:java}
> Identifier{database='testDb', object='testTable'}=ManifestCommittable {
> identifier = 34056,
> watermark = 34056,
> logOffsets = {},
> commitMessages = [
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 0, totalBuckets = -1,
> newFilesIncrement = DataIncrement
> {newFiles = [], ...}, ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 1, totalBuckets = -1,
> newFilesIncrement = DataIncrement {newFiles = [], ...}
> , ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 2, totalBuckets = -1,
> newFilesIncrement = DataIncrement
> {newFiles = [], ...}, ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 3, totalBuckets = -1,
> newFilesIncrement = DataIncrement {newFiles = [], ...}
> , ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 4, totalBuckets = -1,
> newFilesIncrement = DataIncrement
> {newFiles = [], ...}, ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 5, totalBuckets = -1,
> newFilesIncrement = DataIncrement {newFiles = [], ...}
> , ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 134, totalBuckets = -1,
> newFilesIncrement = DataIncrement
> {newFiles = [], ...}
> , ...},
> ...{code}
> When no data is written, we should avoid creating an empty commit to prevent
> an increase in the number of useless metadata files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
