[
https://issues.apache.org/jira/browse/FLINK-39213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanquan Lv updated FLINK-39213:
-------------------------------
Description:
Currently, we might see such commit content:
{code:java}
Identifier{database='testDb', object='testTable'}=ManifestCommittable {
identifier = 34056,
watermark = 34056,
logOffsets = {},
commitMessages = [
FileCommittable {partition = org.apache.paimon.data.BinaryRow@9c67b85d,
bucket = 0, totalBuckets = -1, newFilesIncrement = DataIncrement
{newFiles = [], ...}, ...},
FileCommittable {partition = org.apache.paimon.data.BinaryRow@9c67b85d,
bucket = 1, totalBuckets = -1, newFilesIncrement = DataIncrement {newFiles =
[], ...}
, ...},
FileCommittable {partition = org.apache.paimon.data.BinaryRow@9c67b85d,
bucket = 2, totalBuckets = -1, newFilesIncrement = DataIncrement
{newFiles = [], ...}, ...},
FileCommittable {partition = org.apache.paimon.data.BinaryRow@9c67b85d,
bucket = 3, totalBuckets = -1, newFilesIncrement = DataIncrement {newFiles =
[], ...}
, ...},
FileCommittable {partition = org.apache.paimon.data.BinaryRow@9c67b85d,
bucket = 4, totalBuckets = -1, newFilesIncrement = DataIncrement
{newFiles = [], ...}, ...},
FileCommittable {partition = org.apache.paimon.data.BinaryRow@9c67b85d,
bucket = 5, totalBuckets = -1, newFilesIncrement = DataIncrement {newFiles =
[], ...}
, ...},
FileCommittable {partition = org.apache.paimon.data.BinaryRow@9c67b85d,
bucket = 134, totalBuckets = -1, newFilesIncrement = DataIncrement
{newFiles = [], ...}
, ...},
...{code}
When no data is written, we should avoid creating an empty commit to prevent an
increase in the number of useless metadata files.
was:
Currently, we might see such commit content:
```
Identifier\{database='testDb', object='testTable'}=ManifestCommittable {
identifier = 34056,
watermark = 34056,
logOffsets = {},
commitMessages = [
FileCommittable \{partition =
org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 0, totalBuckets = -1,
newFilesIncrement = DataIncrement {newFiles = [], ...}, ...},
FileCommittable \{partition =
org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 1, totalBuckets = -1,
newFilesIncrement = DataIncrement {newFiles = [], ...}, ...},
FileCommittable \{partition =
org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 2, totalBuckets = -1,
newFilesIncrement = DataIncrement {newFiles = [], ...}, ...},
FileCommittable \{partition =
org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 3, totalBuckets = -1,
newFilesIncrement = DataIncrement {newFiles = [], ...}, ...},
FileCommittable \{partition =
org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 4, totalBuckets = -1,
newFilesIncrement = DataIncrement {newFiles = [], ...}, ...},
FileCommittable \{partition =
org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 5, totalBuckets = -1,
newFilesIncrement = DataIncrement {newFiles = [], ...}, ...},
FileCommittable \{partition =
org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 134, totalBuckets = -1,
newFilesIncrement = DataIncrement {newFiles = [], ...}, ...},
FileCommittable {partition = org.apache.paimon.data.BinaryRow@9c67b85d,
bucket = 6, totalBuckets = -1, newFilesIncrement = DataIncrement
```
When no data is written, we should avoid creating an empty commit to prevent an
increase in the number of useless metadata files.
> Avoid creating an empty commit when no data is written in Paimon Sink
> ---------------------------------------------------------------------
>
> Key: FLINK-39213
> URL: https://issues.apache.org/jira/browse/FLINK-39213
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.2.0, cdc-3.3.0, cdc-3.2.1, cdc-3.4.0, cdc-3.5.0
> Reporter: Yanquan Lv
> Priority: Major
>
> Currently, we might see such commit content:
> {code:java}
> Identifier{database='testDb', object='testTable'}=ManifestCommittable {
> identifier = 34056,
> watermark = 34056,
> logOffsets = {},
> commitMessages = [
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 0, totalBuckets = -1,
> newFilesIncrement = DataIncrement
> {newFiles = [], ...}, ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 1, totalBuckets = -1,
> newFilesIncrement = DataIncrement {newFiles = [], ...}
> , ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 2, totalBuckets = -1,
> newFilesIncrement = DataIncrement
> {newFiles = [], ...}, ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 3, totalBuckets = -1,
> newFilesIncrement = DataIncrement {newFiles = [], ...}
> , ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 4, totalBuckets = -1,
> newFilesIncrement = DataIncrement
> {newFiles = [], ...}, ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 5, totalBuckets = -1,
> newFilesIncrement = DataIncrement {newFiles = [], ...}
> , ...},
> FileCommittable {partition =
> org.apache.paimon.data.BinaryRow@9c67b85d, bucket = 134, totalBuckets = -1,
> newFilesIncrement = DataIncrement
> {newFiles = [], ...}
> , ...},
> ...{code}
> When no data is written, we should avoid creating an empty commit to prevent
> an increase in the number of useless metadata files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
