[
https://issues.apache.org/jira/browse/FLINK-40651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanquan Lv updated FLINK-40651:
-------------------------------
Description:
h2. Description
The Flink CDC Iceberg sink currently hard-codes the target file size to 256 MB
and does not pass Iceberg write properties to {{{}RowDataTaskWriterFactory{}}}.
As a result, configurations such as:
table.properties.write.target-file-size-bytes: "4194304"
table.properties.write.parquet.row-group-size-bytes: "4194304"
do not affect runtime file rolling as expected.
h2. Expected Behavior
* Honor {{write.target-file-size-bytes}} when creating the Iceberg writer.
* Pass Iceberg/Parquet write properties to the underlying writer.
* Allow multiple files to be rolled within one checkpoint while committing
them only after the checkpoint succeeds.
* Optionally support a connector-specific record-count rolling option.
h2. Acceptance Criteria
* Multiple files can be generated within one checkpoint when the configured
target size is reached.
* Exactly-once checkpoint semantics are preserved.
* Existing behavior remains unchanged when no related configuration is
provided.
was:
h2. Description
The Flink CDC Iceberg sink currently hard-codes the target file size to 256 MB
and does not pass Iceberg write properties to {{{}RowDataTaskWriterFactory{}}}.
As a result, configurations such as:
{{table.properties.write.target-file-size-bytes: "4194304"
table.properties.write.parquet.row-group-size-bytes: "4194304"}}
do not affect runtime file rolling as expected.
h2. Expected Behavior
* Honor {{write.target-file-size-bytes}} when creating the Iceberg writer.
* Pass Iceberg/Parquet write properties to the underlying writer.
* Allow multiple files to be rolled within one checkpoint while committing
them only after the checkpoint succeeds.
* Optionally support a connector-specific record-count rolling option.
h2. Acceptance Criteria
* Multiple files can be generated within one checkpoint when the configured
target size is reached.
* Exactly-once checkpoint semantics are preserved.
* Existing behavior remains unchanged when no related configuration is
provided.
> Support configurable Iceberg file rolling within a checkpoint
> -------------------------------------------------------------
>
> Key: FLINK-40651
> URL: https://issues.apache.org/jira/browse/FLINK-40651
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Reporter: Yanquan Lv
> Priority: Major
>
> h2. Description
> The Flink CDC Iceberg sink currently hard-codes the target file size to 256
> MB and does not pass Iceberg write properties to
> {{{}RowDataTaskWriterFactory{}}}.
> As a result, configurations such as:
> table.properties.write.target-file-size-bytes: "4194304"
> table.properties.write.parquet.row-group-size-bytes: "4194304"
> do not affect runtime file rolling as expected.
> h2. Expected Behavior
> * Honor {{write.target-file-size-bytes}} when creating the Iceberg writer.
> * Pass Iceberg/Parquet write properties to the underlying writer.
> * Allow multiple files to be rolled within one checkpoint while committing
> them only after the checkpoint succeeds.
> * Optionally support a connector-specific record-count rolling option.
> h2. Acceptance Criteria
> * Multiple files can be generated within one checkpoint when the configured
> target size is reached.
> * Exactly-once checkpoint semantics are preserved.
> * Existing behavior remains unchanged when no related configuration is
> provided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
