Sbaia commented on issue #17863: URL: https://github.com/apache/iceberg/issues/17863#issuecomment-5464953084
I checked Flink's SinkV2 API in 2.1.2 and current master. `SupportsCommitter` only exposes committer creation and serialization. `SupportsPreCommitTopology` controls the stream before the committer. The actual `CommitterOperator` is created afterward by Flink's `SinkTransformationTranslator`, which assigns it the sink transformation's parallelism and max parallelism when unset. Therefore, Iceberg currently has no public API through which `DynamicIcebergSink` can configure the generated committer transformation independently. I propose splitting the change: 1. Flink adds an optional SinkV2 capability exposing committer parallelism and max parallelism. 2. `SinkTransformationTranslator` applies these values to the generated `CommitterOperator`. 3. Iceberg adds `committerParallelism(...)` and `committerMaxParallelism(...)` to `DynamicIcebergSink.Builder`. 4. Empty values preserve the current inherited behaviour. This capability would provide a connector-level bound. The incorrect autoscaler target-rate calculation observed for the multi-input committer remains a separate Flink autoscaler issue. I will open the companion Flink issue and link it here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
