pvary commented on issue #17140: URL: https://github.com/apache/iceberg/issues/17140#issuecomment-4923999787
> Hi Krish! There should only be one committer instance per Flink job. If that’s not the case, we need to fix this in Flink. I'd prefer not to do a lazy init to fail fast before the first commit. I remember a discussion about this on the Flink dev list: https://lists.apache.org/thread/82bgvlton9olb591bfg2djv0cshj1bxj I don't remember that there were a solution for that in Flink. So basically we instantiate multiple committers, but only use a single one (IcebergSink.addPreCommitTopology). AI suggests that something like this should be a solution: ``` @Override public void addPostCommitTopology(DataStream<CommittableMessage<CommT>> committables) { // Force Flink to funnel all committables into a single worker thread (Parallelism = 1) StandardSinkTopologies.addGlobalCommitter( committables, ContextualCommitterAdapter::new, // Your committer provider GlobalCommittableSerializer::new // Serializer for the aggregated committables ); } ``` I'm not sure how this behaves along with the current topology -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
