Guosmilesmile opened a new pull request, #17251: URL: https://github.com/apache/iceberg/pull/17251
Closes https://github.com/apache/iceberg/issues/17140 `IcebergSink#addPreCommitTopology` uses a `.global()` shuffle to force all committables to subtask 0 of the downstream committer operator, so that commits only happen on a single subtask. Even though only subtask 0 will ever receive a `CommitRequest`, every committer subtask still eagerly loads the Iceberg table and creates a worker thread pool in its constructor. This pr add code to avoid unnecessary TableLoader.loadTable() in IcebergCommitter subtasks > 0 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
