wombatu-kun opened a new issue, #20311: URL: https://github.com/apache/hudi/issues/20311
### Bug Description **What happened:** If the topic partition lookup fails while the Kafka Connect transaction coordinator starts a commit, the coordinator never starts another one, and the sink silently stops ingesting until the coordinator is recreated by a task restart or by a rebalance that reassigns partition 0. `ConnectTransactionCoordinator.startNewCommit()` calls `getLatestNumPartitions` before its `try` block, so the `HoodieException` skips the method's own "log and retry START_COMMIT" handler and lands in the event loop's catch-all, which only logs a WARN. Nothing else schedules a new START_COMMIT, and every participant keeps its partition paused until it receives one, so the task stays RUNNING while consumer lag grows. In production the lookup is `KafkaConnectUtils.getLatestNumPartitions`, which wraps every failure into that `HoodieException`, for example when the brokers are briefly unreachable as a commit starts and `describeTopics` times out. **What you expected:** The coordinator retries after `RESTART_COMMIT_DELAY_MS`, as it already does when `startCommit()` fails, and commits resume once the lookup succeeds. **Steps to reproduce:** 1. Create a `ConnectTransactionCoordinator` whose `KafkaPartitionProvider` throws a `HoodieException` on the first call and returns the partition count afterwards. 2. Call `start()`. 3. START_COMMIT is never published: the coordinator logs the WARN below once and then sits idle. ### Environment **Hudi version:** master (1.3.0-SNAPSHOT); the lookup has been outside the `try` since the Kafka Connect sink was introduced **Query engine:** Kafka Connect (Hudi sink) **Relevant configs:** defaults ### Logs and Stack Trace From the reproduction above: ``` WARN org.apache.hudi.connect.transaction.ConnectTransactionCoordinator [] - Error received while polling the event loop in Partition Coordinator org.apache.hudi.exception.HoodieException: Failed to fetch the latest number of partitions ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
