tongwai-wong-appier commented on issue #16016: URL: https://github.com/apache/iceberg/issues/16016#issuecomment-4956202968
We hit the same coordinator leak on **1.11.0**. Our logs directly confirm the second leak path @gdmachado described (`close()` skipping `stopCoordinator()` during a rebalance). Sharing as an independent confirmation with a few new data points. ### Environment - Iceberg 1.11.0, JDBC catalog, GCS. - Reproduced with a **single Connect worker** — not a multi-worker race. - **No storage failure involved** — unlike the original report, an ordinary consumer-group rebalance is enough. ### Log evidence for the `close()` gate failing One connector, ~8s window during a rebalance: - `task-1` (the coordinator) entered `close()` — logged `Seeking to last committed offsets for worker ...` ([`CommitterImpl.java:185`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java#L185)), so `close()` ran to completion. - Same thread logged `found no partitions assigned across all members, cannot determine leader` ([`CommitterImpl.java:101`](https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/kafka-connect/kafka-connect/src/main/java/org/apache/iceberg/connect/channel/CommitterImpl.java#L101)) — mid-rebalance the Admin snapshot had no partitions assigned, so `findFirstTopicPartition()` returned null and `hasLeaderPartition()` returned false. - **No** `Stopping coordinator.` line after that — `stopCoordinator()` was never called. - ~4s later `task-0` logged `received leader partition. Starting Coordinator.` → two live coordinators. So the check fails open: an empty-assignment snapshot is treated as "not the leader", leaving the coordinator running exactly when leadership can't be determined. This code is unchanged on current `main`. ### A different zombie presentation Here the task stays alive, so the leaked coordinator remains fully functional: - It keeps its commit cycle, and stays in the `<connect-group>-coord` consumer group (second member with `#PARTITIONS = 0`). - It periodically commits its **frozen in-memory offsets** back to the group — the committed offset oscillates between the active coordinator's advancing value and a fixed stale value, firing consumer-lag alerts even though the active coordinator is healthy. - Restarting the worker clears it; it recurs on a later rebalance. ### Fix direction +1 to [gdmachado's suggestion](https://github.com/apache/iceberg/issues/16016#issuecomment-4587580400): remember the leader partition at `open()` time and stop the coordinator in `close()` when it's in `closedPartitions` (plus the join from #16020). We'd be happy to help get a PR for this over the line. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
