jun-su-lee opened a new issue, #17811:
URL: https://github.com/apache/iceberg/issues/17811
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Kafka Connect
### Please describe the bug 🐞
A Kafka Connect sink task that holds partitions but receives no records never
answers the coordinator's commit requests, so every commit round stalls.
### Symptom
- Commit latency is pinned to `iceberg.control.commit.timeout-ms` instead of
completing as soon as the writers report.
- Each round falls back to a partial commit, and partial commits never set
the
`kafka.connect.valid-through-ts` snapshot property, so the watermark stops
advancing.
### Root cause
The worker is the task's side of the commit protocol — it reports every
assigned
partition on a commit request (with a null offset for partitions that got no
data). That code is correct, but nothing reached it: the worker's lifetime
was
tied to record arrival rather than to the partition assignment.
`CommitterImpl#save`
created the worker only inside the branch guarded by
`sinkRecords != null && !sinkRecords.isEmpty()`, so an idle task never had a
worker at all.
`close` also stops the worker on every rebalance, and a cooperative rebalance
that only revokes partitions is not followed by `open`, so a task can keep
partitions while its worker stays gone until the next record arrives.
### Proposed fix
Start the worker whenever the task holds an assignment (`ownsPartitions()`),
driven from `save` where `processControlEvents` polls the control topic.
A fix with tests is up in #17593.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
