kuldeep0508 opened a new issue, #17193:
URL: https://github.com/apache/iceberg/issues/17193
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Kafka Connect
### Please describe the bug 🐞
### Summary
After upgrading from Iceberg 1.10.1 to 1.11.0, the Kafka Connect Iceberg
Sink connector's `cg-control` consumer group cannot reach STABLE state when the
data topic goes idle after initial consumption. This causes all commits to
timeout with "committed to 0 table(s)".
The regression was introduced by PR #14395 ("Don't check that consumer group
is stable for coordinator leader election"), which removed the stability check
that previously ensured the Worker's `cg-control` consumer had joined before
the Coordinator started firing commits.
### Root Cause
In 1.10.1, the Coordinator waited for the `cg-control` consumer group to be
STABLE before starting commit cycles. In 1.11.0, this check was removed. The
unintended consequence:
1. Connector starts → consumes all available messages in the first `poll()`
call
2. Data topic now has no new messages → next `poll()` blocks for
`max.poll.interval.ms` (default 60s)
3. During this 60s block, the `cg-control` consumer's `poll()` is not called
(single-threaded task loop)
4. `cg-control` session timeout is 45s (default `session.timeout.ms`)
5. After 45s with no heartbeat → broker kicks `cg-control` member →
`UnknownMemberIdException`
6. `cg-control` can never reach STABLE → Worker never processes START_COMMIT
→ commit always times out
The 45s session timeout < 60s poll block creates a guaranteed failure
condition whenever the data topic goes idle.
### Steps to Reproduce
1. Deploy Iceberg Sink Connector 1.11.0 in distributed mode with
`tasks.max=1`
2. Have a data topic with a finite set of messages (e.g., a CDC snapshot —
no continuous flow)
3. Configure `iceberg.control.commit.interval-ms: 60000` (or any value >
session timeout)
4. Start the connector from a clean state (no internal topics)
5. Connector consumes all messages in first poll
6. Topic goes idle → `cg-control` consumer can never join → all commits
timeout with 0 tables
### Expected Behavior
The `cg-control` consumer group should reach STABLE state regardless of data
topic activity. Either:
- The Coordinator should wait for `cg-control` stability before firing the
first commit (as in 1.10.1)
- Or the `cg-control` consumer's `poll()` should be decoupled from the data
consumer's poll loop (separate thread)
### Actual Behavior
```
[TRACE] Polling returned 454 messages ← All consumed in first poll
[TRACE] Delivering batch of 454 messages ← Delivered to put()
... 60s passes (data poll blocks) ...
[INFO] Commit timeout reached → committed to 0 table(s)
[INFO] [cg-control-*] Request joining group due to: rebalance failed due to
'The coordinator is not aware of this member.'
(UnknownMemberIdException)
```
The `cg-control` consumer repeatedly fails to join:
```
sessionTimeoutMs=45000 ← Broker kicks after 45s
rebalanceTimeoutMs=300000
```
But the main task loop only iterates every 60s (poll timeout), so
`cg-control` gets only one `poll()` call per 60s — insufficient to maintain the
45s session.
### Comparison: 1.10.1 vs 1.11.0
| Version | `cg-control` State | Commits Work? | Behavior |
|---------|-------------------|---------------|----------|
| 1.10.1 | STABLE ✅ | Yes | Coordinator waits for stability |
| 1.11.0 | EMPTY ❌ | No | Coordinator fires immediately |
Tested with identical connector config on the same Kafka cluster. The only
difference is the Iceberg plugin version.
### Workaround
Setting `iceberg.kafka.session.timeout.ms` to a value greater than the data
consumer's poll timeout:
```json
"iceberg.kafka.session.timeout.ms": "120000",
"iceberg.kafka.heartbeat.interval.ms": "10000"
```
This passes through to the `cg-control` consumer via the `iceberg.kafka.*`
prefix, preventing the session from expiring during the idle poll block.
Alternatively, ensuring continuous data flow keeps the poll loop short,
allowing `cg-control` to join — but this is not a viable solution for
batch/snapshot workloads.
### Related Issues
- [#11818 ](https://github.com/apache/iceberg/issues/11818)— "Iceberg Kafka
Connector experiences a constant hanging lag for low-volume topics" (same
symptom, reported on 1.7.1, closed as stale)
- [#14395](https://github.com/apache/iceberg/pull/14395)— The PR that
introduced this regression
### Environment
- Kafka Connect: 3.9.2 (distributed mode)
- Kafka Broker: Amazon MSK 3.9.2
- Iceberg Sink: 1.11.0 (built from `apache-iceberg-1.11.0` tag)
- `tasks.max`: 1
- `iceberg.control.commit.interval-ms`: 60000
- Data topic: 454 messages (CDC snapshot, no continuous flow)
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
