gmr opened a new issue, #4273:
URL: https://github.com/apache/iggy/issues/4273
### Bug description
A consumer group that existed before a `0.9.0-edge.6` → `0.9.0-edge.8`
server upgrade stopped consuming at the moment of the upgrade and never
recovered. A member belonging to a process that terminated during the upgrade
kept ownership of the group's only partition for six days and across a full
server restart. Live members that joined afterwards were listed in the group
with zero partitions and polled indefinitely, receiving no messages and no
client-visible error.
Consumer groups created *after* the upgrade are unaffected, including four
other topics in the same stream drained by the same consumer process.
### Environment
- Server: `0.9.0-edge.8`, single node, TCP, one partition per topic,
upgraded in place from `0.9.0-edge.6`
- Consumer: `iggy-connectors` runtime with the ClickHouse sink plugin, same
build
- Storage: local disk, no purge or retention pressure (topic held 2.5 GB
across 3 segments)
### Timeline
| Time (UTC) | Event |
|---|---|
| 2026-09-17T21:34:32 | StatefulSet rolls `0.9.0-edge.6-0` →
`0.9.0-edge.8-0`. The consumer process is terminated by the rollout. |
| 2026-09-17T21:34:28 | Last message this group ever committed. Its stored
offset freezes at `156292`. |
| 2026-09-17 → 09-23 | Producers keep writing. The topic grows to 2.5 GB,
roughly 120k uncommitted messages. The consumer is restarted several times;
each restart joins the group and consumes nothing. |
| 2026-09-23T14:06 | Full server restart. The stale member and its partition
assignment are still present afterwards. |
| 2026-09-23T14:16 | Group deleted and recreated. Consumption resumes
immediately. |
### Observed state
`consumer-group get`, after three consumer restarts:
```
| Consumer group id | 0 |
| Consumer group name | events |
| Partitions count | 1 |
| Members count | 3 |
| Members | Member id | Partitions count | Partitions |
| | 0 | 1 | 0 |
| | 1 | 0 | |
| | 2 | 0 | |
```
Member `0` belongs to the process terminated on 09-17. Members `1` and `2`
are live consumers. Each consumer restart added a member; none was ever
removed, and the assignment never moved.
With one partition and three members this is a *balanced* assignment, so the
distribution logic is not obviously at fault. The problem is that member `0`
stayed eligible after its client went away.
The server logged the fencing path exactly once, when the previous
consumer's client was still attached:
```
WARN shard-1 server::dispatch::partition: poll_messages fenced; replying
re-sync sentinel
transport_client_id=... error=Consumer group member with client ID: ...
does not own
partition: 0 at the current generation (rebalance in progress).
```
After that the live members polled continuously with no server-side warning
and no client-side error — just empty results. On the client side this is
indistinguishable from an idle topic, which is why it went unnoticed for six
days.
### What is not the cause
- **Storage and offset are healthy.** `message poll --offset 156293 -m 1
<stream> <topic> 0` returns the message.
- **Not the sink or the client process.** The same runtime, same process,
same TCP client concurrently drained four other topics in the same stream plus
six other streams without interruption.
- **Not a server restart artifact.** A full restart did not clear the stale
membership.
### Expected behavior
A member whose client connection is gone should be evicted from the group,
and its partitions reassigned to a live member.
### Actual behavior
The stale member retains the partition. New members join, own nothing, and
poll empty forever with no error surfaced to the client.
### Workaround
Delete and recreate the consumer group:
```
iggy consumer-group delete <stream> <topic> <group>
iggy consumer-group create -g 0 <stream> <topic> <group>
```
Consumption resumed immediately. Two caveats for anyone hitting this:
1. Despite passing `-g 0`, the recreated group was assigned id `1`.
2. The delete removed the stored offset
(`.../partitions/0/offsets/groups/0`), so the consumer restarted from the
oldest retained offset and re-delivered roughly 120k already-processed
messages. That was harmless for us because the sink target is idempotent, but a
non-idempotent consumer would double-process.
### Notes
- The failure began at the same moment as the `edge.6` → `edge.8` upgrade. I
have not bisected or reproduced on a downgrade, so I can't state that the
upgrade *introduced* it as opposed to exposing a pre-existing path; the
correlation is exact to the second.
- I have not determined whether this is specific to the
terminate-during-upgrade sequence or applies to any ungraceful client
disconnect.
Suggested reproduction to separate an upgrade-migration defect from ordinary
stale-member cleanup:
1. On `edge.6`, create a single-partition topic and a consumer group;
consume and commit an offset.
2. Upgrade the server to `edge.8`, terminating the consumer without a
graceful leave.
3. Start a replacement consumer with a different client id.
4. Restart the server.
5. Check whether the old member still owns the partition.
Relevant code paths, from a read of the repo rather than a debugger:
`core/server/src/dispatch/partition.rs` (the fencing log) and
`core/metadata/src/stm/consumer_group.rs` (membership and rebalancing).
Related but distinct: #3334 covers cooperative rebalance *distribution*;
this is about a member that should no longer be eligible at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
