almostintuitive commented on issue #4273: URL: https://github.com/apache/iggy/issues/4273#issuecomment-5818309071
Another trigger for the same stale member, this time in a 3-node cluster and fully deterministic: **kill the replica that holds the member's connection**. When the node that owns a client's TCP connection dies, nothing submits that client's disconnect `Logout`. The surviving replicas still hold the session in the replicated client table, so the member keeps its partitions. The SDK reconnects to a survivor under a new client id, rejoins, and gets only what is left. A brand-new consumer is in the same position. Restarting the killed node does not remove the member either. Reproduces 2/2 runs of [this repro](https://gist.github.com/almostintuitive/cbff837fce0709ccaa209d1842e957f0) on `apache/iggy:0.9.0` (`sha256:b35fc284…`, Go SDK `v0.9.0`) and 2/2 on `apache/iggy:edge` (`sha256:8cc2475a…`, image revision `85a397d0`, Go SDK `v0.9.1-0.20260924080616-85a397d0e5fd`). Both images report `server 0.9.0`. ### Repro The [repro](https://gist.github.com/almostintuitive/cbff837fce0709ccaa209d1842e957f0) (`cluster.sh`, `run.sh`, `main.go`) depends only on the Docker image and the Go SDK. `cluster.sh` starts a 3-node VSR cluster on a bridge network at `172.29.0.10-12`, and `run.sh` wraps it: ```bash ./run.sh stale-member apache/iggy:edge 85a397d0e5fd36262a3bbc203165c7a5e01b9029 ``` Steps (`main.go`, `staleMember`): 1. Create a topic with 3 partitions and group `g`. 2. Connect a consumer to the metadata leader, join `g`, and sync. It gets `[0 1 2]`. 3. `docker kill` the leader. The SDK reconnects to a survivor, sync returns `member not found`, and the consumer rejoins. 4. Watch the group for 2 minutes, join a brand-new consumer, then `docker start` the killed node and wait 40 s. Output on edge: ``` [ 1.1s] metadata leader: iggy-0 (172.29.0.10:8090) [ 1.2s] consumer joined: sync generation=1 partitions=[0 1 2] [ 1.2s] group: members=1 [{ID:0 PartitionsCount:3 Partitions:[0 1 2]}] [ 1.2s] docker kill iggy-0 [ 6.4s] consumer sync after kill: error: consumer group member with id: 0 for group with id: 0 for topic with id: 0 was not found. (now via 172.29.0.11:8090) [ 6.4s] consumer rejoined [ 6.5s] consumer sync after rejoin: generation=2 partitions=[2] [ 8.1s] group: members=2 [{ID:0 PartitionsCount:2 Partitions:[0 1]} {ID:1 PartitionsCount:1 Partitions:[2]}] | consumer sync: generation=2 partitions=[2] ... unchanged every 15 s until ... [ 128.1s] group: members=2 [{ID:0 PartitionsCount:2 Partitions:[0 1]} {ID:1 PartitionsCount:1 Partitions:[2]}] | consumer sync: generation=2 partitions=[2] [ 128.6s] a brand-new consumer joined: sync generation=3 partitions=[1] [ 128.6s] docker start iggy-0 [ 169.1s] after the killed node restarted (40s): group: members=2 [{ID:0 PartitionsCount:2 Partitions:[0 2]} {ID:1 PartitionsCount:1 Partitions:[1]}] | consumer sync: generation=4 partitions=[1] ``` Member `0` belongs to the dead connection. It outlives `consumer_group.rebalancing_timeout` (30 s) and the restart of its node. Each rebalance still hands it partitions: first `[0 1]`, then `[0 2]`. A clean disconnect on a live node does not leak: with the same cluster and SDK, a client whose connection is closed while its node stays up is removed immediately (the group drops to `members=0`). ### Where it seems to come from (hypothesis, read at `85a397d0`) This matches your analysis in the report, with one more path that never submits the `Logout`: - The only runtime removal is the `Logout` apply in `core/metadata/src/impls/metadata.rs:645-659`. It calls `remove_consumer_group_member`, which is `core/metadata/src/stm/stream.rs:1561`. - That `Logout` is submitted by the replica that owned the transport (`submit_disconnect_logout`, `core/server/src/dispatch/session_ops.rs:1039`, called from `:476` and `core/server/src/dispatch/mod.rs:188`). If that replica is killed, no survivor ever submits it. - `Register` is replicated, so the survivors keep the dead client's session in their client table. A cleanup keyed on "client id not in the client table" would not catch this case. A liveness or heartbeat-based expiry would. I saw `fix/4273-stale-consumer-group-members` (`23492f133`), with heartbeat reports on the metadata primary and expiry via `Logout`. I have not run this repro against it. If there's an image from that branch, `./run.sh stale-member <image>` checks it: exit code 1 means the member is still there after the restart, 0 means it was removed. The replica that owned the dead connection is exactly the one that stops reporting, so this case exercises the "a crashed node must stop deferring expiry" path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
