gmr commented on issue #4273: URL: https://github.com/apache/iggy/issues/4273#issuecomment-5799523893
Follow-up with more data from the same deployment. This answers the open question in the report: the problem is **not specific to the edge.6 → edge.8 upgrade**. It also happens on edge.8 to groups that were created after the upgrade. ### What happened after the workaround After the 14:06 server restart, 22 other groups (one per topic, 15 streams, all single-partition) consumed normally from 14:08 to 14:15. At 14:15:17 the connectors runtime closed its sinks, and it reopened them at 14:16:16. From then on, none of those 22 groups delivered a message. The server was **not** restarted at that point. With the connectors runtime scaled to 0, so no consumer client was connected, every one of the 22 groups still listed one member that owned partition 0: ``` operations_log/deployments group=operations_log members=1 owners: m1=[0] commits/github group=commits members=1 owners: m1=[0] tags/github group=tags members=1 owners: m1=[0] ... (22 groups, same shape) ``` While the runtime was up, each of these groups had `m0=[]` (the live runtime) and `m1=[0]` (the leftover member). The live member disconnected cleanly (it was gone from every group after the scale-down), but `m1` stayed. A later full server restart (16:46) did not remove `m1` either. Not every disconnect leaks. A later scale-down/scale-up of the same runtime, with the server up, left no extra members. I don't yet know what separates the leaking disconnect from the clean ones. ### Why the member is never removed (from reading `e0027506`) - A member leaves a group only as a side effect of a committed `Logout`: `core/metadata/src/impls/metadata.rs:655` (commit path) and `:2893` (replica path), and `on_replayed_logout` in `core/metadata/src/impls/recovery.rs:628` (WAL replay). - The heartbeat verifier (`core/server/src/dispatch/session_ops.rs:428`) iterates `SessionManager::collect_stale`, so it only considers connections that still exist. A membership whose client has no connection on this node is never a candidate for eviction. - Membership is part of the replicated metadata state, so it is restored at boot (`core/server/src/boot/mod.rs:537` only removes members for logouts found in the WAL). So a restart cannot clear it. So if a disconnect's `Logout` is never committed, for any reason, nothing in the server removes that member afterwards. A possible fix is to drop, at boot and periodically, any group member whose client id is not in the client table. ### Workaround notes (in addition to the ones above) - Group offsets cannot be read over HTTP: `Consumer.kind` is `#[serde(skip)]` (`core/common/src/types/consumer/consumer_kind.rs:34`), so `GET .../consumer-offsets` always resolves a plain consumer. This made it hard to check the stored offset before deleting a group. - The connectors runtime recreates a deleted group on start (`create_consumer_group_if_not_exists`) and, with `PollingStrategy::next()` and no stored offset, replays from offset 0 (`core/partitions/src/iggy_partition.rs:3500`). We deleted the 22 groups with the runtime scaled to 0 and deduplicated the non-idempotent targets afterwards. ### Environment Same as the report: `0.9.0-edge.8`, single node, TCP, `iggy-connectors` `0.5.0-edge.6` with the ClickHouse sink `0.2.0-edge.4`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
