ZhiXiao-Lin opened a new issue, #4361:
URL: https://github.com/apache/iggy/issues/4361
## Summary
Restarting a single-node server (`docker restart`, i.e. process restart with
the data directory preserved) intermittently kills the shard during boot replay:
```
thread 'shard-0' panicked at core/consensus/src/client_table.rs:1129:9:
client_id 0 is reserved for internal use
ERROR shard-0 server::boot::threads: message pump died instead of draining
(task panicked: client_id 0 is reserved for internal use); committed journal
tail may not have flushed
Error: ShardJoinFailures { failures: [ShardJoinFailure { shard_id: 0, kind:
Error(ShardPumpDied { shard_id: 0, reason: "task panicked: client_id 0 is
reserved for internal use" }) }] }
```
The process exits (1) and cannot boot again with the same data directory.
Recreating the container (fresh data) boots clean. The panic is
state-dependent: the same workload sometimes restarts cleanly. Reproduced 4
times in one day of integration testing.
## Repro
Docker (macOS host / Docker Desktop; single-node, TCP only):
```bash
docker run -d --name iggy --security-opt seccomp=unconfined -p 5102:5102 \
-e IGGY_ROOT_USERNAME=iggy -e IGGY_ROOT_PASSWORD=iggy \
-e IGGY_TCP_ADDRESS=0.0.0.0:5102 -e IGGY_NODE_ADVERTISED_ADDRESS=127.0.0.1
\
-e IGGY_SHARDING_CPU_ALLOCATION=1 -e IGGY_SHARDING_PIN_CORES=false \
apache/iggy:0.9.0
```
Then, repeatedly (via the Rust SDK 0.11.0):
1. `login_user("iggy", "iggy")`
2. create a stream + topic, send a few messages
3. create/join a consumer group, poll a batch, `store_consumer_offset`
4. drop the client connection
5. `docker restart iggy`
Observed: every few cycles, boot replay panics as above and the server stays
down. In one reproduction the server died on its own boot with no explicit
restart command issued around that time.
## Suspicion
The persisted client table replays a session whose (reconstructed or
replayed) client id collides with the reserved internal id 0 — plausibly a
client that was mid-registration when the process stopped, or a session whose
id was never durably assigned before the kill. Boot treats the collision as a
panic instead of rejecting/ignoring the stale entry, turning a recoverable
restart into a hard outage requiring a manual data-dir reset.
## Impact
Any production single-node deployment that restarts (deploy, node bounce,
OOM kill) can become permanently unbootable with the same data directory.
Found while chaos-testing an integration against Iggy (broker restart
mid-stream with durable consumer offsets).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
