ryankert01 opened a new issue, #3710:
URL: https://github.com/apache/iggy/issues/3710
### Description
Two C++ e2e tests fail intermittently:
`LowLevelE2E_Client.GetClientsReflectsSessionRemovalAfterDisconnect` and
`...AfterShutdown`.
Across the last 60 `Pre-merge` runs, 14 failed. **11 of those failed on the
`C++ • e2e` job, and every one was one of these two tests** (5x Disconnect, 6x
Shutdown), 278/279 passing each time. They failed on 7 unrelated branches
(connectors, docker image, kafka gateway, metadata, SDK), so the trigger is
environmental rather than any of those changes.
```
low_level_e2e.cpp:439: Failure
Value of: found_first
Actual: true; Expected: false
```
### Root cause
Both tests disconnect one client and then, over a second connection, assert
**once immediately** that the session is already absent from `get_clients()`.
Nothing guarantees that ordering:
- `disconnect`/`shutdown` closes the socket without a command or ack
([tcp_client.rs#L675](https://github.com/apache/iggy/blob/master/core/sdk/src/tcp/tcp_client.rs#L675)).
- The server removes the session only when the first connection's read loop
observes EOF and runs `cleanup_connection`
([tcp_listener.rs#L159](https://github.com/apache/iggy/blob/master/core/server/src/tcp/tcp_listener.rs#L159)),
with no happens-before edge to the second connection's query.
The adjacent `GetClientsReflectsLoggedOutSessionAsUnauthenticated` does not
flake, because logout is a command with a response and is observable as soon as
the reply returns.
I could not reproduce it locally in 700 iterations against the Rust SDK,
consistent with a narrow timing window that widens on loaded CI runners.
### Possibly related (not confirmed)
`client_id` is `xxHash32` of the source `IP:port`
([client_manager.rs#L63](https://github.com/apache/iggy/blob/master/core/server/src/streaming/clients/client_manager.rs#L63)).
Since ephemeral ports are recycled by the OS, IDs are not unique over a
session lifecycle, and `insert` overwrites on collision. That could also
present as a removed session still appearing. Noting it as a lead only.
### Proposed fix
Make the two tests poll `get_clients()` until the session disappears, with a
short timeout, instead of asserting once. Server behaviour is unchanged.
### Affected area / component
C++ SDK, CI / build / tooling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
