leizhiyuan opened a new issue, #1448: URL: https://github.com/apache/rocketmq-clients/issues/1448
### Before Creating the Bug Report - [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq-clients/discussions). - [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq-clients/issues) and [GitHub Discussions](https://github.com/apache/rocketmq-clients/discussions) of this repository and believe that this is not a duplicate. - [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ. ### Programming Language of the Client Rust ### Runtime Platform Environment Linux 6.6 x86_64, 32 cores ### RocketMQ Version of the Client/Server Client: Rust `rocketmq` 5.0.0, master `e90ece7a` Server: a mock proxy that speaks the real gRPC protocol over TCP (see the benchmark in the tracking issue); the defects are also visible by reading the code against the Java client. ### Run or Compiler Version rustc 1.97.1 (also checked with 1.74.0, the MSRV) ### Describe the Bug Every RPC of every client gets its session through `SessionManager::get_or_create_session`, which locks one `tokio::sync::Mutex` over the whole session map ([client.rs#L1061](https://github.com/apache/rocketmq-clients/blob/e90ece7a/rust/src/client.rs#L1061), [#L1112](https://github.com/apache/rocketmq-clients/blob/e90ece7a/rust/src/client.rs#L1112)). The lock is held across network I/O: - creating the session of a new endpoint: `Session::new` connects ([session.rs#L215](https://github.com/apache/rocketmq-clients/blob/e90ece7a/rust/src/session.rs#L215), connect timeout = request timeout, 3 s by default), then `session.start` awaits the telemetry call ([client.rs#L1127](https://github.com/apache/rocketmq-clients/blob/e90ece7a/rust/src/client.rs#L1127), [session.rs#L396](https://github.com/apache/rocketmq-clients/blob/e90ece7a/rust/src/session.rs#L396)); - `sync_settings` awaits `update_settings` of every session in turn ([client.rs#L1240-L1243](https://github.com/apache/rocketmq-clients/blob/e90ece7a/rust/src/client.rs#L1240-L1243)), which may rebuild a telemetry stream; - `reconnect_session`, `clear_dead_and_idle_sessions` and `shutdown` lock it as well. So one slow or unreachable broker stalls send / receive / ack / heartbeat to **every** endpoint. A producer round-robins over the queues of all brokers, so it keeps hitting the slow one. Java looks clients and sessions up under a read lock and does nothing blocking under the write lock: `ClientManagerImpl#getRpcClient`, `ClientImpl#getClientSession` (Netty channels connect lazily, the telemetry call returns a `StreamObserver` right away), and `ClientImpl#syncSettings` writes to each session without holding the table lock. ### Steps to Reproduce 8 tasks send to a healthy broker in a loop; at t=2 s one send goes to a topic on a new broker whose telemetry call answers after 3 s. ### What Did You Expect to See? Sends to the healthy broker are not affected. ### What Did You See Instead? | healthy broker, during those 3 s | `master` | fix | |---|---|---| | sends completed | 47 | 59,966 | | send latency p99 | 3,002 ms | 0.62 ms | ### Additional Context Part of #1445. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
