bzs1118 opened a new issue, #3479: URL: https://github.com/apache/brpc/issues/3479
**Describe the bug** When a brpc server is started with `SOCKET_MODE_RDMA`, the server-side RDMA socket's _read_buf can be concurrently accessed by two independent bthreads, causing IOBuf internal state corruption and intermittent core dumps. There are three race scenarios, all stemming from the same root cause: `PollCq`(CQ socket bthread) and `OnNewMessages`(main socket bthread) can run concurrently and both operate on the same Socket::_read_buf, which is not thread-safe. **Scenario 1: After RDMA handshake is established** RdmaTransport::Init sets the server-side edge trigger to `InputMessenger::OnNewMessages` (to drive the handshake via the standard InputMessenger path). After the handshake reaches ESTABLISHED, the edge trigger is never switched. If the TCP fd becomes readable (e.g., TCP keepalive, peer close), OnNewMessages is called and reads from the TCP fd into _read_buf. At the same time, PollCq processes RDMA completions and writes RDMA data into _read_buf via HandleCompletion. Both bthreads modify _read_buf concurrently → IOBuf corruption → core dump in IOBuf::cutn or IOBuf::clear. **Scenario 2: During RDMA handshake (after QP is brought up)** In ExecuteServerHandshake Phase 1, `BringUpQp`puts the QP into RTS state. The client may complete its handshake and start sending RDMA data before the server's handshake finishes. PollCq receives the RDMA completion and HandleCompletion writes to _read_buf. Meanwhile, the server's OnNewMessages is still driving the handshake (Phase 2: draining the 4-byte ACK) and accessing _read_buf via source->cutn(). Concurrent access → core dump in IOBuf::cutn. **Scenario 3: Handshake ACK rejection** ExecuteServerHandshake Phase 2 checks source->size() > HELLO_ACK_LEN and rejects the connection. When a client falls back to TCP, it sends the 4-byte ACK followed immediately by the first baidu_std RPC request. If both arrive before the server's readv(), they are read into _read_buf in one call. The server sees source->size() > 4 and drops the connection. This causes mass handshake failures for all TCP-fallback clients. **To Reproduce** 1. Start a brpc server with ServerOptions.socket_mode = `SOCKET_MODE_RDMA` 2. Have RDMA clients connect; some will negotiate RDMA successfully, others may fall back to TCP 3. After RDMA is established, if the TCP fd becomes readable (keepalive, peer close, etc.), the race triggers 4. Core dump occurs intermittently (~20% probability at startup, sporadically during runtime) 5. Stack trace shows crash in `IOBuf::clear()` or `IOBuf::cutn()` during `Socket::BeforeRecycled()` or `ExecuteServerHandshake` **Expected behavior** No core dump. _read_buf should not be accessed concurrently by PollCq and OnNewMessages. **Versions** OS: Rocky 9.2 Compiler: GCC 13.3 brpc: master 2026.08.23 protobuf: - **Additional context/screenshots** The client-side RDMA socket does not have this issue because its edge trigger is RdmaEndpoint::OnNewDataFromTcp, which checks the RDMA state and only reads 1 byte for EOF detection in ESTABLISHED state (without touching _read_buf). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
