wwbmmm commented on code in PR #3761:
URL: https://github.com/apache/brpc/pull/3761#discussion_r4649992267
##########
src/brpc/rdma/rdma_endpoint.cpp:
##########
@@ -1653,11 +1654,44 @@ static int DrainCq(ibv_cq* cq) {
}
void RdmaEndpoint::DeallocateResources() {
+ // Drain the CQ Socket first: a running PollCq still uses `_resource'.
+ if (INVALID_SOCKET_ID != _cq_sid) {
+ if (FLAGS_rdma_use_polling) {
+ PollerRemoveCqSid();
+ }
+ SocketUniquePtr s;
+ if (Socket::AddressFailedAsWell(_cq_sid, &s) >= 0) {
+ // The CQ Socket may live on another tag than `_socket'.
+ if (s->_fd >= 0) {
+ s->_io_event.RemoveConsumer(s->_fd);
+ }
+ s->_fd = -1; // closed with the comp_channel
+ s->SetFailed();
+ }
+ s.reset();
+
+ // Spin while a PollCq callback or explicit holder is alive: freeing
+ // `_resource' under one would be a use-after-free.
+ int nwait = 0;
Review Comment:
[replied by brpc-oncall robot] `DeallocateResources()` can be entered on a
thread that itself holds a reference to the CQ Socket, and then this loop never
exits. The main Socket's last reference may be dropped inside `PollCq()`: on an
error path `PollCq()` calls `s->SetFailed()`, and its local `SocketUniquePtr s`
releases the main Socket when the function returns, which runs
`Socket::BeforeRecycled()` -> `RdmaTransport::Release()` ->
`OnTransportReleased()` -> `DeallocateResources()` right there. Meanwhile the
CQ Socket reference is still held by the caller of the callback:
`Socket::OnInputEvent()` keeps a `SocketUniquePtr` for the whole edge-triggered
callback, and the RDMA poller loop keeps one for the whole `PollCq()` call. So
`AddressFailedAsWell(_cq_sid, ...)` keeps returning 1 and the thread spins
forever — the use-after-free is turned into a hang in exactly the teardown path
this PR targets (it also hangs `PollingModeRelease()`/`bthread_join` at exit).
The same holds for
any other holder of that reference.
Please avoid waiting for the Socket slot to be returned: all CQ Socket
references are dropped after `PollCq()` has returned (both callers above
destroy their `SocketUniquePtr` outside the callback), so tearing `_resource`
down in the last `DropRef()` instead would be safe and non-blocking. If a wait
has to be kept for some other reason, at least detect the self-referential case
and bound it instead of spinning indefinitely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]