jcferretti opened a new issue, #41919: URL: https://github.com/apache/arrow/issues/41919
### Describe the bug, including details regarding any error messages, version, and platform. This is the cython code (pyx) implementation of flight_client.authenticate: https://github.com/apache/arrow/blob/6a28035c2b49b432dc63f5ee7524d76b4ed2d762/python/pyarrow/_flight.pyx#L1440 ``` 1440 def authenticate(self, auth_handler, options: FlightCallOptions = None): 1441 """Authenticate to the server. 1442 1443 Parameters 1444 ---------- 1445 auth_handler : ClientAuthHandler 1446 The authentication mechanism to use. 1447 options : FlightCallOptions 1448 Options for this call. 1449 """ 1450 cdef: 1451 unique_ptr[CClientAuthHandler] handler 1452 CFlightCallOptions* c_options = FlightCallOptions.unwrap(options) 1453 1454 if not isinstance(auth_handler, ClientAuthHandler): 1455 raise TypeError( 1456 "FlightClient.authenticate takes a ClientAuthHandler, " 1457 "not '{}'".format(type(auth_handler))) 1458 handler.reset((<ClientAuthHandler> auth_handler).to_handler()) 1459 with nogil: 1460 check_flight_status( 1461 self.client.get().Authenticate(deref(c_options), 1462 move(handler))) ``` The call to Authenticate on line 1461 is to grpc_client.cc [line 860](https://github.com/apache/arrow/blob/6a28035c2b49b432dc63f5ee7524d76b4ed2d762/cpp/src/arrow/flight/transport/grpc/grpc_client.cc#L860): ``` 860 Status Authenticate(const FlightCallOptions& options, 861 std::unique_ptr<ClientAuthHandler> auth_handler) override { 862 auth_handler_ = std::move(auth_handler); 863 ClientRpc rpc(options); 864 return AuthenticateInternal(rpc); 865 } ``` Many calls in the same file use the `auth_handler_` member of the struct (https://github.com/apache/arrow/blob/6a28035c2b49b432dc63f5ee7524d76b4ed2d762/cpp/src/arrow/flight/transport/grpc/grpc_client.cc#L1078) that is being assigned on line 862 above, eg, [DoPut](https://github.com/apache/arrow/blob/6a28035c2b49b432dc63f5ee7524d76b4ed2d762/cpp/src/arrow/flight/transport/grpc/grpc_client.cc#L1002): ``` 997 Status DoPut(const FlightCallOptions& options, 998 std::unique_ptr<internal::ClientDataStream>* out) override { 999 using GrpcStream = ::grpc::ClientReaderWriter<pb::FlightData, pb::PutResult>; 1000 1001 auto rpc = std::make_shared<ClientRpc>(options); 1002 RETURN_NOT_OK(rpc->SetToken(auth_handler_.get())); 1003 std::shared_ptr<GrpcStream> stream = stub_->DoPut(&rpc->context); 1004 *out = std::make_unique<GrpcClientPutStream>(std::move(rpc), std::move(stream)); 1005 return Status::OK(); 1006 } ``` SetToken on the same file, [line 86](https://github.com/apache/arrow/blob/6a28035c2b49b432dc63f5ee7524d76b4ed2d762/cpp/src/arrow/flight/transport/grpc/grpc_client.cc#L87): ``` 86 /// \brief Add an auth token via an auth handler 87 Status SetToken(ClientAuthHandler* auth_handler) { 88 if (auth_handler) { 89 std::string token; 90 RETURN_NOT_OK(auth_handler->GetToken(&token)); 91 context.AddMetadata(kGrpcAuthHeader, token); 92 } 93 return Status::OK(); 94 } ``` There is a race between calling `auth_handler_.get()` to get out a raw pointer out of the `auth_handler_` `shared_ptr` and using it inside `SetToken`, and another thread changing the value of `auth_handler_` via the its assignment operator, which can trigger the deletion of the previously held pointer value. That deletion can happen after the call to `auth_handler_.get()` in the other thread, and `SetToken` using that raw pointer value. My team builds a server and client libraries based on flight. One of our customers ran into an issue while using hundreds of concurrent sessions to our service. Our client library was using calls to FlightClient.authenticate triggered by a timer once every 5 minutes, which triggered the problem. More details about the symptoms we saw here: https://github.com/deephaven/deephaven-core/pull/5489 ### Component(s) FlightRPC -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
