juliuszsompolski opened a new pull request, #42806: URL: https://github.com/apache/spark/pull/42806
### What changes were proposed in this pull request? After an "INVALID_HANDLE.OPERATION_NOT_FOUND" error, client would realize that the failure in ReattachExecute was because the initial ExecutePlan didn't reach the server. It would then call another ExecutePlan, and it will throw a RetryException to let the retry logic handle retrying. However, the retry logic would then immediately send a ReattachExecute, and the client will want to use the iterator of the reattach. However, on the server the ExecutePlan and ReattachExecute could race with each other: * ExecutePlan didn't reach executeHolder.runGrpcResponseSender(responseSender) in SparkConnectExecutePlanHandler yet. * ReattachExecute races around and reaches executeHolder.runGrpcResponseSender(responseSender) in SparkConnectReattachExecuteHandler first. * When ExecutePlan reaches executeHolder.runGrpcResponseSender(responseSender), and executionObserver.attachConsumer(this) is called in ExecuteGrpcResponseSender of ExecutePlan, it will kick out the ExecuteGrpcResponseSender of ReattachExecute. So even though ReattachExecute came later, it will get interrupted by the earlier ExecutePlan and finish with a INVALID_CURSOR.DISCONNECTED error. After this change, such a race between ExecutePlan / ReattachExecute can still happens, but the client should no longer send these requests in such quick succession. ### Why are the changes needed? While deemed unlikely in the ticket, this turned out to happen in practice in testing. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Integration testing. ### Was this patch authored or co-authored using generative AI tooling? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
