WeichenXu123 edited a comment on issue #25235: [SPARK-28483][Core] Fix canceling a spark job using barrier mode but barrier tasks blocking on BarrierTaskContext.barrier() URL: https://github.com/apache/spark/pull/25235#issuecomment-515644786 @jiangxb1987 This approach is very similar with my first version code, link is here https://github.com/apache/spark/commit/b47577483935c51e19402a1c0c45f98526e692e6 although it is simple, the issue is that it cannot release the RPC resource when killed. See my old comment here: https://github.com/apache/spark/pull/25235#discussion_r307563900 i.e., we cannot do the following which RPC timeout will do: https://github.com/WeichenXu123/spark/blob/7021588ba8365b2018f9f7b5d3b05285886cd0be/core/src/main/scala/org/apache/spark/rpc/netty/NettyRpcEnv.scala#L242 Check code we can see `rpcMessage.onTimeout()` will do `client.removeRpcRequest(requestId)`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
