cxzl25 commented on issue #25078: [SPARK-28305][YARN] Request GetExecutorLossReason to use a smaller timeout parameter URL: https://github.com/apache/spark/pull/25078#issuecomment-510433736 In the yarn-client mode, the driver closes the AM connection, causing the entire job to exit, causing unnecessary failures. I searched the code and used the ask+recover method very rarely. Adjust parameters should be work. ``` spark.rpc.askTimeout 120s spark.rpc.io.connectionTimeout 130s ``` Parameter priority | module | first | second | | ----------------------- | ------------------------------ | --------------------- | | RpcEndpointRef#ask | spark.rpc.askTimeout | spark.network.timeout | | TransportChannelHandler | spark.rpc.io.connectionTimeout | spark.network.timeout | I found a problem with the abnormal exit of the yarn client mode last time, and found and fixed a problem. https://github.com/apache/spark/pull/23989 In the production environment, this patch is used, and there is still a probability of abnormal exit. Later, I carefully looked at the logic of this piece and found the problem.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
