otterc commented on PR #37533: URL: https://github.com/apache/spark/pull/37533#issuecomment-1219784129
@wankunde > Send finalize RPCs will block the main thread due to creating connection to some unreachable nodes. Which main thread are you referring to here? Could you please explain which thread is being blocked. AFAICT this is being done by `shuffle-merge-finalizer` threads. Why can't you reduce the `SPARK_NETWORK_IO_CONNECTIONCREATIONTIMEOUT_KEY` default for the cluster? One of the reasons this configuration was introduced was because creating connection should have a lower timeout that connection idle. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
