venkata91 edited a comment on pull request #33896: URL: https://github.com/apache/spark/pull/33896#issuecomment-923323263
Also regarding the comment in the JIRA that not to finalize when there are no blocks pushed due to the threshold. I almost implemented that change, but then I thought anyway we are having a timeout (`spark.shuffle.push.results.timeout (default 10 secs)`) if finalize request don't get responded by then, scheduler moves forward submitting the next stages. Do you think we really get much with avoiding finalize when there are no blocks pushed? Thoughts? cc @Ngone51 @Victsm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
