redsanket commented on pull request #30710: URL: https://github.com/apache/spark/pull/30710#issuecomment-756777639
Sure thanks for the suggestion, i do agree this is not optimal and it might be better to make it configurable per stage. In general this is decent change to give more control on the job behavior in order to avoid unnecessary speculative tasks being scheduled and incurring additional shuffle penalty. Also this is more in the direction of conserving cluster resources on a global level. Although I think this needs to be configured instead of hardcoding it anyway. I will make the suggested changes. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
