tgravescs commented on issue #26614: [SPARK-29976][CORE] New conf for single task stage speculation URL: https://github.com/apache/spark/pull/26614#issuecomment-558660758 it seems like either the number of slots on 1 executor or a config might be best. I think there are corner cases for all of these its just picking the one that seems to cover most. just a side note, Tez has a similar configs but it only applies when a single task is run. it is obviously different but did solve a problem was saw at Yahoo. So based on what we saw there I'm kind of leaning towards the tasks <= number of slots of 1 executor. That doesn't add another config, covers the 1 task case, plus the 1 bad executor case. @squito thoughts?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
