tgravescs commented on issue #26614: [SPARK-29976][CORE] Trigger speculation for stages with too few tasks URL: https://github.com/apache/spark/pull/26614#issuecomment-561679568 it minimizes impact and makes it predictable when the new speculationTaskDurationThresOpt is applied. If you only apply it when the number of tasks is small < number of slots per executors, its easier to reason about, if it can apply during any stage then I need to worry about it being applied to my large stages even if I configured the other speculation configs to be what I really want it to use. I agree with you that if you have 2 stages of 1 task each picking the timeout here can be tricky, which is why the normal speculation configs use a multiple of the run time. You can't do that with only 1 task though. But I don't see how to get around that. My point is with using the unfinished, it now expands that same impact to not only stages with 1 task but all my stages.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
