bmarcott edited a comment on issue #26696: [WIP][SPARK-18886][CORE] Only reset scheduling delay timer if allocated slots are fully utilized URL: https://github.com/apache/spark/pull/26696#issuecomment-561438844 What were the particular critiques of this PR? I suggest reading, particularly Kay's comments, [SPARK-18886](https://issues.apache.org/jira/browse/SPARK-18886?jql=project%20%3D%20SPARK%20AND%20text%20~%20delay) as well as the comments [in this old PR](https://github.com/apache/spark/pull/9433). Below ideas I have seen (and some I tried) have the following issues: 1. Never reset timer: delay scheduling only works on first wave (as @tgravescs pointed out) 2. Per slot timer: delay scheduling should apply per task/taskset (as I pointed out above) which might lead you to # 3 3. Per slot per stage timer: tasks can be starved by being offered many different slots, each starting a new timer). also too much bookkeeping 4. Per task timer: you still need a way to distinguish between when a task is waiting for a slot to become available vs it has them available but is not utilizing them (which is what this PR does). To do this right seems to be this PR + more timers.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
