bmarcott opened a new pull request #28188: [SPARK-18886][CORE] remove requirement to launch a task to reset locality wait timer URL: https://github.com/apache/spark/pull/28188 ### What changes were proposed in this pull request? Remove the requirement to launch a task in order to reset locality wait timer. ### Why are the changes needed? Recently https://github.com/apache/spark/pull/27207 was merged, but contained a bug which leads to undesirable behavior. The crux of the issue is that there was a requirement that a task be launched for the locality timer to be reset. This lead to a problem where once locality level reached ANY, single resource offers are all accepted, leading allResourceOffers to be left with no resources to utilize (hence no task being launched on an all resource offer). Removing this condition also makes intuitive sense, because the condition that matters is whether a task was not launched due to locality levels. Tasks may not be launched for other reasons such as blacklisting or not meeting resource requirements, and these shouldn't count towards delay scheduling ### Does this PR introduce any user-facing change? No ### How was this patch tested? TaskSchedulerImplSuite Also manually tested similar to how I tested in https://github.com/apache/spark/pull/27207 using [this simple app](https://github.com/bmarcott/spark-test-apps/blob/master/src/main/scala/TestLocalityWait.scala). With the new changes, given locality wait of 10s the behavior is generally: 10 seconds of locality being respected, followed by a single full utilization of resources using ANY locality level, followed by 10 seconds of locality being respected, and so on If the legacy flag is enabled (spark.locality.wait.legacyResetOnTaskLaunch=true), the behavior is only scheduling PROCESS_LOCAL tasks (only utilizing a single executor) @cloud-fan @tgravescs
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
