bmarcott edited a comment on issue #26696: [WIP][SPARK-18886][CORE] Make locality wait time be the time since a TSM's available slots were fully utilized URL: https://github.com/apache/spark/pull/26696#issuecomment-568617329 [Here is an approach](https://github.com/apache/spark/compare/master...bmarcott:nmarcott-fulfill-slots-2?expand=1) which avoids trying to simulate any type of scheduling logic. This change resets locality timers only when no resources have been rejected due to delay scheduling. The complication in this change is that resetting the locality timer is dependent on when `TaskSchedulerImpl.resourceOffers` is called with all resources, rather than a single resource. That is guaranteed to happen only as frequent as `spark.scheduler.revive.interval` which defaults to 1s. This means the `locality.wait.time` wouldn't we completely respected. Ways around this would be to always offer all free resources on calls to `TaskSchedulerImpl.resourceOffers`. Thoughts or know of any other issues with this approach?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
