bmarcott commented on pull request #28656: URL: https://github.com/apache/spark/pull/28656#issuecomment-635651858
@Ngone51 Thanks for looking into this. Let's make a few corrections to the description: This happened with previous delay scheduling as well (although likely less noticeable as @tgravescs pointed out) It won't "always" use ANY for "all" tasks in the taskset. For example if a single executor is added that is NODE_LOCAL and is used for scheduling, the very first resourceOffer would switch the taskset to NODE_LOCAL. I do see how this handles the race condition on startup, although a more specific handling of that case may be preferable (@tgravescs also mentioned). It seems questionable whether executor added should instantly reset the locality levels. Deferring to @tgravescs and @cloud-fan to make the call here. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
