HyukjinKwon edited a comment on issue #28168: [SPARK-31395][CORE]reverse preferred location to make schedule more even URL: https://github.com/apache/spark/pull/28168#issuecomment-611505430 So are you saying you have 3 replica in three nodes and Spark job is only being executed in one specific node because of the locality? Then, how does reversing hosts help? You shouldn't use your driver node as a cluster ideally. In production you should better use Yarn cluster mode for such reason as an example. You're arguing that one specific case the driver and executor exist in one specific note together, and the workload is heavy in the specific node. What if the last node has both driver and executor? Reversing hosts doesn't solve anything.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
