ChenjunZou commented on issue #28168: [SPARK-31395][CORE]reverse preferred location to make schedule more even URL: https://github.com/apache/spark/pull/28168#issuecomment-611509646 Thanks for your explanation @HyukjinKwon I have 2 questions more. > You shouldn't use our driver as a cluster ideally. what does this mean. > In production you should better use cluster mode for such reason. Actually I use the cluster mode. if I use client mode, the client writes HDFS block will be even. single node hot will not happen. > What if the last node has both driver and executor? that is unrelated. the PROBLEM is: the first location of all blocks will always be the same xxx.93. which made scheduler try executors in that node first.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
