printstacktrace commented on issue #23402: [SPARK-26500][CORE] Add conf to support ingoring data locality URL: https://github.com/apache/spark/pull/23402#issuecomment-450477662 > I don't understand this propose add a configuration used for disable data locality.There are exists a configuration `spark.locality.wait` is charge of the function.You can set zero for `spark.locality.wait`. When submit a job, the driver will go into DAGScheduler.handleJobSubmitted, it will call **getPreferredLocs** for each partition of a rdd. It's single thread and time-consuming when there are huge partitions, and this code path is not controlled by spark.locality.wait.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
