liupc commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode URL: https://github.com/apache/spark/pull/27871#issuecomment-601599878 ` > IIUC Spark didn't optimize the workload toward the direction that the query delayed time should be minimized. Actually, I think scheduling Stages from the same sql execution would lead to a few nodes with the shuffle output files become very hot, thus lead to worse performance for the whole cluster compared to current approach. In real clusters, resources is more important than locality. And users expect the FIFO behave like this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
