dongjoon-hyun commented on issue #27871: [SPARK-31105][CORE]Respect sql execution id for FIFO scheduling mode URL: https://github.com/apache/spark/pull/27871#issuecomment-602139486 Hi, @liupc . - Currently, `Core` module's `FIFO` concept doesn't mean SQL level execution. The following is not Apache Spark's contract. > And users expect the FIFO behave like this. - Also, this is a trade-off. And, in general, this hurts the global throughput (as @jiangxb1987 mentioned already). Apache Spark cannot accept that kind of general performance degradation for those rare use cases. Please note that not every users are SQL users. - Lastly, I'm -1 for the current architectural design which makes a dependency from `core` module's `TaskSetManager` to the external `sql` module. Given that, this PR will not considered as a mergeable PR. If you want to proceed, please split the scopes. First, you may need to focus on adding a new option to respect `job` level priority in the `core` module first. The configuration should be `false` by default. After the suggestion is accepted into `master` branch, you may need to make a second PR to add another option for the whole SQL Query optimization. Thanks!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
