cloud-fan commented on pull request #33644: URL: https://github.com/apache/spark/pull/33644#issuecomment-897406866
The idea looks fine. In Spark SQL, global aggregate (no grouping key) also launches a single reduce task at the executor side to do the aggregation work, and send back the result to the driver later. It's useful when you have many concurrent SQL queries, as you have many reduce tasks to do the computation work instead of a single driver node. And also +1 to use parameters instead of configs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
