manuzhang edited a comment on issue #26813: [SPARK-30188][SQL][WIP] Enable adaptive query execution by default URL: https://github.com/apache/spark/pull/26813#issuecomment-564905469 @dongjoon-hyun I tried same query with latest master branch (shuffle services are still 2.3) and saw the same regression and executor GC issue with AQE enabled. IIUC, reducing partitions would also reduce number of tasks and executors. Hence, each executor has to process more data, which could cause GC issue and slow things down.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
