pan3793 commented on PR #56323: URL: https://github.com/apache/spark/pull/56323#issuecomment-4658134937
@dongjoon-hyun, when both `...useHashAggregateExec` and `...useObjectHashAggregateExec` are disabled, all logic agg node plans to `SortAggregateExec`, which is a slower but safer option. I added the tests to cover this case. Also, update the config docs to mention that when both configs are enabled, planner prefers to choose `HashAggregateExec` then `ObjectHashAggregateExec` when possible, otherwise `SortAggregateExec`. @LuciferYang, I experimented with your idea by introducing `shouldSpillBeforeAppendNewKey`, OOM risk gets reduced, but still happens. The previous consistently failed job with `HashAggregateExec` now sometimes succeeded with that change. Given that, I think the current approach is still adorable. <img width="1624" height="330" alt="image" src="https://github.com/user-attachments/assets/4e56d16b-10be-4da8-8f57-9f3f1ed99b5e"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
