2010YOUY01 commented on PR #15302: URL: https://github.com/apache/datafusion/pull/15302#issuecomment-2739050824
> Thanks @2010YOUY01 -- this makes sense. Did you run any benchmark numbers for this change? Thank you for the review! Now `RowConverter` is only used when sort key includes a `List` type, I have run the `sort_tpch` benchmark and verified the run time is unchanged. > > It seems like we have an external aggregation benchmark in https://github.com/apache/datafusion/tree/main/benchmarks#external-aggregation but not an external sorting benchmark 🤔 After `Rows` are used by default for sorting more benchmarking is definitely required, I think for external sorting, an easy way to extend the benchmark will be: 1. Profile each query in `sort_tpch` benchmark for memory consumption 2. Include a new configuration `--memory-limit-tier` to let each query run in 50%, 20% of the actual memory consumption, and see how the performance change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
