itang06 commented on issue #27268: URL: https://github.com/apache/datafusion/issues/27268#issuecomment-6952469521
I reproduced the trade-off locally with 1M rows across four parquet files and eight target partitions. With balanced files, preserving partitions took 92 ms versus 114 ms for the shuffle plan. With 97% of rows in one file, it took 320 ms versus 138 ms. The shuffle plan also added a round robin repartition before partial aggregation so the timings compare the full plans. The Parquet scan had exact row counts for each partition in my test. I’ll try a few less extreme splits and a query with fewer groups to see when the shuffle starts paying off. For a first PR would it be reasonable to only consider shuffling when the row counts are exact and one partition has most of the rows? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
