itang06 commented on issue #27268:
URL: https://github.com/apache/datafusion/issues/27268#issuecomment-6952469521

   I reproduced the trade-off locally with 1M rows across four parquet files 
and eight target partitions. With balanced files, preserving partitions took 92 
ms versus 114 ms for the shuffle plan. With 97% of rows in one file, it took 
320 ms versus 138 ms. The shuffle plan also added a round robin repartition 
before partial aggregation so the timings compare the full plans.
   
   The Parquet scan had exact row counts for each partition in my test. I’ll 
try a few less extreme splits and a query with fewer groups to see when the 
shuffle starts paying off. For a first PR would it be reasonable to only 
consider shuffling when the row counts are exact and one partition has most of 
the rows?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to