adriangb commented on PR #21580: URL: https://github.com/apache/datafusion/pull/21580#issuecomment-4248896930
> > > I wonder if the ordering should be done _before_ the files / row groups are assigned to partitions? So they are more _globally_ sorted instead of just per partition? It seems now they are sorted _within_ each partition, which should help, but perhaps not nearly as much as it would be if all the partitions contain the optimal row groups? > > > This would also help in the case of #21581 > > > > > > Great point @Dandandan — you're right that global reorder is much more effective than per-partition reorder. With global reorder + round-robin distribution, each partition's first RG is close to the global optimum, so: > > > > 1. All partitions quickly converge to tight local TopK thresholds in parallel > > 2. SPM merging finishes faster because each partition's first few batches contain optimal values → LIMIT can be satisfied with minimal reads across partitions > > > > The current per-partition reorder is limited because even after sorting, partition 0's "best" RG might be much worse than the global best (which may have landed in partition 2). > > Moving to global reorder would require changes at the planning / EnforceDistribution layer to load RG statistics and redistribute RGs across partitions. I'd prefer to keep this PR as an incremental step (per-partition) and track global reorder as a follow-up — it would benefit both #21317 and #21581. > > Does this make sense? > > Sure, makes sense. I also wonder if this is something that will be easier to do once morselized scans are in place. Right now it's pretty problematic / clunky to mess with the distribution of files within partitions, but if we had a single queue of files to process that is shared across all partitions we can sort the files before we put them in there and then sort the row groups when we morselize and we get an optimally sorted stream in each partition that is robust to data skew. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
