Dandandan commented on PR #21580: URL: https://github.com/apache/datafusion/pull/21580#issuecomment-4234648190
> > I wonder if the ordering should be done _before_ the files / row groups are assigned to partitions? So they are more _globally_ sorted instead of just per partition? It seems now they are sorted _within_ each partition, which should help, but perhaps not nearly as much as it would be if all the partitions contain the optimal row groups? > > This would also help in the case of #21581 > > Great point @Dandandan — you're right that global reorder is much more effective than per-partition reorder. With global reorder + round-robin distribution, each partition's first RG is close to the global optimum, so: > > 1. All partitions quickly converge to tight local TopK thresholds in parallel > 2. SPM merging finishes faster because each partition's first few batches contain optimal values → LIMIT can be satisfied with minimal reads across partitions > > The current per-partition reorder is limited because even after sorting, partition 0's "best" RG might be much worse than the global best (which may have landed in partition 2). > > Moving to global reorder would require changes at the planning / EnforceDistribution layer to load RG statistics and redistribute RGs across partitions. I'd prefer to keep this PR as an incremental step (per-partition) and track global reorder as a follow-up — it would benefit both #21317 and #21581. > > Does this make sense? Sure, makes sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
