zhuqi-lucas commented on PR #21580:
URL: https://github.com/apache/datafusion/pull/21580#issuecomment-4234614265
> I wonder if the ordering should be done _before_ the files / row groups
are assigned to partitions? So they are more _globally_ sorted instead of just
per partition? It seems now they are sorted _within_ each partition, which
should help, but perhaps not nearly as much as it would be if all the
partitions contain the optimal row groups?
>
> This would also help in the case of #21581
Great point — you're right that global reorder is much more effective than
per-partition reorder. With global reorder + round-robin distribution, each
partition's first RG is close to the global optimum, so:
1. All partitions quickly converge to tight local TopK thresholds in
parallel
2. SPM merging finishes faster because each partition's first few batches
contain optimal values → LIMIT can be satisfied with minimal reads across
partitions
The current per-partition reorder is limited because even after sorting,
partition 0's "best" RG might be much worse than the global best (which may
have landed in partition 2).
Moving to global reorder would require changes at the planning /
EnforceDistribution layer to load RG statistics and redistribute RGs across
partitions. I'd prefer to keep this PR as an incremental step (per-partition)
and track global reorder as a follow-up — it would benefit both #21317 and
#21581.
Does this make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
