lakshaydevoo-arch commented on issue #26862: URL: https://github.com/apache/datafusion/issues/26862#issuecomment-6721744119
I've reproduced this on my local DataFusion checkout (55.1.0). With 100,000 rows and `target_partitions=4`, `sort().select()` with a computed expression produces a `ProjectionExec → RepartitionExec(RoundRobinBatch(4)) → SortExec` plan, and the collected rows are not consistently globally ordered. From tracing the optimizer, `require_top_ordering_helper` attaches `OutputRequirementExec` below the projection. `EnsureRequirements` can then introduce repartitioning above that requirement, without restoring global order. Before proposing a fix, I'd like to clarify the intended semantics: **should `DataFrame::sort()` followed by order-maintaining operators such as `select()` preserve global output ordering?** If so, I can add a focused regression test and investigate whether the output requirement should cover the final projection, while accounting for projected-away sort keys and the parallelism tradeoff. I haven't made production changes yet and would appreciate guidance on the expected contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
