mustafasrepo commented on issue #8572: URL: https://github.com/apache/arrow-datafusion/issues/8572#issuecomment-1862179527
Actually, it sets `preserve_order`, flag to `true` for the second repartition in the plan (What I call `SortPreservingRepartitionExec` in Plan V1). In this mode, during repartitioning `streaming_merge` helper is used. Hence this mode preserves input ordering during repartitioning. This decreases speed as compared to the direct repartitioning (Similar to the `CoalescePartitionsExec` and `SortPreservingMergeExec`) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
