EnricoMi commented on PR #38358: URL: https://github.com/apache/spark/pull/38358#issuecomment-1288156851
@cloud-fan this fixes a very peculiar bug introduced via AQE in Spark 3.0. With Spark 3.4, the issue disappeared. A user-defined ordering, that starts with the partition columns, is broken by `FileFormaWriter` because it cannot see the actual order as `AdaptiveSparkPlanExec` does not know the final plan yet. Sorting by the partition columns breaks the existing user-defined order **only when spills occur**, because `UnsafeSorterSpillMerger` round-robins over the spill files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
