shivadarshan-devadiga opened a new pull request, #18990: URL: https://github.com/apache/iceberg/pull/18990
Closes #18952. [SPARK-60044](https://github.com/apache/spark/pull/59509) freezes the ordering of a planned `GroupPartitionsExec` k-way merge. Before it, turning `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` off after planning emptied the ordering the merge uses, so a parent that relied on the merge read the rows out of order. This adds an Iceberg-backed version of the upstream regression to `TestStoragePartitionedJoins` in `spark/v4.2`, as #18294 left it untested. The new test, `testPlannedSortedMergeKeepsOrderingAfterConfigChange`: - creates a table partitioned by `(id, days(arrive_time), name)`, with one row per partition. The scan reports no ordering, so Spark derives `[id, name]` from the partition keys. The splits are sorted by the full key, so within `id = 1` the `bb` split comes before the `aa` one. - plans `ROW_NUMBER() OVER (PARTITION BY id ORDER BY name)` with SPJ, `partitionKeyOrdering` and `preserveOrderingOnCoalesce` on, and checks that the window's ordering comes from a k-way merge (`SortedMerge`) and not a sort. - turns `partitionKeyOrdering` off, runs the already planned `Dataset`, and checks that the rows of `id = 1` are numbered `aa = 1`, `bb = 2`. - runs with AQE off and on, as the upstream test does. It is added to Spark 4.2 only: Spark 4.0 and 4.1 have no `GroupPartitionsExec` k-way merge, which SPARK-55715 added in 4.2. Testing: - The new test and the rest of `TestStoragePartitionedJoins` pass on Spark 4.2.1, the version on main. - Against Spark 4.2.1 RC2 (staging repository `orgapachespark-1537`, the last build without SPARK-60044), Spark plans the k-way merge but the new test fails in both AQE modes: the rows of `id = 1` are numbered in split order (`expected: "aa" but was: "bb"`). - On Spark 4.2.0, Spark doesn't plan a k-way merge for this Iceberg scan at all and adds a `Sort` instead, with vectorized and row-based reads, so the bug can't be reached there and the test fails at its plan check. - `spotlessCheck` passes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
