manuzhang opened a new issue, #18952: URL: https://github.com/apache/iceberg/issues/18952
### Feature Request / Improvement Add an Iceberg-backed regression for [SPARK-60044](https://github.com/apache/spark/pull/59306), which freezes the ordering used by a planned `GroupPartitionsExec` k-way merge. #18294 explicitly leaves the exact upstream scenario untested. Its computed-key ORDER BY cases exercise SPARK-59905; they do not establish that a planned sorted merge keeps its ordering after a later SQL configuration change. Acceptance criteria: - [ ] Use an Iceberg table with mixed identity and transformed partition keys, multiple splits per grouping key, and a window such as `ROW_NUMBER() OVER (PARTITION BY id ORDER BY name)`. - [ ] Enable the subset-key and preserve-ordering-on-coalesce paths, and plan a sorted `GroupPartitionsExec` merge. - [ ] Change `spark.sql.sources.v2.bucketing.partitionKeyOrdering.enabled` after planning and before execution. - [ ] Assert exact row numbers, retained merge ordering, and a plan that actually uses the sorted merge without an extra sort masking the regression. - [ ] Cover AQE on/off and applicable Iceberg planning modes, restoring configuration after each case. Start from the [upstream regression and its Spark 4.2 backport](https://github.com/apache/spark/pull/59509), adapting it to real Iceberg scans in `spark/v4.2`. Current main already pins Spark 4.2.1. Related: #17959 and the [existing SPJ suite](https://github.com/apache/iceberg/blob/234d55774d3c7355f92c8e93f8e117b70726e556/spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java). ### Query engine Spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
