manuzhang opened a new issue, #18948: URL: https://github.com/apache/iceberg/issues/18948
### Feature Request / Improvement Keep the useful Iceberg integration tests from #18340 (Spark 4.0.5), #18339 (Spark 4.1.4), and #18294 (Spark 4.2.1) as permanent tests after those candidate-verification drafts are closed. At [main `234d55774d3c`](https://github.com/apache/iceberg/commit/234d55774d3c7355f92c8e93f8e117b70726e556), the three `TestStoragePartitionedJoins` suites do not contain these draft additions. [The version catalog](https://github.com/apache/iceberg/blob/234d55774d3c7355f92c8e93f8e117b70726e556/gradle/libs.versions.toml) already pins Spark 4.0.5, 4.1.4, and 4.2.1, so this work can use published dependencies. Coverage to retain: - Mixed-type partition keys with a non-leading subset join key, with partial clustering on/off and AQE/skew enabled. Cover [SPARK-60203 on 4.0/4.1](https://github.com/apache/spark/pull/59410); the corresponding 4.2 case exercises `GroupPartitionsExec` and should be described as compatibility coverage. - Broadcast joins ordered by computed join-key expressions, ascending and descending, for [SPARK-59905](https://github.com/apache/spark/pull/59189). - Runtime filtering with a trailing partition key and one-side shuffle with binary partition keys. - Chained/successive joins whose intermediate output has unknown partition keys. - Duplicate partition-key aliases and different projected key subsets in the 4.1 draft. - Columnar UNION ALL preserving key grouping in the 4.1 draft. - Joining checkpointed Iceberg scans, with one or both scans checkpointed, in the 4.0 draft. Use the draft cases as starting points and adapt each to the supported Spark lines where its API and behavior apply: - [4.0 source at the verified draft head](https://github.com/apache/iceberg/blob/61f07d1e6d1c602731f86887242ede94176f6467/spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java) - [4.1 source at the verified draft head](https://github.com/apache/iceberg/blob/5556fa5476ab611665a2852172a495ec83a543e8/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java) - [4.2 source at the verified draft head](https://github.com/apache/iceberg/blob/35711c6a0be10942e55ea5b824c71bb7b87e9ff6/spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java) Acceptance criteria: - [ ] Preserve exact query results against an SPJ-disabled baseline in LOCAL and DISTRIBUTED planning modes. - [ ] Assert the relevant plan path, key projection, grouping, columnar behavior, and shuffle counts so an ordinary shuffled join cannot mask a regression. - [ ] Run the containing SPJ suites on the applicable supported Spark lines. - [ ] Keep this work test-only, without candidate staging repositories or unrelated version changes. Related: #18383 already covers an aggregate directly over a partially clustered join. Coordinate with that PR rather than duplicating its case. #10450 is an older, closed multi-table SPJ report; the cases above retain the more specific draft coverage. ### Query engine Spark -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
