manuzhang opened a new issue, #18948:
URL: https://github.com/apache/iceberg/issues/18948

   ### Feature Request / Improvement
   
   Keep the useful Iceberg integration tests from #18340 (Spark 4.0.5), #18339 
(Spark 4.1.4), and #18294 (Spark 4.2.1) as permanent tests after those 
candidate-verification drafts are closed.
   
   At [main 
`234d55774d3c`](https://github.com/apache/iceberg/commit/234d55774d3c7355f92c8e93f8e117b70726e556),
 the three `TestStoragePartitionedJoins` suites do not contain these draft 
additions. [The version 
catalog](https://github.com/apache/iceberg/blob/234d55774d3c7355f92c8e93f8e117b70726e556/gradle/libs.versions.toml)
 already pins Spark 4.0.5, 4.1.4, and 4.2.1, so this work can use published 
dependencies.
   
   Coverage to retain:
   
   - Mixed-type partition keys with a non-leading subset join key, with partial 
clustering on/off and AQE/skew enabled. Cover [SPARK-60203 on 
4.0/4.1](https://github.com/apache/spark/pull/59410); the corresponding 4.2 
case exercises `GroupPartitionsExec` and should be described as compatibility 
coverage.
   - Broadcast joins ordered by computed join-key expressions, ascending and 
descending, for [SPARK-59905](https://github.com/apache/spark/pull/59189).
   - Runtime filtering with a trailing partition key and one-side shuffle with 
binary partition keys.
   - Chained/successive joins whose intermediate output has unknown partition 
keys.
   - Duplicate partition-key aliases and different projected key subsets in the 
4.1 draft.
   - Columnar UNION ALL preserving key grouping in the 4.1 draft.
   - Joining checkpointed Iceberg scans, with one or both scans checkpointed, 
in the 4.0 draft.
   
   Use the draft cases as starting points and adapt each to the supported Spark 
lines where its API and behavior apply:
   
   - [4.0 source at the verified draft 
head](https://github.com/apache/iceberg/blob/61f07d1e6d1c602731f86887242ede94176f6467/spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java)
   - [4.1 source at the verified draft 
head](https://github.com/apache/iceberg/blob/5556fa5476ab611665a2852172a495ec83a543e8/spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java)
   - [4.2 source at the verified draft 
head](https://github.com/apache/iceberg/blob/35711c6a0be10942e55ea5b824c71bb7b87e9ff6/spark/v4.2/spark/src/test/java/org/apache/iceberg/spark/sql/TestStoragePartitionedJoins.java)
   
   Acceptance criteria:
   
   - [ ] Preserve exact query results against an SPJ-disabled baseline in LOCAL 
and DISTRIBUTED planning modes.
   - [ ] Assert the relevant plan path, key projection, grouping, columnar 
behavior, and shuffle counts so an ordinary shuffled join cannot mask a 
regression.
   - [ ] Run the containing SPJ suites on the applicable supported Spark lines.
   - [ ] Keep this work test-only, without candidate staging repositories or 
unrelated version changes.
   
   Related: #18383 already covers an aggregate directly over a partially 
clustered join. Coordinate with that PR rather than duplicating its case. 
#10450 is an older, closed multi-table SPJ report; the cases above retain the 
more specific draft coverage.
   
   ### Query engine
   
   Spark
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to