adiranjan25 opened a new pull request, #18918:
URL: https://github.com/apache/iceberg/pull/18918

   ## Summary
   
   Adds Spark 4.2 regression coverage for subset-key `MERGE INTO` operations 
involving Iceberg tables partitioned by `days(event_date)` and `bucket(2, id)`.
   
   Related to #18597.
   
   ## Changes
   
   - Adds a synthetic MERGE workload joining on `id`, a subset of the target's 
partition keys.
   - Exercises Spark Storage Partitioned Join behavior with partially clustered 
distribution enabled and disabled.
   - Checks physical-plan partition-group counts and validates the final MERGE 
results.
   
   ## Observations
   
   In the reproduced test environment:
   
   - **Partial clustering disabled:** 10 target input partitions were grouped 
into 2 execution groups.
   - **Partial clustering enabled:** The workload produced 10 execution groups.
   - Both configurations completed the MERGE correctly.
   
   These observations demonstrate partition-concentration behavior. They do not 
establish an out-of-memory failure or a performance improvement.
   
   ## Validation
   
   - Spark 4.2 targeted regression tests passed.
   - Full `TestStoragePartitionedJoinsInRowLevelOperations` test class passed.
   - Spotless formatting completed successfully.
   
   ## Scope
   
   This PR adds regression coverage; it does not implement node-local shuffle.
   
   The grouping-count assertion does not yet conclusively identify the target 
scan by physical-plan lineage. Maintainer feedback on strengthening this 
assertion and on the appropriate implementation approach for #18597 is welcome.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to