laskoviymishka opened a new pull request, #2613:
URL: https://github.com/apache/iceberg-go/pull/2613

   This adds the last missing Phase 3 piece for dynamic partition overwrite: a 
Spark interop test. The table is partitioned by day(dt) and identity(cat), so 
the test covers more than plain identity partitions. iceberg-go appends rows 
across several partitions, including a null one, then dynamically overwrites 
two of them. Spark must see the untouched partitions unchanged, the replaced 
partitions holding only the new rows, and an overwrite snapshot with 
replace-partitions=true. In the other direction, Spark runs its own dynamic 
INSERT OVERWRITE on one partition, and iceberg-go must read the 
replace-partitions snapshot and the merged rows correctly.
   
   Every validation.py call starts a new Spark session, so a SET in a separate 
call would be lost. The script now accepts an optional, repeatable --conf 
KEY=VALUE, and the test uses it to switch partitionOverwriteMode to dynamic for 
that one statement. No library code changes and the exported API stays the 
same. One composite literal in the DynamoDB integration test is also 
reformatted, because `golangci-lint fmt` on main flags it.
   
   The test is in SparkIntegrationTestSuite and runs with -tags integration 
against the docker recipe. It needs nothing specific to Spark 4.
   
   Part of #2062.
   
   AI-assisted, reviewed by a human.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to