laskoviymishka opened a new pull request, #2613: URL: https://github.com/apache/iceberg-go/pull/2613
This adds the last missing Phase 3 piece for dynamic partition overwrite: a Spark interop test. The table is partitioned by day(dt) and identity(cat), so the test covers more than plain identity partitions. iceberg-go appends rows across several partitions, including a null one, then dynamically overwrites two of them. Spark must see the untouched partitions unchanged, the replaced partitions holding only the new rows, and an overwrite snapshot with replace-partitions=true. In the other direction, Spark runs its own dynamic INSERT OVERWRITE on one partition, and iceberg-go must read the replace-partitions snapshot and the merged rows correctly. Every validation.py call starts a new Spark session, so a SET in a separate call would be lost. The script now accepts an optional, repeatable --conf KEY=VALUE, and the test uses it to switch partitionOverwriteMode to dynamic for that one statement. No library code changes and the exported API stays the same. One composite literal in the DynamoDB integration test is also reformatted, because `golangci-lint fmt` on main flags it. The test is in SparkIntegrationTestSuite and runs with -tags integration against the docker recipe. It needs nothing specific to Spark 4. Part of #2062. AI-assisted, reviewed by a human. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
