AynSiddiqui opened a new pull request, #2639: URL: https://github.com/apache/iceberg-go/pull/2639
## Rationale for this change After partition spec evolution, `Delete` on a table with data files from an older spec wrote position delete files using the table's current spec. The delete file's spec ID and partition did not match the data file it refers to, so readers could not apply it (and with an unpartitioned current spec, the unpartitioned path was taken regardless of the data file's spec). Fixes #2606 ## What changes are included in this PR? - `positionDeleteRecordsToDataFiles` only uses the unpartitioned writer when the current spec is unpartitioned and every targeted data file is on the current spec; otherwise it uses the partitioned fanout writer. - The position delete fanout writer passes each data file's own spec to the rolling writer. - `openFileWriter`, `newRollingDataWriter` and `getOrCreateRollingDataWriter` take the spec as a parameter instead of always using `currentSpec`. The data-file write paths pass `factory.currentSpec`, so their behaviour is the same (hence the many one-line caller and test updates); the position-delete fanout writer passes the data file's spec. - Rolling writers are keyed by (spec ID, partition path) so two specs with the same path don't share a writer, and removal from the map uses that key (it previously used the path only). - The v3 deletion vector path is not changed. #2624 also changes `writePositionDeletesForFiles` for the v3 path; this PR changes the shared rolling-writer code and the v2 position-delete path, and I'll rebase once #2624 lands. ## Are these changes tested? Run locally on main `3d1d676`: - `golangci-lint run` (2.15.0; the Makefile pins 2.14.0 for install): 0 issues - `go test -count=1 ./...`: pass - `go test -race -run 'SpecEvolution|PosDelete|PositionDelete' ./table/`: pass - New `TestPositionDeleteAfterSpecEvolution` (old-spec, mixed-specs, evolve-to-unpartitioned): fails on `3d1d676` without the change in all 3 cases and passes with it. It checks that each delete file's spec ID and partition equal its data file's. - New `TestWriterKeysIncludeSpecAndAreRemoved`. Not run: the integration suites (Spark/REST/catalog) and any interop check against Java Iceberg reading these delete files. ## Are there any user-facing changes? Bug fix: position delete files written after spec evolution now carry the correct spec ID and partition. No API changes (the changed functions are unexported). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
