AynSiddiqui opened a new pull request, #2639:
URL: https://github.com/apache/iceberg-go/pull/2639

   ## Rationale for this change
   
   After partition spec evolution, `Delete` on a table with data files from an 
older spec wrote position delete files using the table's current spec. The 
delete file's spec ID and partition did not match the data file it refers to, 
so readers could not apply it (and with an unpartitioned current spec, the 
unpartitioned path was taken regardless of the data file's spec).
   
   Fixes #2606
   
   ## What changes are included in this PR?
   
   - `positionDeleteRecordsToDataFiles` only uses the unpartitioned writer when 
the current spec is unpartitioned and every targeted data file is on the 
current spec; otherwise it uses the partitioned fanout writer.
   - The position delete fanout writer passes each data file's own spec to the 
rolling writer.
   - `openFileWriter`, `newRollingDataWriter` and 
`getOrCreateRollingDataWriter` take the spec as a parameter instead of always 
using `currentSpec`. The data-file write paths pass `factory.currentSpec`, so 
their behaviour is the same (hence the many one-line caller and test updates); 
the position-delete fanout writer passes the data file's spec.
   - Rolling writers are keyed by (spec ID, partition path) so two specs with 
the same path don't share a writer, and removal from the map uses that key (it 
previously used the path only).
   - The v3 deletion vector path is not changed. #2624 also changes 
`writePositionDeletesForFiles` for the v3 path; this PR changes the shared 
rolling-writer code and the v2 position-delete path, and I'll rebase once #2624 
lands.
   
   ## Are these changes tested?
   
   Run locally on main `3d1d676`:
   - `golangci-lint run` (2.15.0; the Makefile pins 2.14.0 for install): 0 
issues
   - `go test -count=1 ./...`: pass
   - `go test -race -run 'SpecEvolution|PosDelete|PositionDelete' ./table/`: 
pass
   - New `TestPositionDeleteAfterSpecEvolution` (old-spec, mixed-specs, 
evolve-to-unpartitioned): fails on `3d1d676` without the change in all 3 cases 
and passes with it. It checks that each delete file's spec ID and partition 
equal its data file's.
   - New `TestWriterKeysIncludeSpecAndAreRemoved`.
   
   Not run: the integration suites (Spark/REST/catalog) and any interop check 
against Java Iceberg reading these delete files.
   
   ## Are there any user-facing changes?
   
   Bug fix: position delete files written after spec evolution now carry the 
correct spec ID and partition. No API changes (the changed functions are 
unexported).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to