hcrosse opened a new pull request, #2427: URL: https://github.com/apache/iceberg-go/pull/2427
Closes https://github.com/apache/iceberg-go/issues/2426. Equality deletes can miss rows when a binary column is stored as an Arrow dictionary. I fixed the key encoding to use the dictionary values. ## What I added a dictionary case to `makeColEncoder` that encodes each row's value through the encoder for the dictionary's value type. Both data columns and delete-file columns use this encoder. Dictionary columns still take the per-row path, not the single-column fast path in `applySingleBytes`. ## Why Dictionary columns fell back to `ValueStr`, which returns base64 for binary values. Those keys did not match the raw bytes from the delete file. Delete files read with a stored dictionary type take the same path, so I checked one with a test I did not commit. It produced base64 keys on `main` and raw-byte keys with the fix. String dictionaries already worked for non-null values, as the issue reports. Arrow also allows a null entry inside a dictionary. On `main`, a valid index pointing to one was encoded as the text `(null)`, so null-key deletes missed it, for both string and binary values. Dictionaries read from Parquet keep nulls in the indices, so this affects in-memory Arrow input. ## Testing I added `TestProcessEqualityDeletesDictionary` with five dictionary shapes covering binary, large binary, string and large string values. Each shape has a value-key case, a null-key case and a two-column case. On `main`, 13 of the 15 subtests fail. Only the two string value cases pass. All 15 pass with the fix. The issue's reproduction returns `rows=1` with the fix and `rows=3` on `main`. `go test ./table/...`, `go vet ./table/...` and `golangci-lint run ./table/...` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
