hcrosse opened a new pull request, #1449: URL: https://github.com/apache/arrow-go/pull/1449
### Rationale for this change A Substrait field reference rejects a dictionary column even when its values have the expected type. This prevents iceberg-go from filtering columns written by PyIceberg from pandas categoricals or dictionary arrays (https://github.com/apache/iceberg-go/issues/2425). ### What changes are included in this PR? `execFieldRef` checks a dictionary's value type against the expected type. The existing exception for large string and binary types from https://github.com/apache/arrow-go/pull/440 applies to dictionary values too. Mismatched value types still return `arrow.ErrInvalid`, with the original dictionary type in the message. `execFieldRef` returns the dictionary undecoded, as it already returns large types unchanged. Decoding there would copy every value in each batch, and the functions iceberg-go applies to columns already handle dictionaries. The comparison, `is_in` and `is_nan` kernels decode them through `DispatchBest`. `is_null` and `is_not_null` read the index validity bitmap, so a valid index that points to a null dictionary value counts as not null. Dictionaries read from Parquet never have null values, because dictionary pages hold no nulls. An expression that is only a field reference now returns the dictionary array itself, as one on a `large_string` column returns `large_string`. ### Are these changes tested? Yes. I added `TestDictionaryTypes`, which covers dictionaries of `string`, `large_string`, `binary` and `large_binary` with `int8` and `int32` indices. It runs `equal` and `is_null` against a schema that declares the plain value type, and checks that a mismatched value type returns `arrow.ErrInvalid`. The test fails on main. On Linux amd64, `go test ./arrow/compute/...`, `go vet`, `gofmt` and `golangci-lint` passed, along with the `prek` hooks on the changed files, including the Release Audit Tool. I also ran iceberg-go `8ef0a092` with this change on tables that PyIceberg wrote from a pandas categorical, from `dictionary_encode()` and through `add_files()`. The 11 filters cover `=`, `IN`, `!=`, `IS NULL` and `IS NOT NULL AND !=`. All 11 failed with the dictionary error on main and returned the expected rows with this change. I did not test arm64, non-string dictionary values end to end, files from other writers or remote catalogs and storage. ### Are there any user-facing changes? Yes. A Substrait field reference now accepts a dictionary column whose value type matches the expected type, or is its large counterpart. There is no API change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
