hcrosse opened a new pull request, #1596: URL: https://github.com/apache/arrow-go/pull/1596
### Rationale for this change Dictionary builders panic for `string_view`, `binary_view`, `large_utf8` and `large_binary` through `NewBuilder`, `FromJSON`, `NewRecordBuilder` and `NewDictionaryBuilderWithDict`. `NewDictionaryUnifier` rejects them. I reproduced this on v18.9.0 and main, with Go 1.26.10 on macOS arm64. ### What changes are included in this PR? The four types use the binary memo table and `BinaryDictionaryBuilder`, with large offsets for the large types, as in Arrow C++: https://github.com/apache/arrow/blob/976779b2aac587f51b87c3bd2e6ae4c36f75ab6c/cpp/src/arrow/util/hashing.h#L943-L957 View dictionaries reserve space for values longer than 12 bytes, so these land in one data buffer. Null slots in an initial dictionary are inserted as empty values without reading them, since a null view may hold any bytes. ### Are these changes tested? `TestDictionaryBuilderBinaryLikeValueTypes` covers the four types. All its cases fail on main and pass with this change. `TestDictionaryBuilderLargeValuesOver2GiB` builds dictionaries over 2 GiB, past the limit of `int32` offsets. It needs about 18 GiB of memory, so it runs only with the new `ARROW_TEST_LARGE_MEMORY` variable set. Full `go test -race ./arrow/...` and `golangci-lint` 2.13.2 passed on linux/amd64 with pinned test data, and focused tests passed on macOS arm64. ### Are there any user-facing changes? Yes. Dictionary builders and unifiers now support these four value types. There is no API change. Closes https://github.com/apache/arrow-go/issues/1562. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
