hcrosse opened a new pull request, #1596:
URL: https://github.com/apache/arrow-go/pull/1596

   ### Rationale for this change
   
   Dictionary builders panic for `string_view`, `binary_view`, `large_utf8` and 
`large_binary` through `NewBuilder`, `FromJSON`, `NewRecordBuilder` and 
`NewDictionaryBuilderWithDict`. `NewDictionaryUnifier` rejects them.
   
   I reproduced this on v18.9.0 and main, with Go 1.26.10 on macOS arm64.
   
   ### What changes are included in this PR?
   
   The four types use the binary memo table and `BinaryDictionaryBuilder`, with 
large offsets for the large types, as in Arrow C++:
   
   
https://github.com/apache/arrow/blob/976779b2aac587f51b87c3bd2e6ae4c36f75ab6c/cpp/src/arrow/util/hashing.h#L943-L957
   
   View dictionaries reserve space for values longer than 12 bytes, so these 
land in one data buffer. Null slots in an initial dictionary are inserted as 
empty values without reading them, since a null view may hold any bytes.
   
   ### Are these changes tested?
   
   `TestDictionaryBuilderBinaryLikeValueTypes` covers the four types. All its 
cases fail on main and pass with this change. 
`TestDictionaryBuilderLargeValuesOver2GiB` builds dictionaries over 2 GiB, past 
the limit of `int32` offsets. It needs about 18 GiB of memory, so it runs only 
with the new `ARROW_TEST_LARGE_MEMORY` variable set.
   
   Full `go test -race ./arrow/...` and `golangci-lint` 2.13.2 passed on 
linux/amd64 with pinned test data, and focused tests passed on macOS arm64.
   
   ### Are there any user-facing changes?
   
   Yes. Dictionary builders and unifiers now support these four value types. 
There is no API change.
   
   Closes https://github.com/apache/arrow-go/issues/1562.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to