zeroshade opened a new pull request, #1473: URL: https://github.com/apache/arrow-go/pull/1473
### Rationale for this change Since #1011, reading dictionary-encoded columns allocates more than in v18.7.0. `dictConverter.ensure` decodes into a temporary slice and appends it to the dictionary each time a batch reaches a higher index, so a large dictionary is reallocated many times. #1011's bounds checks are correct; this restores the single up-front allocation without trusting the declared dictionary length. The analysis and the fix are from @zoobst in #1446. This PR ports their patch to current main. ### What changes are included in this PR? - `ensure` decodes straight into the dictionary's spare capacity, growing geometrically up to the declared length when it needs more room. - `NewDictConverter` allocates `min(declared length, page bytes / minimum PLAIN value size)` up front. For a well-formed page that is the declared length. A header that claims far more values than its bytes can hold only gets capacity for what the bytes can hold. - Indexes are still checked against the declared length before anything is allocated, and a short dictionary is still reported as an error. Reading the same 1M-row file (float64, int64 and binary columns, Snappy) through `pqarrow.ReadTable`: | Distinct values | v18.7.0 | main | this PR | | --- | ---: | ---: | ---: | | 10k | 121.3 MiB | 122.7 MiB | 121.0 MiB | | 100k | 136.1 MiB | 147.8 MiB | 135.9 MiB | Read time was unchanged or slightly faster. ### Are these changes tested? Yes. ### Are there any user-facing changes? Lower memory use when reading dictionary-encoded columns. There is no API change. Closes #1446. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
