zeroshade opened a new pull request, #1554: URL: https://github.com/apache/arrow-go/pull/1554
This builds on #1553 and includes its commit until that one is merged. Only the last commit is new. ### Rationale for this change When no scaling fits a vector, the best one still encodes a few values. Those few lie far apart, so they pack at nearly the full integer width, and every other value is stored as an exception on top. The vector comes out larger than its values stored verbatim: random doubles take about 97 bits per value, where `PLAIN` takes 64. ### What changes are included in this PR? A vector that is larger packed than it would be with every value stored as an exception is written that way: a bit width of zero and one exception for each value. That bounds every vector at an exception's size per value: 80 bits for `DOUBLE` and 48 for `FLOAT`. `EstimatedDataEncodedSize` uses the same bound for the vector it has buffered, instead of the larger one it needed before. Bits per value for a page of 131,072 doubles: | Data | Before | After | |---|---|---| | Uniform random in [0, 1) | 97.2 | 80.1 | | Two decimals, 30% of the values random | 44.2 | 44.2 | | Two decimals | 20.1 | 20.1 | ### Are these changes tested? Yes. ### Are there any user-facing changes? Pages of data that ALP cannot compress get smaller. Vectors that pack smaller than their exceptions are written as before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
