zeroshade opened a new pull request, #1554:
URL: https://github.com/apache/arrow-go/pull/1554

   This builds on #1553 and includes its commit until that one is merged. Only 
the last commit is new.
   
   ### Rationale for this change
   
   When no scaling fits a vector, the best one still encodes a few values. 
Those few lie far apart, so they pack at nearly the full integer width, and 
every other value is stored as an exception on top. The vector comes out larger 
than its values stored verbatim: random doubles take about 97 bits per value, 
where `PLAIN` takes 64.
   
   ### What changes are included in this PR?
   
   A vector that is larger packed than it would be with every value stored as 
an exception is written that way: a bit width of zero and one exception for 
each value. That bounds every vector at an exception's size per value: 80 bits 
for `DOUBLE` and 48 for `FLOAT`.
   
   `EstimatedDataEncodedSize` uses the same bound for the vector it has 
buffered, instead of the larger one it needed before.
   
   Bits per value for a page of 131,072 doubles:
   
   | Data | Before | After |
   |---|---|---|
   | Uniform random in [0, 1) | 97.2 | 80.1 |
   | Two decimals, 30% of the values random | 44.2 | 44.2 |
   | Two decimals | 20.1 | 20.1 |
   
   ### Are these changes tested?
   
   Yes.
   
   ### Are there any user-facing changes?
   
   Pages of data that ALP cannot compress get smaller. Vectors that pack 
smaller than their exceptions are written as before.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to