zeroshade commented on PR #1289:
URL: https://github.com/apache/arrow-go/pull/1289#issuecomment-6780086073
I did an independent review of this and pushed some fixes directly to the
branch, so please pull before making further changes:
- `85c4a970`, `be4b786d`: `SetData` rejects a header that claims more values
than the page holds, and offsets that point into the offset array. Panic
messages quote the column path.
- `ac12ed5c`: the decoder now checks the offset chain the spec requires: the
first offset equals the offset array size, and each vector ends where the next
begins. Before this, a corrupt offset could decode another vector's values
without an error. `Discard` now reports a page that encodes fewer values than
requested, as `Decode` already did.
- `eee34375`: two parameter-choice fixes and a benchmark fix:
- Sampling shifts its position within each block, so data that repeats
with the sampling stride can't fill the sample. With 32768 doubles and a NaN
every 128 values in one 32768-value vector, the page goes from 327,564 B to
59,928 B.
- When the preset shortlist does no better than PLAIN, the encoder
searches again in case the data changed scale, backing off while nothing
compresses. Integers followed by four-decimal prices drop from 10,254 B per
1024 values to under 3 KiB. On random doubles the cost is about 1.5% at 1M
values.
- The benchmark fixtures now produce the exception rates their names say.
Other things I checked that look good:
- Decoding matches the spec's normative rules, including float32 arithmetic
for FLOAT, which matches Arrow C++.
- The power-of-ten tables equal the decimal literals.
- Fuzzing each decoder for 2 minutes found no panics, and the tests pass on
linux/386.
- A pyarrow nightly reads Go-written ALP files bit-identically, including
NaN payloads, -0, infinities, subnormals and nulls.
- The footer encodings-list change only removes duplicates and lists the
encodings the pages actually used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
