tom-eon opened a new pull request, #1027: URL: https://github.com/apache/arrow-go/pull/1027
### Rationale for this change The Parquet spec requires DELTA_BINARY_PACKED delta arithmetic to wrap in two's complement at the physical type's width ([Encodings.md](https://github.com/apache/parquet-format/blob/master/Encodings.md#delta-encoding-delta_binary_packed--5)). The encoder computed INT32 deltas in `int64`, so wide-range int32 columns (e.g. values spanning most of the int32 range) produced deltas outside the int32 range and miniblock bit-widths up to 34. parquet-cpp/pyarrow reject such pages as corrupt (`Unexpected end of stream: InitBlock EOF`), as does arrow-rs (`'min_delta' is too large`) — only arrow-go itself could read them back, because its decoder uses the same widened arithmetic. parquet-cpp explicitly declined to accept >32-bit INT32 miniblocks (apache/arrow#20374), so the writer must not emit them. ### What changes are included in this PR? `deltaBitPackEncoder[T].Put` now computes each delta at the physical type's width (`int64(in[idx] - T(enc.currentVal))`) so overflow wraps in two's complement as the spec requires, bounding miniblock bit-widths at 32 for INT32. INT64 encoding is unchanged (the arithmetic was already at type width). The decoder needs no change: its int64 accumulation with a final cast to `T` decodes wrapped deltas correctly, and it still reads previously written out-of-spec files. ### Are these changes tested? Yes. New `TestDeltaBitPackedDeltaOverflow` covers alternating `MinInt32`/`MaxInt32`, random full-range int32, a golden-bytes case for wrapped-delta encoding, and an int64 no-regression case. A new `checkDeltaBitPackedWidths` helper parses the encoded stream and asserts every miniblock bit-width is within the physical type's width — round-trip tests alone cannot catch this, since arrow-go tolerates its own out-of-spec output. The test fails on the previous encoder and passes with the fix. Also verified end-to-end: a file written via `pqarrow` with a wide-range int32 DELTA_BINARY_PACKED column is now read correctly by pyarrow, where it previously failed with `InitBlock EOF`. ### Are there any user-facing changes? DELTA_BINARY_PACKED INT32 pages written for wide-value-range columns are now spec-compliant and readable by parquet-cpp/pyarrow, arrow-rs, and other strict readers. Files previously written with the out-of-spec widths remain readable by arrow-go. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
