abstractdog opened a new pull request, #3770:
URL: https://github.com/apache/parquet-java/pull/3770
### Rationale for this change
`RunLengthBitPackingHybridDecoder.readNext()` allocates a fresh `int[]`, a
fresh `byte[]`, and a new `DataInputStream` wrapper on every bit-packed run.
It's the primary decode path for def/rep levels and dictionary indices on every
column read, so those allocations directly inflate young-gen GC on scan-heavy
workloads. The code has even flagged it: `// TODO: reuse a buffer`.
### What changes are included in this PR?
- `RunLengthBitPackingHybridDecoder`: reuse `int[] currentBuffer` and
`byte[] packedBuffer` as grow-only instance fields; track the current run size
in a new `packedRunSize` field (so `readInt()` no longer uses
`currentBuffer.length` as the boundary); zero-fill the tail on short
end-of-stream reads to preserve the old `new byte[]` implicit-zero semantics;
read directly from the underlying `InputStream` instead of wrapping in
`DataInputStream`.
- New `RleDecodingBenchmark` in `parquet-benchmarks` to lock in the
improvement and guard against regressions.
### Are these changes tested?
Yes. All 692 tests in `parquet-column` pass, including
`TestRunLengthBitPackingHybridEncoder` (round-trips through the decoder) and
`TestDictionary` (exercises dictionary-index decoding, which is the primary
caller). The new JMH benchmark also verifies the behavior end-to-end via encode
→ decode round trips.
JMH results (100K values decoded, JDK 17, `bitWidth ∈ {1, 3, 10}`):
| Metric | Before | After |
|---|---:|---:|
| Alloc/value | 4.9–5.9 B/op | 0.055–0.060 B/op (~90× less) |
| Alloc rate | ~4.2–4.7 GB/s | ~55–62 MB/s (~70× less) |
| Young-gen GCs/trial | 55–63 | ≈ 0 |
| Throughput | 830–910 M vals/s | 1.06–1.18 B vals/s (+21–28%) |
### Are there any user-facing changes?
No. Purely internal implementation change to a single decoder class.
<!-- Please uncomment the line below and replace ${GITHUB_ISSUE_ID} with the
actual Github issue id. -->
<!-- Closes #3769 -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
