Stephen0421 opened a new pull request, #9307:
URL: https://github.com/apache/paimon/pull/9307
### Purpose
Eliminate redundant temporary allocations on the `BlobFormatWriter` close
path when
encoding the blob footer index.
`DeltaVarintCompressor.compress(long[])` previously materialized all
deltas into a
`LongArrayList` (one transient `long[N]`) and preallocated the output
buffer at 10
bytes per element. Combined with the caller's `LongArrayList.toArray()` (a
second
transient `long[N]`), close time added roughly `16N + 10N` bytes of
avoidable
allocation, and the `N * 10` preallocation could overflow `int` for very
large inputs.
Changes:
- Encode deltas in two passes: first compute the exact encoded size without
allocating, then encode into a precisely sized `byte[]`, eliminating
buffer
reallocation and the final `toByteArray()` copy.
- Bound the index size against `Integer.MAX_VALUE` explicitly instead of
relying on
`N * 10` arithmetic.
- Add `compressLongArrayList(LongArrayList)` to encode straight from the
list without
a `toArray()` copy.
- Switch `BlobFormatWriter.close()` to the list entry point.
Footer bytes, writer close/abort behavior, and the public
`compress(long[])`/`decompress(byte[])` semantics are unchanged.
### Test
- `DeltaVarintCompressorTest` (10012 tests):
- byte-for-byte equivalence between the array and list paths across
null/empty,
single-element, repeated, monotonic, negative-delta, `-1`/`-2`
sentinel,
`Long.MIN_VALUE`/`Long.MAX_VALUE`, and overflow-adjacent sequences,
plus
randomized sequences.
- fixed golden byte anchors for both paths (e.g. `{42}` → `0x54`,
`{80,50,90,80,70}` → `A0 01 3B 50 13 13`).
- existing round-trip and corruption tests.
- `BlobFormatWriterTest` (39 tests): raw/array/map golden-file byte tests
still pass,
confirming the compressed index, index length, and version are
byte-identical.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
