dtenedor opened a new pull request, #12273:
URL: https://github.com/apache/arrow-rs/pull/12273

   # Which issue does this PR close?
   
   None.
   
   # Rationale for this change
   
   The column-level delta byte array encoders, which the non-Arrow column 
writer API uses, do extra
   work for every value:
   - `DeltaLengthByteArrayEncoder` keeps a `ByteArray` clone of every value, an 
atomic increment and
     decrement each, and copies the bytes at flush.
   - Both encoders allocate temporary `Vec`s on every `put`.
   - `DeltaByteArrayEncoder` copies every value into its previous-value buffer 
and slices a new
     `ByteArray` for every suffix.
   
   The Arrow writer has its own byte array encoders and is not affected.
   
   # What changes are included in this PR?
   
   - Values are copied once, as they are put, into a single byte buffer that is 
reserved from the
     input's total length.
   - Lengths are encoded in stack batches of 128.
   - `DeltaByteArrayEncoder` compares each value with its predecessor in place 
and keeps only the last
     value of each `put`.
   
   Output is byte-identical.
   
   The first commit adds the benchmark cases, so it can serve as the benchmark 
baseline. #12272 starts
   with the same commit.
   
   # Are these changes tested?
   
   Covered by the existing encoder tests. Output was also checked to be 
byte-identical to main's for
   DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY over several datasets, value 
lengths and `put`
   chunkings.
   
   The first commit adds delta byte array cases to the `encoding` bench:
   - 16K sorted keys with shared prefixes
   - 16K random strings of 8-31 bytes
   - 1K values of 1 KiB
   
   The encoder is reused across iterations. Measured on an Intel Xeon Platinum 
8559C, default x86-64
   target, median of 5 alternating rounds:
   
   | `encoding` bench | Speedup |
   | :--- | ---: |
   | DELTA_BYTE_ARRAY encode: random / sorted_keys / large_1k | 3.26x / 2.27x / 
1.70x |
   | DELTA_LENGTH_BYTE_ARRAY encode: random / sorted_keys | 2.90x / 1.90x |
   | **DELTA_LENGTH_BYTE_ARRAY encode: large_1k** | **0.78x** |
   | decode, all cases | 0.99-1.01x |
   
   DELTA_LENGTH_BYTE_ARRAY gets slower on large values because their bytes are 
now copied twice: once
   at `put` and again at flush. Before, `put` only took a clone. The Arrow 
writer's fallback encoder
   already works this way.
   
   # Are there any significant user-facing changes?
   
   `estimated_memory_size` of `DeltaLengthByteArrayEncoder` now counts the byte 
buffer's capacity.
   Before, it counted the number of buffered values.
   
   # AI usage disclosure
   
   An AI coding agent (Claude Code) generated the code, the benchmark cases and 
this description, and
   ran the benchmarks and the output checks. This PR is a draft until the 
submitter has reviewed it.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to