adriangb commented on PR #10549: URL: https://github.com/apache/arrow-rs/pull/10549#issuecomment-5183278307
## Update: there is now a writer-level benchmark that resolves this My comment above concluded that the justification rested on an out-of-tree microbenchmark, because the existing 2 MiB-value benches degenerate to `PLAIN` at the default page limit and cannot see the change. That gap is now filled by https://github.com/apache/arrow-rs/pull/10550, which adds 1 KiB-value benches — small enough to share a data page, so the encoder's previous-value state survives and the shared-prefix scan runs to real depth. Measured against those, A/B/A with the baseline run twice, means in ms, aarch64. The `plain` rows are controls (`PLAIN` never calls `common_prefix_length`): | bench | base (pre) | this PR | base (post) | | | --- | --- | --- | --- | --- | | `small_string_shared_prefix/plain` (control) | 0.683 | 0.694 | 0.699 | flat | | `small_string_shared_prefix/delta_byte_array` | 2.832 | **0.682** | 2.870 | **4.2x faster** | | `small_string_partial_prefix/plain` (control) | 0.603 | 0.853 | 0.639 | noisy | | `small_string_partial_prefix/delta_byte_array` | 1.876 | **0.703** | 1.925 | **2.7x faster** | | `small_string_distinct/plain` (control) | 0.454 | 0.471 | 0.474 | flat | | `small_string_distinct/delta_byte_array` | 0.680 | 0.710 | 0.699 | flat | The distinct case being flat is the expected outcome — there is no shared prefix to scan. The `small_string_partial_prefix/plain` control had one noisy reading (0.853 against baselines of 0.603 / 0.639), so treat that row's noise floor as wider; the effect is still several times larger than the excursion. Worth pulling out of the baseline column: on `main`, the shared-prefix column costs **2.83 ms with `DELTA_BYTE_ARRAY` versus 0.68 ms with `PLAIN`** — the encoding is about 4x more expensive than storing the values in full, on exactly the data it exists for. With this change it reaches parity with `PLAIN` while still deduplicating. I would suggest reviewing https://github.com/apache/arrow-rs/pull/10550 first; this PR is then a one-function change measurable against benchmarks already on `main`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
