lichuang opened a new pull request, #12353:
URL: https://github.com/apache/arrow-rs/pull/12353

   # Which issue does this PR close?
   
   - Part of #11717
   - Part of #9298
   
   # Rationale for this change
   
   `#12228` moved the primitive, boolean, byte and fixed-size-binary builders 
off `ArrayData`, leaving the remaining builders on the old path:
   
   ```text
   values + nulls
       → ArrayDataBuilder
       → ArrayData { buffers: Vec<Buffer> }
       → typed array
   ```
   
   Every `finish` in these builders still pays for a `Vec<Buffer>`, the 
`ArrayData` construction, buffer-count checks, and the `Vec` teardown. The 
overhead is a fixed cost per array, so it dominates when callers finish many 
small arrays.
   
   `NullBuilder` was the most extreme case: building a `NullArray` of any 
length does not need any buffer at all, yet it went through `ArrayData`.
   
   # What changes are included in this PR?
   
   - `NullBuilder::finish` / `finish_cloned` now construct 
`NullArray::new(len)` directly.
   - `MapBuilder::finish_helper` now builds `OffsetBuffer` and calls 
`MapArray::new_unchecked` instead of assembling `ArrayData`. The now-derivable 
`len` parameter was removed from `finish_helper` and its three callers.
   - The three dictionary builders (`PrimitiveDictionaryBuilder`, 
`GenericByteDictionaryBuilder`, `FixedSizeBinaryDictionaryBuilder`) now call 
`DictionaryArray::new_unchecked(keys, Arc::new(values))` in `finish`, 
`finish_cloned`, and `finish_preserve_values` instead of building data types 
and child data through `ArrayDataBuilder`.
   
   The dictionary builders no longer construct the dictionary `DataType` 
themselves; `new_unchecked` derives it from the keys and values arrays.
   
   Imports that became unused (`arrow_data::ArrayData`, `ArrayDataBuilder`, and 
production-side `DataType`) were removed. Tests that still need `DataType` 
import it locally.
   
   This PR intentionally does not change `UnionBuilder`, which needs dynamic 
dispatch from `DataType` to a typed constructor.
   
   # Are these changes tested?
   
   The existing builder suites cover `finish`, `finish_cloned`, 
`finish_preserve_values`, builder reuse, null handling, empty maps, and 
dictionary key overflow. The full `arrow-array` test suite passes, including 
doctests, as does the `force_validate` configuration and the `arrow` array 
transform tests. Clippy and formatting checks pass.
   
   A new `arrow-array/benches/builder_finish.rs` benchmark measures the whole 
build-then-finish path for these builders at 1, 8 and 64 elements, where the 
fixed `finish` cost dominates. Local numbers (release, criterion):
   
   | Bench | Before | After | Change |
   |---|---:|---:|---:|
   | `null_builder/len:8` | 45.1 ns | 342 ps | -99% |
   | `map_builder/len:1` | 1.018 µs | 586 ns | -42% |
   | `map_builder/len:64` | 1.471 µs | 1.048 µs | -29% |
   | `primitive_dictionary/len:1` | 381 ns | 210 ns | -45% |
   | `primitive_dictionary/len:64` | 808 ns | 628 ns | -22% |
   | `byte_dictionary/len:1` | 602 ns | 410 ns | -32% |
   | `fixed_size_binary_dictionary/len:1` | 513 ns | 305 ns | -41% |
   | `fixed_size_binary_dictionary/len:64` | 1.47 µs | 1.03 µs | -30% |
   
   The gain shrinks as the array grows because appending values starts to 
dominate. These are local numbers; the CI benchmark for `builder_finish` is the 
authoritative measurement.
   
   # Are there any significant user-facing changes?
   
   No public API changes. The output arrays keep the same data types, values, 
nulls, and builder-reset behavior. `MapBuilder::finish_helper` is private, so 
removing its `len` parameter is not user-visible.
   
   # AI usage disclosure
   
   This change and its tests were prepared with AI assistance. I reviewed the 
implementation and verified the tests, benchmark, Clippy, and formatting 
locally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to