mightsleep opened a new pull request, #11435: URL: https://github.com/apache/arrow-rs/pull/11435
# Which issue does this PR close? None. Depends on #11434 (benchmarks); the first commit is from that PR. # Rationale for this change FixedSizeList is the only list type that falls through to concat_fallback, which builds an ArrayData per input. With many short arrays that overhead is most of the cost, and that is the common case: DataFusion's ScalarValue::iter_to_array (GroupsAccumulatorAdapter::evaluate, first_value/last_value) and Rerun's chunk batcher concatenate one-row arrays. List and LargeList already have concat_lists. # What changes are included in this PR? concat_fixed_size_list concatenates the child values and the validity bitmaps directly. Equal list types imply equal child types, so the child concat skips the type check: concat is split into the check and concat_same_type. # Are these changes tested? Yes, test_concat_fixed_size_list_matches_fallback compares against concat_fallback and runs validate_full: slices, mixed nulls, nested lists, size 0, 64 one-row arrays. | benchmark | main | this PR | speedup | |---|---|---|---| | concat 1024 arrays fixed size list f32x3 1 | 194.8 µs | 13.6 µs | 14x | | concat 1024 arrays fixed size list f32x3 nulls 1 | 208.1 µs | 16.2 µs | 12.8x | | concat fixed size lists (two large arrays) | 207.7 µs | 211.6 µs | within noise | # Are there any significant user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
