nevi-me commented on pull request #715:
URL: https://github.com/apache/arrow-rs/pull/715#issuecomment-917619317
@alamb reviewing the individual commits helped me make more sense of it. In
generic functions where we use `T: ArrowNativeType / ArrowPrimitiveType &
T::Native ...`, we end up compiling the same function n times for say i64,
because i64 can have different logical types.
So reducing some of these functions down to only their physical types
instead of logical types, reduces the amount of code that's generated.
Here's the output on the latest tip:
```
2294551 (100%) 46197 (100%) (TOTAL)
133456 (5.8%) 238 (0.5%) core::slice::sort::partition_in_blocks
76896 (3.4%) 316 (0.7%)
arrow::buffer::mutable::MutableBuffer::try_from_trusted_len_iter
56620 (2.5%) 1044 (2.3%) core::option::Option<T>::map
56086 (2.4%) 594 (1.3%) core::iter::traits::iterator::Iterator::fold
52090 (2.3%) 238 (0.5%) core::slice::sort::partition
43760 (1.9%) 238 (0.5%) core::slice::sort::partition_equal
41491 (1.8%) 136 (0.3%) arrow::util::trusted_len::trusted_len_unzip
38998 (1.7%) 238 (0.5%) core::slice::sort::shift_tail
37836 (1.6%) 714 (1.5%) core::slice::sort::choose_pivot::{{closure}}
35099 (1.5%) 119 (0.3%) core::slice::sort::recurse
29036 (1.3%) 238 (0.5%) core::slice::sort::choose_pivot
28322 (1.2%) 238 (0.5%)
core::iter::traits::iterator::Iterator::reduce
27612 (1.2%) 78 (0.2%) arrow::compute::kernels::take::take_primitive
26853 (1.2%) 139 (0.3%)
arrow::array::array_primitive::PrimitiveArray<T>::from_trusted_len_iter
26763 (1.2%) 240 (0.5%)
core::iter::traits::iterator::Iterator::try_fold
26175 (1.1%) 119 (0.3%) core::slice::sort::partition_at_index_loop
25228 (1.1%) 119 (0.3%) core::slice::sort::partition_at_index
23679 (1.0%) 293 (0.6%) <alloc::vec::Vec<T> as
alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
22920 (1.0%) 253 (0.5%)
<core::iter::adapters::enumerate::Enumerate<I> as
core::iter::traits::iterator::Iterator>::fold::enumerate::{{closure}}
19626 (0.9%) 296 (0.6%) <alloc::vec::Vec<T,A> as
alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
19208 (0.8%) 338 (0.7%)
core::iter::adapters::map::map_fold::{{closure}}
19201 (0.8%) 209 (0.5%)
<core::iter::adapters::enumerate::Enumerate<I> as
core::iter::traits::iterator::Iterator>::next
19040 (0.8%) 119 (0.3%) core::slice::sort::heapsort
18309 (0.8%) 119 (0.3%) core::slice::sort::shift_head
17782 (0.8%) 338 (0.7%) <core::iter::adapters::map::Map<I,F> as
core::iter::traits::iterator::Iterator>::fold
17380 (0.8%) 316 (0.7%)
arrow::buffer::immutable::Buffer::try_from_trusted_len_iter
17344 (0.8%) 64 (0.1%)
arrow::compute::kernels::cast::pack_numeric_to_dictionary
```
After:
```
Lines Copies Function name
----- ------ -------------
1583106 (100%) 37652 (100%) (TOTAL)
44504 (2.8%) 804 (2.1%) core::option::Option<T>::map
41491 (2.6%) 136 (0.4%) arrow::util::trusted_len::trusted_len_unzip
32516 (2.1%) 58 (0.2%) core::slice::sort::partition_in_blocks
31084 (2.0%) 360 (1.0%) core::iter::traits::iterator::Iterator::fold
27612 (1.7%) 78 (0.2%) arrow::compute::kernels::take::take_primitive
26853 (1.7%) 139 (0.4%)
arrow::array::array_primitive::PrimitiveArray<T>::from_trusted_len_iter
26763 (1.7%) 240 (0.6%)
core::iter::traits::iterator::Iterator::try_fold
24760 (1.6%) 124 (0.3%)
arrow::buffer::mutable::MutableBuffer::try_from_trusted_len_iter
20403 (1.3%) 239 (0.6%) <alloc::vec::Vec<T> as
alloc::vec::spec_from_iter_nested::SpecFromIterNested<T,I>>::from_iter
18918 (1.2%) 206 (0.5%)
<core::iter::adapters::enumerate::Enumerate<I> as
core::iter::traits::iterator::Iterator>::next
17344 (1.1%) 64 (0.2%)
arrow::compute::kernels::cast::pack_numeric_to_dictionary
16292 (1.0%) 284 (0.8%)
core::iter::adapters::map::map_fold::{{closure}}
14956 (0.9%) 284 (0.8%) <core::iter::adapters::map::Map<I,F> as
core::iter::traits::iterator::Iterator>::fold
14730 (0.9%) 242 (0.6%) <alloc::vec::Vec<T,A> as
alloc::vec::spec_extend::SpecExtend<T,I>>::spec_extend
14641 (0.9%) 37 (0.1%) arrow::array::array::print_long_array
14402 (0.9%) 37 (0.1%)
<arrow::array::array_string::GenericStringArray<OffsetSize> as
core::iter::traits::collect::FromIterator<core::option::Option<Ptr>>>::from_iter
14036 (0.9%) 76 (0.2%)
<arrow::array::array_primitive::PrimitiveArray<T> as
core::iter::traits::collect::FromIterator<Ptr>>::from_iter
12885 (0.8%) 335 (0.9%)
core::iter::traits::iterator::Iterator::for_each
12722 (0.8%) 58 (0.2%) core::slice::sort::partition
12393 (0.8%) 84 (0.2%)
arrow::buffer::mutable::MutableBuffer::extend_from_iter
12256 (0.8%) 64 (0.2%)
arrow::array::builder::PrimitiveDictionaryBuilder<K,V>::append
12066 (0.8%) 698 (1.9%) core::iter::adapters::map::Map<I,F>::new
11294 (0.7%) 86 (0.2%) <core::iter::adapters::ResultShunt<I,E> as
core::iter::traits::iterator::Iterator>::try_fold::{{closure}}
10787 (0.7%) 67 (0.2%) alloc::raw_vec::RawVec<T,A>::grow_amortized
10692 (0.7%) 58 (0.2%) core::slice::sort::partition_equal
9864 (0.6%) 3 (0.0%) arrow::compute::kernels::take::take_impl
9615 (0.6%) 76 (0.2%) <arrow::buffer::immutable::Buffer as
core::iter::traits::collect::FromIterator<T>>::from_iter
```
Benchmarks are still running, but no changes seen so far. I'll post the cast
and sort ones when they're done running
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
