Two other things to keep in mind: * Arrow's Sum does a bit of extra work to support different behaviors when nulls are present (this should generally be trivial, but as Niranda notes for short arrays it may be noticeable) * Arrow's Sum does a pairwise summation for floating-point values which reduces numeric error compared to a naive sum
On Thu, Mar 3, 2022, at 08:48, Niranda Perera wrote: > Hi Gyozo, > > If the arrays are small, then using arrow::compute::Sum might not give you > huge performance improvement. IMO the advantage would be, the code base would > be more organized. > > IINM by default Sum benefits from SSE2 optimizations. But there are kernels > implementations for AVX/AVX512 as well. But for these make real impact, you'd > need a sizeable array. If the array size is small, it's possible that you'd > be working within cache or even a single cache line. So I doubt vectorization > would help there. > > Best > > On Thu, Mar 3, 2022, 05:30 Győző Gáspár <[email protected]> wrote: >> Hi All, >> >> I am summing values like: >> >> #+BEGIN_SRC >> scalar->value = scalar->value + array.Value(row); >> #+END_SRC >> >> I plan to change this to use arrow compute API instead. The arrays I am >> summing up are small, with ten or fewer items inside. >> >> My questions: >> >> - Can I expect arrow compute to be faster? >> - Are there general guidelines? What is the minimal array size when the >> compute API is faster than just summing values? >> - Do I need specialized hardware to speed up arrow compute? >> >> -- >> Győző Gáspár >> Software Engineer >> >> Tenzir GmbH >> Nagelsweg 41 >> 20097 Hamburg >> >> https://tenzir.com >> +49 40 209337260 >> >> Commercial Register Hamburg: HRB 148081 >> VAT Number: DE314897407 >> CEO: Matthias Vallentin, PhD
