Would you guys be interested in perhaps having a Hangout with my team from JICS/NICS?
We have some major experts and research thrusts in this area (code optimizations for Intel chipsets, MKL and other kernels, memory/IO optimizations, etc) We are a research shop. People just get excited over things like this. On Wednesday, March 2, 2016, Wes McKinney <w...@cloudera.com> wrote: > hi Edmon, > > Since Arrow arrays are arranged with like-data in contiguous memory > regions (for example, in an array of strings, the UTF8 bytes are all > laid out in contiguous memory -- see > https://github.com/apache/arrow/blob/master/format/Layout.md), it is > cache-friendly for scan operations and amenable to SIMD computations > (for example: SIMD-accelerated hash functions). This is especially > important for nested data, as all the "leaf nodes" in a nested > structure generally contain contiguous memory. > > We have not started doing this yet, but it would be useful to begin > assembling kernels that use CPU intrinsics (and SSE/AVX) in the Arrow > codebase, and to make them easily accessible. Having a standard > benchmark suite and other performance experimentation tools available > for users to run on their hardware would also be great. > > best, > Wes > > On Wed, Mar 2, 2016 at 10:21 AM, Edmon Begoli <ebeg...@gmail.com > <javascript:;>> wrote: > > Hey folks, > > > > How could I get more details on what and how Arrow uses Intel CPUs for > > whatever computational advantage? > > > > At JICS, we run very large experimental Intel HPC systems, and I would > like > > to learn how can we possibly run some interesting Arrow on Intel CPUs > > experiments. > > > > Thank you, > > Edmon >
