Re: [C++] Runtime SIMD dispatching for Arrow

2020-05-12 Thread Micah Kornfield
> > Since I develop on an AVX512-capable machine, if we have runtime > dispatching then it should be able to test all variants of a function > from a single executable / test run rather than having to produce > multiple builds and test them separately, right? Yes, but I think the same of true

Re: [C++] Runtime SIMD dispatching for Arrow

2020-05-12 Thread Wes McKinney
On Tue, May 12, 2020 at 9:47 PM Yibo Cai wrote: > > Thanks Wes, I'm glad to see this feature coming. > > From history talks, the main concern is runtime dispatcher may cause > performance issue. > Personally, I don't think it's a big problem. If we're using SIMD, it must be > targeting some

Re: [C++] Runtime SIMD dispatching for Arrow

2020-05-12 Thread Wes McKinney
On Tue, May 12, 2020 at 10:19 PM Micah Kornfield wrote: > > Hi Wes, > I think you highlighted the two issues well, but I think they are somewhat > orthogonal and runtime dispatching only addresses the binary availability > of the optimizations (but actually makes testing harder because it can >

Re: [C++] Runtime SIMD dispatching for Arrow

2020-05-12 Thread Micah Kornfield
Hi Wes, I think you highlighted the two issues well, but I think they are somewhat orthogonal and runtime dispatching only addresses the binary availability of the optimizations (but actually makes testing harder because it can potentially hide untested code paths). Personally, I think it is

Re: [C++] Runtime SIMD dispatching for Arrow

2020-05-12 Thread Yibo Cai
Thanks Wes, I'm glad to see this feature coming. From history talks, the main concern is runtime dispatcher may cause performance issue. Personally, I don't think it's a big problem. If we're using SIMD, it must be targeting some time consuming code. But we do need to take care some issues.

RE: [C++] Runtime SIMD dispatching for Arrow

2020-05-12 Thread Du, Frank
Hi, I totally agree that arrow should has a built-in support for runtime dispatching facilities just like other popular computing libs to fully utilize the modern hardware capacity, we feel arrow has great potential performance chance with the advanced cpu SIMD feature. It's ok for me to

[C++] Runtime SIMD dispatching for Arrow

2020-05-12 Thread Wes McKinney
hi, We've started to receive a number of patches providing SIMD operations for both x86 and ARM architectures. Most of these patches make use of compiler definitions to toggle between code paths at compile time. This is problematic for a few reasons: * Binaries that are shipped (e.g. in Python)

Re: [DISCUSS] Need for Arrow 0.17.1 patch release (binary only?)

2020-05-12 Thread Krisztián Szűcs
Just pushed the maint-0.17.x branch - I had conflicts here and there. Applied the following git commands: https://gist.github.com/kszucs/ea75f09090a9ffdff07e51582af1f436 Submitted the corssbow packaging tasks to see nothing is missing. I'll start with cutting RC0 tomorrow. On Mon, May 11, 2020

[jira] [Created] (ARROW-8779) [R] Unable to write Struct Layout to file (.arrow, .parquet)

2020-05-12 Thread Dominic Dennenmoser (Jira)
Dominic Dennenmoser created ARROW-8779: -- Summary: [R] Unable to write Struct Layout to file (.arrow, .parquet) Key: ARROW-8779 URL: https://issues.apache.org/jira/browse/ARROW-8779 Project:

[jira] [Created] (ARROW-8778) [C++][gandiva] SelectionVector related test failed on big-endian platforms

2020-05-12 Thread Kazuaki Ishizaki (Jira)
Kazuaki Ishizaki created ARROW-8778: --- Summary: [C++][gandiva] SelectionVector related test failed on big-endian platforms Key: ARROW-8778 URL: https://issues.apache.org/jira/browse/ARROW-8778

[jira] [Created] (ARROW-8777) [Rust] Parquet.rs does not support reading fixed-size binary fields.

2020-05-12 Thread Max Burke (Jira)
Max Burke created ARROW-8777: Summary: [Rust] Parquet.rs does not support reading fixed-size binary fields. Key: ARROW-8777 URL: https://issues.apache.org/jira/browse/ARROW-8777 Project: Apache Arrow

[jira] [Created] (ARROW-8776) [FlightRPC][C++] Flight/C++ middleware don't receive headers on failed calls to Java servers

2020-05-12 Thread David Li (Jira)
David Li created ARROW-8776: --- Summary: [FlightRPC][C++] Flight/C++ middleware don't receive headers on failed calls to Java servers Key: ARROW-8776 URL: https://issues.apache.org/jira/browse/ARROW-8776

[jira] [Created] (ARROW-8775) [C++][FlightRPC] Integration client doesn't run integration tests

2020-05-12 Thread David Li (Jira)
David Li created ARROW-8775: --- Summary: [C++][FlightRPC] Integration client doesn't run integration tests Key: ARROW-8775 URL: https://issues.apache.org/jira/browse/ARROW-8775 Project: Apache Arrow

[Rust] DataFusion threading model design discussion

2020-05-12 Thread Andy Grove
As part of ARROW-8774 [1], I have created a Google doc [2] where any interested parties can collaborate on a design discussion, which I will then document in the JIRA. [1] https://issues.apache.org/jira/browse/ARROW-8774 [2]

Re: [NIGHTLY] Arrow Build Report for Job nightly-2020-05-12-0

2020-05-12 Thread Neal Richardson
The r-as-cran failure is spurious; I'm working on a fix on https://issues.apache.org/jira/browse/ARROW-8768. Neal On Tue, May 12, 2020 at 3:11 AM Crossbow wrote: > > Arrow Build Report for Job nightly-2020-05-12-0 > > All tasks: >

[jira] [Created] (ARROW-8774) [Rust] [DataFusion] Improve threading model

2020-05-12 Thread Andy Grove (Jira)
Andy Grove created ARROW-8774: - Summary: [Rust] [DataFusion] Improve threading model Key: ARROW-8774 URL: https://issues.apache.org/jira/browse/ARROW-8774 Project: Apache Arrow Issue Type:

[jira] [Created] (ARROW-8773) pyarrow schema.empty_table() does not preserve nullability of fields

2020-05-12 Thread Al Taylor (Jira)
Al Taylor created ARROW-8773: Summary: pyarrow schema.empty_table() does not preserve nullability of fields Key: ARROW-8773 URL: https://issues.apache.org/jira/browse/ARROW-8773 Project: Apache Arrow

[NIGHTLY] Arrow Build Report for Job nightly-2020-05-12-0

2020-05-12 Thread Crossbow
Arrow Build Report for Job nightly-2020-05-12-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-05-12-0 Failed Tasks: - test-conda-python-3.7-spark-master: URL:

[jira] [Created] (ARROW-8772) [C++] Expand SumKernel benchmark to more types

2020-05-12 Thread Frank Du (Jira)
Frank Du created ARROW-8772: --- Summary: [C++] Expand SumKernel benchmark to more types Key: ARROW-8772 URL: https://issues.apache.org/jira/browse/ARROW-8772 Project: Apache Arrow Issue Type: