[jira] [Created] (ARROW-10995) [Rust] [DataFusion] Improve parallelism when reading Parquet files
Andy Grove created ARROW-10995: -- Summary: [Rust] [DataFusion] Improve parallelism when reading Parquet files Key: ARROW-10995 URL: https://issues.apache.org/jira/browse/ARROW-10995 Project: Apache Arrow Issue Type: Improvement Components: Rust - DataFusion Reporter: Andy Grove Currently the unit of parallelism is the number of parquet files being read. For example, if we run a query against a Parquet table that consists of 8 partitions then we will attempt to run 8 async tasks in parallel and if there is a single Parquet file then we will only try and run 1 async task so this does not scale well. A better approach would be to have one parallel task per "chunk" in each parquet file. This would involve an upfront step in the planner to scan the parquet meta-data to get a list of chunks and then split these up between the configured number of parallel tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-10994) [Rust] Fix bugs in TPC-H file conversion
Andy Grove created ARROW-10994: -- Summary: [Rust] Fix bugs in TPC-H file conversion Key: ARROW-10994 URL: https://issues.apache.org/jira/browse/ARROW-10994 Project: Apache Arrow Issue Type: Bug Components: Rust Reporter: Andy Grove Assignee: Andy Grove Fix For: 3.0.0 The utility in tpch for converting tbl files to CSV/Parquet no longer works now that we have multiple tables. The UX is also terrible and the tool only supports generating uncompressed parquet files. I'm going to create one PR to fix these things to make this more usable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-10993) [CI][macOS] Fix Python 3.9 installation by Homebrew
Kouhei Sutou created ARROW-10993: Summary: [CI][macOS] Fix Python 3.9 installation by Homebrew Key: ARROW-10993 URL: https://issues.apache.org/jira/browse/ARROW-10993 Project: Apache Arrow Issue Type: Improvement Components: Continuous Integration Reporter: Kouhei Sutou Assignee: Kouhei Sutou https://github.com/apache/arrow/runs/1579780011#step:4:520 {noformat} ==> Pouring python@3.9-3.9.1_1.catalina.bottle.tar.gz Error: The `brew link` step did not complete successfully The formula built, but is not symlinked into /usr/local Could not symlink bin/2to3 Target /usr/local/bin/2to3 already exists. You may want to remove it: rm '/usr/local/bin/2to3' To force the link and overwrite all conflicting files: brew link --overwrite python@3.9 To list all files that would be deleted: brew link --overwrite --dry-run python@3.9 Possible conflicting files are: /usr/local/bin/2to3 -> /Library/Frameworks/Python.framework/Versions/2.7/bin/2to3 {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-10992) [C++] Arrow Cmake/-march compile flags conflict with Intel compiler (icc/icpc)
Daniel Jewell created ARROW-10992: - Summary: [C++] Arrow Cmake/-march compile flags conflict with Intel compiler (icc/icpc) Key: ARROW-10992 URL: https://issues.apache.org/jira/browse/ARROW-10992 Project: Apache Arrow Issue Type: Bug Components: C++ Affects Versions: 2.0.0 Reporter: Daniel Jewell Compiler support for Intel ICC/ICPC was added in ARROW-10849 However, there are still a few cases where {code:java} -march={code} is being added to C/CXXFLAGS. While this kinda-sorta works, for icc/icpc there is different set of optimization options (I use " -xCORE-AVX2" typically). The list of optimization options is quite verbose (well beyond what just the flags show) and this really needs the eyes of someone who is an expert on just what exactly happens to the generated code with the specific flags. Specific warnings: {code:java} icpc: command line warning #10121: overriding '-xCORE-AVX2' with '-march=skylake-avx512' icpc: command line warning #10121: overriding '-xCORE-AVX2' with '-march=skylake-avx512' icpc: command line warning #10006: ignoring unknown option '-mbmi2' icpc: command line warning #10121: overriding '-xCORE-AVX2' with '-march=haswell' icpc: command line warning #10121: overriding '-xCORE-AVX2' with '-march=haswell' {code} The haswell warning appears to come from the compilation of cpp/src/arrow/util/bpacking_avx2.cc See: [https://software.intel.com/content/www/us/en/develop/articles/performance-tools-for-software-developers-intel-compiler-options-for-sse-generation-and-processor-specific-optimizations.html] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-10991) [Rust]: Valgrind warnings in filter kernel
Jörn Horstmann created ARROW-10991: -- Summary: [Rust]: Valgrind warnings in filter kernel Key: ARROW-10991 URL: https://issues.apache.org/jira/browse/ARROW-10991 Project: Apache Arrow Issue Type: Bug Components: Rust Reporter: Jörn Horstmann {code} ==42927== Conditional jump or move depends on uninitialised value(s) ==42927==at 0x384DE0: arrow::compute::kernels::filter::FilterContext::filter (in /home/jhorstmann/Source/github/apache/arrow/rust/target/release/deps/arrow-421088df9db9800e) ==42927==by 0x396394: arrow::compute::kernels::filter::filter (in /home/jhorstmann/Source/github/apache/arrow/rust/target/release/deps/arrow-421088df9db9800e) ==42927==by 0x34B786: core::ops::function::FnOnce::call_once (in /home/jhorstmann/Source/github/apache/arrow/rust/target/release/deps/arrow-421088df9db9800e) {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-10990) [Rust]: SIMD implementation of compare kernels reads out of bounds
Jörn Horstmann created ARROW-10990: -- Summary: [Rust]: SIMD implementation of compare kernels reads out of bounds Key: ARROW-10990 URL: https://issues.apache.org/jira/browse/ARROW-10990 Project: Apache Arrow Issue Type: Bug Components: Rust Reporter: Jörn Horstmann The simd compare kernels use the following pattern to handle the remainder that is not a multiple of the number of vector lanes: {code} if rem > 0 { let simd_left = T::load(left.value_slice(len - rem, lanes)); let simd_right = T::load(right.value_slice(len - rem, lanes)); let simd_result = op(simd_left, simd_right); let rem_buffer_size = (rem as f32 / 8f32).ceil() as usize; T::bitmask(&simd_result, |b| { result.extend_from_slice(&b[0..rem_buffer_size]); }); } {code} While this avoids writing into result out of bounds, it still reads from the {{left}} and {{right}} arrays at out of bounds indices and valgrind complains about that. I propose to rewrite the logic to use chunked iteration, with a scalar loop for the remainder, similar to the change for arithmetic kernels in ARROW-10914. -- This message was sent by Atlassian Jira (v8.3.4#803005)