[jira] [Created] (ARROW-4656) [Rust] Implement CSV Writer

2019-02-21 Thread Paddy Horan (JIRA)
Paddy Horan created ARROW-4656: -- Summary: [Rust] Implement CSV Writer Key: ARROW-4656 URL: https://issues.apache.org/jira/browse/ARROW-4656 Project: Apache Arrow Issue Type: Improvement

[jira] [Created] (ARROW-4655) [Packaging] Parallelize binary upload

2019-02-21 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4655: --- Summary: [Packaging] Parallelize binary upload Key: ARROW-4655 URL: https://issues.apache.org/jira/browse/ARROW-4655 Project: Apache Arrow Issue Type:

[jira] [Created] (ARROW-4654) [C++] Implicit Flight target dependencies cause compilation failure

2019-02-21 Thread Francois Saint-Jacques (JIRA)
Francois Saint-Jacques created ARROW-4654: - Summary: [C++] Implicit Flight target dependencies cause compilation failure Key: ARROW-4654 URL: https://issues.apache.org/jira/browse/ARROW-4654

[jira] [Created] (ARROW-4653) [C++] decimal multiply broken when both args are negative

2019-02-21 Thread Pindikura Ravindra (JIRA)
Pindikura Ravindra created ARROW-4653: - Summary: [C++] decimal multiply broken when both args are negative Key: ARROW-4653 URL: https://issues.apache.org/jira/browse/ARROW-4653 Project: Apache

Re: Flight / gRPC scalability issue

2019-02-21 Thread Antoine Pitrou
We're talking about the BCC tools, which are not based on perf: https://github.com/iovisor/bcc/ Apparently, using Linux perf for the same purpose is some kind of hassle (you need to write perf scripts?). Regards Antoine. Le 21/02/2019 à 18:40, Francois Saint-Jacques a écrit : > You can

Re: Flight / gRPC scalability issue

2019-02-21 Thread Francois Saint-Jacques
You can compile with dwarf (-g/-ggdb) and use `--call-graph=dwarf` to perf, it'll help the unwinding. Sometimes it's better than the stack pointer method since it keep track of inlined functions. On Thu, Feb 21, 2019 at 12:39 PM Antoine Pitrou wrote: > > Ah, thanks. I'm trying it now. The

Re: Flight / gRPC scalability issue

2019-02-21 Thread Antoine Pitrou
Ah, thanks. I'm trying it now. The problem is that it doesn't record userspace stack traces properly (it probably needs all dependencies to be recompiled with -fno-omit-frame-pointer :-/). So while I know that a lot of time is spent waiting for futextes, I don't know if that is for a

[jira] [Created] (ARROW-4652) [JS] RecordBatchReader throughNode should respect autoDestroy

2019-02-21 Thread Paul Taylor (JIRA)
Paul Taylor created ARROW-4652: -- Summary: [JS] RecordBatchReader throughNode should respect autoDestroy Key: ARROW-4652 URL: https://issues.apache.org/jira/browse/ARROW-4652 Project: Apache Arrow

Re: Flight / gRPC scalability issue

2019-02-21 Thread Hatem Helal
I was thinking of this variant: http://www.brendangregg.com/FlameGraphs/offcpuflamegraphs.html but I must admit that I haven't tried that technique myself. On 2/21/19, 4:41 PM, "Antoine Pitrou" wrote: I don't think that's the answer here. The question is not how to

Re: Flight / gRPC scalability issue

2019-02-21 Thread Antoine Pitrou
I don't think that's the answer here. The question is not how to /visualize/ where time is spent waiting, but how to /measure/ it. Normal profiling only tells you where CPU time is spent, not what the process is idly waiting for. Regards Antoine. On Thu, 21 Feb 2019 16:29:15 + Hatem

Re: Flight / gRPC scalability issue

2019-02-21 Thread Hatem Helal
I like flamegraphs for investigating this sort of problem: https://github.com/brendangregg/FlameGraph There are likely many other techniques for inspecting where time is being spent but that can at least help narrow down the search space. On 2/21/19, 4:03 PM, "Francois Saint-Jacques" wrote:

Re: Flight / gRPC scalability issue

2019-02-21 Thread Hatem Helal
I like flamegraphs for investigating this sort of problem: https://github.com/brendangregg/FlameGraph There are likely many other techniques for inspecting where time is being spent but that can at least help narrow down the search space. On 2/21/19, 4:29 PM, "Wes McKinney" wrote: Hi

Re: Flight / gRPC scalability issue

2019-02-21 Thread Wes McKinney
Hi Francois, It *should* work out of the box. I spent some time to make sure it does. Can you open a JIRA? I recommend using the grpc-cpp conda-forge package. Wes On Thu, Feb 21, 2019, 11:03 AM Francois Saint-Jacques < fsaintjacq...@gmail.com> wrote: > Can you remind us what's the easiest way

Re: Flight / gRPC scalability issue

2019-02-21 Thread Antoine Pitrou
On Thu, 21 Feb 2019 11:02:58 -0500 Francois Saint-Jacques wrote: > Can you remind us what's the easiest way to get flight working with grpc? > clone + make install doesn't really work out of the box. You can install the "grpc-cpp" package from conda-forge. Our CMake configuration should pick it

[jira] [Created] (ARROW-4651) [Format] Flight Location should be more flexible than a (host, port) pair

2019-02-21 Thread Antoine Pitrou (JIRA)
Antoine Pitrou created ARROW-4651: - Summary: [Format] Flight Location should be more flexible than a (host, port) pair Key: ARROW-4651 URL: https://issues.apache.org/jira/browse/ARROW-4651 Project:

Re: Flight / gRPC scalability issue

2019-02-21 Thread Francois Saint-Jacques
Can you remind us what's the easiest way to get flight working with grpc? clone + make install doesn't really work out of the box. François On Thu, Feb 21, 2019 at 10:41 AM Antoine Pitrou wrote: > > Hello, > > I've been trying to saturate several CPU cores using our Flight > benchmark (which

[jira] [Created] (ARROW-4650) The patch for PARQUET-1508 leads to infinite loop and infinite memory allocation when reading very sparse ByteArray columns

2019-02-21 Thread Valery Meleshkin (JIRA)
Valery Meleshkin created ARROW-4650: --- Summary: The patch for PARQUET-1508 leads to infinite loop and infinite memory allocation when reading very sparse ByteArray columns Key: ARROW-4650 URL:

Flight / gRPC scalability issue

2019-02-21 Thread Antoine Pitrou
Hello, I've been trying to saturate several CPU cores using our Flight benchmark (which spawns a server process and attempts to communicate with it using multiple clients), but haven't managed to. The typical command-line I'm executing is the following: $ time taskset -c 1,3,5,7

[jira] [Created] (ARROW-4649) [C++/CI/R] Add nightly job that builds `brew install apache-arrow --HEAD`

2019-02-21 Thread Uwe L. Korn (JIRA)
Uwe L. Korn created ARROW-4649: -- Summary: [C++/CI/R] Add nightly job that builds `brew install apache-arrow --HEAD` Key: ARROW-4649 URL: https://issues.apache.org/jira/browse/ARROW-4649 Project: Apache

[jira] [Created] (ARROW-4648) [C++/Question] Naming/organizational inconsistencies in cpp codebase

2019-02-21 Thread Krisztian Szucs (JIRA)
Krisztian Szucs created ARROW-4648: -- Summary: [C++/Question] Naming/organizational inconsistencies in cpp codebase Key: ARROW-4648 URL: https://issues.apache.org/jira/browse/ARROW-4648 Project:

[jira] [Created] (ARROW-4647) [Packaging] dev/release/00-prepare.sh fails for minor version changes

2019-02-21 Thread Uwe L. Korn (JIRA)
Uwe L. Korn created ARROW-4647: -- Summary: [Packaging] dev/release/00-prepare.sh fails for minor version changes Key: ARROW-4647 URL: https://issues.apache.org/jira/browse/ARROW-4647 Project: Apache

[jira] [Created] (ARROW-4645) [C++/Packaging] Ship Gandiva with OSX and Windows wheels

2019-02-21 Thread Krisztian Szucs (JIRA)
Krisztian Szucs created ARROW-4645: -- Summary: [C++/Packaging] Ship Gandiva with OSX and Windows wheels Key: ARROW-4645 URL: https://issues.apache.org/jira/browse/ARROW-4645 Project: Apache Arrow

[jira] [Created] (ARROW-4644) [C++/Docker] Build Gandiva in the docker containers

2019-02-21 Thread Krisztian Szucs (JIRA)
Krisztian Szucs created ARROW-4644: -- Summary: [C++/Docker] Build Gandiva in the docker containers Key: ARROW-4644 URL: https://issues.apache.org/jira/browse/ARROW-4644 Project: Apache Arrow