[Discuss] Benchmarking infrastructure

Back to the benchmarking per commit. So currently I have fired a community TeamCity Edition here http://arrow-publi-1wwtu5dnaytn9-2060566241.us-east-1.elb.amazonaws.com and dedicated pool of two Skylake bare metal machines (Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz) This can go to up to 4 if need

RE: FPGA support for Apache Arrow

Hi Chris, Do you have plans to contribute the infrastructure part back to the community so the others can build hybrid pipelines? -Original Message- From: Wes McKinney [mailto:wesmck...@gmail.com] Sent: Thursday, March 28, 2019 10:51 AM To: dev@arrow.apache.org Cc: ch...@inaccel.com Sub

Re: [Discuss][Format] Arrow Flight URI scheme proposal

hi David, This seems like a reasonable evolution from where we are now. I will defer to others to comment on the low-level details This is sort of scope and kind of a can of worms, but one area where we should invest some thought is alternative FlightData transports, while allowing the "command l

[jira] [Created] (ARROW-5067) Adding support for versioned documentation

Helen Ngo created ARROW-5067: Summary: Adding support for versioned documentation Key: ARROW-5067 URL: https://issues.apache.org/jira/browse/ARROW-5067 Project: Apache Arrow Issue Type: Improveme

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

That's good to know. In the case of the Ubuntu 18.10 test run I did not supply a value for ARROW_CUDA and I believe it was automatically detected On Thu, Mar 28, 2019 at 9:38 PM Kouhei Sutou wrote: > > Hi Wes, > > Thanks for verifying the RC. > We need to use "ON" for ARROW_CUDA instead of "1". B

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

Hi Wes, Thanks for verifying the RC. We need to use "ON" for ARROW_CUDA instead of "1". Because we compare ARROW_CUDA with "ON" in the verify script: https://github.com/apache/arrow/blob/master/dev/release/verify-release-candidate.sh#L243-L245 https://github.com/apache/arrow/blob/master/dev/relea

Re: [R] Improving documentation and transparency for Arrow build and packaging work for R

thanks Javier, I just gave you edit permissions on the wiki On Mon, Mar 25, 2019 at 4:55 PM Javier Luraschi wrote: > > I signed up as "Javier Luraschi" with this email, if you could please > give me access that would be great. Thanks! > > I'm assuming the CRAN documentation would go under: > http

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

Oh, I also checked that the Python 3.7 wheel can be installed in a fresh environment with pip and the imports work On Thu, Mar 28, 2019 at 8:29 PM Wes McKinney wrote: > > +1 (binding) > > * Verified source on Ubuntu 14.04 with > > ARROW_CUDA=0 ARROW_BOOST_VENDORED=1 > dev/release/verify-release-c

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

+1 (binding) * Verified source on Ubuntu 14.04 with ARROW_CUDA=0 ARROW_BOOST_VENDORED=1 dev/release/verify-release-candidate.sh source 0.13.0 4 I ran into ARROW-5059 but otherwise all the tests passed * Verified source on Ubuntu 18.10 with ARROW_CUDA=1 and system boost * Ran verification script

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

Hi, This was a GLib bug: https://github.com/apache/arrow/pull/4071 But we can mark this as a known bug because this is not a critical bug. Thanks, -- kou In <9e4b9e1d-31a2-6895-be3b-762aeaf60...@python.org> "Re: [VOTE] Release Apache Arrow 0.13.0 - RC4" on Thu, 28 Mar 2019 11:42:39 +0100,

[jira] [Created] (ARROW-5066) [Integration] Add flags to enable/disable implementations in integration/integration_test.py

Wes McKinney created ARROW-5066: --- Summary: [Integration] Add flags to enable/disable implementations in integration/integration_test.py Key: ARROW-5066 URL: https://issues.apache.org/jira/browse/ARROW-5066

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

+1 (binding) Both the source and binary verification passed on OSX. A minor plasma issue occurred in python and ruby tests: https://issues.apache.org/jira/browse/ARROW-5060 Installed wheels on OSX-py36 and debian-py37, both are working. If anyone has a windows box, please install the windows whee

[jira] [Created] (ARROW-5065) [Rust] cast kernel does not support casting from Int64

Andy Grove created ARROW-5065: - Summary: [Rust] cast kernel does not support casting from Int64 Key: ARROW-5065 URL: https://issues.apache.org/jira/browse/ARROW-5065 Project: Apache Arrow Issue T

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

Thanks Kouhei, I guess it ran with CUDA then and worked, great! Last time, something must have been broken in my CUDA install, so I had to disable to run the tests. Thanks for the update. On Thu, Mar 28, 2019 at 2:12 PM Kouhei Sutou wrote: > Hi Bryan, > > Thanks for verifying this RC. > > We've

[jira] [Created] (ARROW-5064) [Release] Pass PKG_CONFIG_PATH to glib in the verification script

Krisztian Szucs created ARROW-5064: -- Summary: [Release] Pass PKG_CONFIG_PATH to glib in the verification script Key: ARROW-5064 URL: https://issues.apache.org/jira/browse/ARROW-5064 Project: Apache A

[jira] [Created] (ARROW-5063) [Java] FlightClient should not create a child allocator

Bryan Cutler created ARROW-5063: --- Summary: [Java] FlightClient should not create a child allocator Key: ARROW-5063 URL: https://issues.apache.org/jira/browse/ARROW-5063 Project: Apache Arrow Is

[jira] [Created] (ARROW-5062) Shade Java Guava dependency for Flight

Bryan Cutler created ARROW-5062: --- Summary: Shade Java Guava dependency for Flight Key: ARROW-5062 URL: https://issues.apache.org/jira/browse/ARROW-5062 Project: Apache Arrow Issue Type: Improve

[jira] [Created] (ARROW-5061) [Packaging] Improve 03-binary performance

Kouhei Sutou created ARROW-5061: --- Summary: [Packaging] Improve 03-binary performance Key: ARROW-5061 URL: https://issues.apache.org/jira/browse/ARROW-5061 Project: Apache Arrow Issue Type: Impr

[jira] [Created] (ARROW-5060) [Plasma][Release] Socket pathname is too long

Krisztian Szucs created ARROW-5060: -- Summary: [Plasma][Release] Socket pathname is too long Key: ARROW-5060 URL: https://issues.apache.org/jira/browse/ARROW-5060 Project: Apache Arrow Issue

[jira] [Created] (ARROW-5059) [C++][Gandiva] cbrt_* floating point tests can fail due to exact comparisons

Wes McKinney created ARROW-5059: --- Summary: [C++][Gandiva] cbrt_* floating point tests can fail due to exact comparisons Key: ARROW-5059 URL: https://issues.apache.org/jira/browse/ARROW-5059 Project: Apa

[jira] [Created] (ARROW-5058) [Release] 02-source.sh generates e-mail template with wrong links

Kouhei Sutou created ARROW-5058: --- Summary: [Release] 02-source.sh generates e-mail template with wrong links Key: ARROW-5058 URL: https://issues.apache.org/jira/browse/ARROW-5058 Project: Apache Arrow

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

Hi Bryan, Thanks for verifying this RC. We've changed ARROW_HAVE_CUDA=NO to ARROW_CUDA=OFF: https://issues.apache.org/jira/browse/ARROW-4696 I've updated the "How to Verify Release Candidates" page: https://cwiki.apache.org/confluence/display/ARROW/How+to+Verify+Release+Candidates Thanks,

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

Hi Andy, Did you use the latest verify-release-candidate.sh? I've fixed it: https://issues.apache.org/jira/browse/ARROW-5048 Anyway, using ARROW_TEST_DATA is better. Thanks, -- kou In "Re: [VOTE] Release Apache Arrow 0.13.0 - RC4" on Thu, 28 Mar 2019 09:50:11 -0600, Andy Grove wrote: >

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

+1 (non-binding) Ran: ARROW_HAVE_CUDA=NO dev/release/verify-release-candidate.sh source 0.13.0 4 With tests: : ${TEST_DEFAULT:=1} : ${TEST_JAVA:=${TEST_DEFAULT}} : ${TEST_CPP:=${TEST_DEFAULT}} : ${TEST_GLIB:=0} : ${TEST_RUBY:=0} : ${TEST_PYTHON:=${TEST_DEFAULT}} : ${TEST_JS:=0} :

Re: Java allocate buffer code

Created jira for it https://issues.apache.org/jira/browse/ARROW-5057 Thanks. Hitesh. On Thursday, March 28, 2019, 11:25:58 AM PDT, Siddharth Teotia wrote: Hitesh, I suggest you file a JIRA for the potential issue you are seeing and if possible raise a PR with a test case that you think i

[jira] [Created] (ARROW-5057) Java: allocate new buffer code doesn't release extra allocated buffer properly

Hitesh Khamesra created ARROW-5057: -- Summary: Java: allocate new buffer code doesn't release extra allocated buffer properly Key: ARROW-5057 URL: https://issues.apache.org/jira/browse/ARROW-5057 Pro

Re: Java allocate buffer code

Hitesh, I suggest you file a JIRA for the potential issue you are seeing and if possible raise a PR with a test case that you think is broken with current code. Happy to discuss on Jira or PR. Thanks, Siddharth On Thu, Mar 28, 2019 at 11:20 AM Hitesh wrote: > Hi Siddharth: > > Here, I see a pr

Re: Java allocate buffer code

Hi Siddharth: Here, I see a problem in line#162, where its taking "bufferSize" to find the extra allocated bytes. It should be "valueCount*typeWidth + valueCount/8". Here is an example for that. Let's take 1000 ints. Then, valueCount = 1000 ints typWidth = 4 bytes validitiyBufferSize = 125 bytes

Re: Java allocate buffer code

Hitesh, Yes, if you see in the code, the sliced buffers have their reference counts bumped up before the compound buffer is released. Bumping up the reference counts of child/sliced buffers allows us to release the compound buffer safely. Does that make sense? Thanks, Siddharth On Wed, Mar 27, 2

[jira] [Created] (ARROW-5056) [Packaging] Adjust conda recipes to use ORC conda-forge package on unix systems

Krisztian Szucs created ARROW-5056: -- Summary: [Packaging] Adjust conda recipes to use ORC conda-forge package on unix systems Key: ARROW-5056 URL: https://issues.apache.org/jira/browse/ARROW-5056 Pr

[jira] [Created] (ARROW-5055) [Ruby][Msys2] libparquet needs to be installed in Msys2 for ruby

Dominic Sisneros created ARROW-5055: --- Summary: [Ruby][Msys2] libparquet needs to be installed in Msys2 for ruby Key: ARROW-5055 URL: https://issues.apache.org/jira/browse/ARROW-5055 Project: Apache

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

+1 (non-binding) Builds and passes tests on Windows 10 Ben Kietzman On Thu, Mar 28, 2019 at 11:50 AM Andy Grove wrote: > > Some of the Rust tests fail unless I manually copy some arrow test files > into the directory, then they pass. I filed > https://issues.apache.org/jira/browse/ARROW-5053 to

Re: FPGA support for Apache Arrow

hi Chris -- in addition to commenting on ARROW-2447, do you want to submit a PR to add yourself to Powered By? https://github.com/apache/arrow/blob/master/site/powered_by.md best Wes On Thu, Mar 28, 2019 at 6:01 AM Antoine Pitrou wrote: > > > Hello Chris, > > Le 28/03/2019 à 11:56, ch...@inacce

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

Some of the Rust tests fail unless I manually copy some arrow test files into the directory, then they pass. I filed https://issues.apache.org/jira/browse/ARROW-5053 to resolve this. However, I don't think this is significant enough to block the release. I will aim to resolve this today though, jus

[jira] [Created] (ARROW-5053) [Rust] [DataFusion] Use env var for location of arrow test data

Andy Grove created ARROW-5053: - Summary: [Rust] [DataFusion] Use env var for location of arrow test data Key: ARROW-5053 URL: https://issues.apache.org/jira/browse/ARROW-5053 Project: Apache Arrow

[jira] [Created] (ARROW-5054) [C++][Release] Test Flight in verify-release-candidate.sh

Wes McKinney created ARROW-5054: --- Summary: [C++][Release] Test Flight in verify-release-candidate.sh Key: ARROW-5054 URL: https://issues.apache.org/jira/browse/ARROW-5054 Project: Apache Arrow

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

+1 (non-binding) No issues with the latest release-verify script on Ubuntu18.04. François On Thu, Mar 28, 2019 at 6:00 AM Kouhei Sutou wrote: > Hi, > > I would like to propose the following release candidate (RC4) of Apache > Arrow version 0.13.0. This is a release consiting of 596 > resolved

[jira] [Created] (ARROW-5052) [C++] Add an incomplete dictionary type

Antoine Pitrou created ARROW-5052: - Summary: [C++] Add an incomplete dictionary type Key: ARROW-5052 URL: https://issues.apache.org/jira/browse/ARROW-5052 Project: Apache Arrow Issue Type: Im

Re: FPGA support for Apache Arrow

Hello Chris, Le 28/03/2019 à 11:56, ch...@inaccel.com a écrit : > > I would like to let you know that we have released our new version > of our framework that > allows FPGAs to talk to any other framework under Arrow. > > That means that any

FPGA support for Apache Arrow

Hi, I would like to let you know that we have released our new version of our framework that allows FPGAs to talk to any other framework under Arrow. That means that any Arrow-based framework can utilize now FPGAs in order to speedup their

Re: [VOTE] Release Apache Arrow 0.13.0 - RC4

I'm getting one Gandiva test failure in Ruby (see https://issues.apache.org/jira/browse/ARROW-5051). Otherwise +1 from me. C++, Python, Java ran fine on Ubuntu 18.04. Regards Antoine. Le 28/03/2019 à 11:00, Kouhei Sutou a écrit : > Hi, > > I would like to propose the following release cand

[jira] [Created] (ARROW-5051) [Gandiva] [Ruby] Test failure in release verification script

Antoine Pitrou created ARROW-5051: - Summary: [Gandiva] [Ruby] Test failure in release verification script Key: ARROW-5051 URL: https://issues.apache.org/jira/browse/ARROW-5051 Project: Apache Arrow

[VOTE] Release Apache Arrow 0.13.0 - RC4

Hi, I would like to propose the following release candidate (RC4) of Apache Arrow version 0.13.0. This is a release consiting of 596 resolved JIRA issues[1]. This release candidate is based on commit: dfb9e7af3cd92722893a3819b6676dfdef08f896 [2] The source release rc4 is hosted at [3]. The binar

Re: [VOTE] Release Apache Arrow 0.13.0 - RC3

I've uploaded source archive and built binaries but I reached Bintray API limit... I have to wait one day for uploading binaries. Thanks, -- kou In <20190328.111049.2112778015738050375@clear-code.com> "Re: [VOTE] Release Apache Arrow 0.13.0 - RC3" on Thu, 28 Mar 2019 11:10:49 +0900 (JST),