[jira] [Created] (ARROW-5974) read_csv returns truncated read for some valid gzip files

2019-07-17 Thread Jordan Samuels (JIRA)
Jordan Samuels created ARROW-5974: - Summary: read_csv returns truncated read for some valid gzip files Key: ARROW-5974 URL: https://issues.apache.org/jira/browse/ARROW-5974 Project: Apache Arrow

[jira] [Created] (ARROW-5973) [Java] Variable width vectors' get methods should return return null when the underlying data is null

2019-07-17 Thread Liya Fan (JIRA)
Liya Fan created ARROW-5973: --- Summary: [Java] Variable width vectors' get methods should return return null when the underlying data is null Key: ARROW-5973 URL: https://issues.apache.org/jira/browse/ARROW-5973

Re: [VOTE] Release Apache Arrow 0.14.1 - RC0

2019-07-17 Thread Wes McKinney
+1 (binding) I also ran the source and binary verification on Ubuntu 18.04 -- I was able to run all the tests in the source verification and all looks good. If it weren't a patch release I'd also run Windows, but since we have Python packages for Windows that gives me confidence =) Note that the

Re: [VOTE] Release Apache Arrow 0.14.1 - RC0

2019-07-17 Thread Wes McKinney
For the record, if Google had taken it upon themselves to work with the PyPA / the Python community generally to fix the wheel standard to suit TensorFlow instead of flagrantly violating it, it would likely also have addressed the core problems that are causing me to give up on wheel development.

Re: [VOTE] Release Apache Arrow 0.14.1 - RC0

2019-07-17 Thread Wes McKinney
Krisz is the RM so he can decide but: We've taken the position that we won't be inconvenienced by issues stemming from TensorFlow's non-compliant wheels. Producing a new release candidate is several hours of work, at least, so if it were me this alone would not be reason enough to cancel an RC.

Re: Re: Sharing Java Arrow Buffer with C++ in same process

2019-07-17 Thread Ravindra Pindikura
*gandiva* does the sharing in one direction i.e buffers allocated in java, and passed along to c++ in jni calls. The java side code of extracting of the address and passing it down is here

Re: [VOTE] Release Apache Arrow 0.14.1 - RC0

2019-07-17 Thread Zhuo Peng
Hi Krisztián, Sorry if it's too late, but is it possible to also include https://github.com/apache/arrow/pull/4883 in the release? This would help resolve https://github.com/apache/arrow/issues/4472 . Thanks, Zhuo On Wed, Jul 17, 2019 at 3:00 AM Antoine Pitrou wrote: > > +1 (binding). > >

Re: [DISCUSS][C++][Proposal] Threading engine for Arrow

2019-07-17 Thread Wes McKinney
I've been looking at little bit at this in the context of Parquet files One of the read hot paths in cpp/src/parquet is the function that reads and decompresses data pages from the stream: (SerializedPageReader::NextPage)

Re: Caution about CI builds on personal forks

2019-07-17 Thread Wes McKinney
Please keep in mind that any code transferred in from a third party repository will probably need to go through the IP clearance process. So if you do intend to move code, let's try to make it a one-time thing so we can avoid this happening again in the future. On Wed, Jul 17, 2019 at 12:45 PM

Re: Caution about CI builds on personal forks

2019-07-17 Thread Antoine Pitrou
On Wed, 17 Jul 2019 19:40:01 +0200 Krisztián Szűcs wrote: > > > > Hi Krisz -- we can't have primary CI configurations in a third party > > repository. > > > I have a solution in mind, most of the arrow related parts are clearly > separated, just need to organize to codebase accordingly. >

Re: Caution about CI builds on personal forks

2019-07-17 Thread Krisztián Szűcs
> > Hi Krisz -- we can't have primary CI configurations in a third party > repository. > I have a solution in mind, most of the arrow related parts are clearly separated, just need to organize to codebase accordingly. (there are a couple of shared services like the github hook handling which

Re: Caution about CI builds on personal forks

2019-07-17 Thread Wes McKinney
Hi Krisz -- we can't have primary CI configurations in a third party repository. We need to figure out how to make 100% of builds runnable from the apache/arrow repository. On Wed, Jul 17, 2019, 12:29 PM Krisztián Szűcs wrote: > Besides the builders listed at

Re: Caution about CI builds on personal forks

2019-07-17 Thread Krisztián Szűcs
Besides the builders listed at https://ci.ursalabs.org/#/builders I've also migrated JS and R is in the queue https://github.com/ursa-labs/ursabot/pull/135 On Wed, Jul 17, 2019 at 6:23 PM Wes McKinney wrote: > Indeed, Docker runs on macOS and Windows > > In our Travis CI build matrix we have >

Re: Caution about CI builds on personal forks

2019-07-17 Thread Krisztián Szűcs
On Wed, Jul 17, 2019 at 5:55 PM Wes McKinney wrote: > Presumably a migration away from Travis also means that we have to > develop tools to allow contributors to test their patches outside of > the GitHub pull request. If something is Docker-based, then it can be > run locally, of course. >

Re: Caution about CI builds on personal forks

2019-07-17 Thread Krisztián Szűcs
Docker on Mac is able to run other archs as well, like aarch64. On Wed, Jul 17, 2019 at 6:16 PM Uwe L. Korn wrote: > Docker works well for all people on all OSes. > > Interesting will be Windows, OSX or aarch64 builds which require a special > system. > > Uwe > > On Wed, Jul 17, 2019, at 6:11

[jira] [Created] (ARROW-5972) [Rust] Installing cargo-tarpaulin and generating coverage report takes over 20 minutes

2019-07-17 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-5972: --- Summary: [Rust] Installing cargo-tarpaulin and generating coverage report takes over 20 minutes Key: ARROW-5972 URL: https://issues.apache.org/jira/browse/ARROW-5972

Re: Caution about CI builds on personal forks

2019-07-17 Thread Wes McKinney
Indeed, Docker runs on macOS and Windows In our Travis CI build matrix we have * 11 Linux builds * 3 macOS builds If we migrated just the Linux builds it might make the Travis wait times more acceptable in the short term On Wed, Jul 17, 2019 at 11:16 AM Uwe L. Korn wrote: > > Docker works

Re: Caution about CI builds on personal forks

2019-07-17 Thread Uwe L. Korn
Docker works well for all people on all OSes. Interesting will be Windows, OSX or aarch64 builds which require a special system. Uwe On Wed, Jul 17, 2019, at 6:11 PM, Antoine Pitrou wrote: > > I'm not sure how Docker will work for people not on Linux though? > (and/or for macOS builds) > >

Re: Caution about CI builds on personal forks

2019-07-17 Thread Antoine Pitrou
I'm not sure how Docker will work for people not on Linux though? (and/or for macOS builds) Regards Antoine. On Wed, 17 Jul 2019 10:54:13 -0500 Wes McKinney wrote: > Presumably a migration away from Travis also means that we have to > develop tools to allow contributors to test their

Aw: Re: Sharing Java Arrow Buffer with C++ in same process

2019-07-17 Thread hans-joachim . bothe
Thanks to Uwe! The Python step from C++ would have been next. You hit it! Perfect shortcut. Jacques, i am new here but i am gladly willing to help and learn. if you could give me a hint for an outline and content of the blog post and where/how to file it. Will read #3191 next. greetings &

Re: Caution about CI builds on personal forks

2019-07-17 Thread Wes McKinney
Presumably a migration away from Travis also means that we have to develop tools to allow contributors to test their patches outside of the GitHub pull request. If something is Docker-based, then it can be run locally, of course. We definitely can't persist under the current circumstances where

Re: Caution about CI builds on personal forks

2019-07-17 Thread Neal Richardson
Won't moving CI away from Travis to our own infrastructure mean that we won't get any CI on our personal forks? On Wed, Jul 17, 2019 at 8:23 AM Wes McKinney wrote: > On Wed, Jul 17, 2019 at 10:22 AM Wes McKinney wrote: > > > > hi folks -- I noticed this last night on > >

[jira] [Created] (ARROW-5971) [Website] Blog post introducing Arrow Flight

2019-07-17 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-5971: --- Summary: [Website] Blog post introducing Arrow Flight Key: ARROW-5971 URL: https://issues.apache.org/jira/browse/ARROW-5971 Project: Apache Arrow Issue Type:

Re: Sharing Java Arrow Buffer with C++ in same process

2019-07-17 Thread Wes McKinney
I agree that a blog post would be helpful (of course, blogs are a lot of work to write!) -- it would be good to let people know that the machinery is available to do this kind of bidirectional zero-copy passing. I was glad to see ARROW-3191 and I would like to see the Java folks "tooting their own

Re: Caution about CI builds on personal forks

2019-07-17 Thread Wes McKinney
On Wed, Jul 17, 2019 at 10:22 AM Wes McKinney wrote: > > hi folks -- I noticed this last night on > https://github.com/apache/arrow/pull/4841 and it surprised me. Others > may not be aware. > > We have been using builds on Appveyor and Travis CI to decide whether > to merge PRs. The trouble is

Caution about CI builds on personal forks

2019-07-17 Thread Wes McKinney
hi folks -- I noticed this last night on https://github.com/apache/arrow/pull/4841 and it surprised me. Others may not be aware. We have been using builds on Appveyor and Travis CI to decide whether to merge PRs. The trouble is these builds are not equivalent to the builds that Travis runs inside

Re: Sharing Java Arrow Buffer with C++ in same process

2019-07-17 Thread Jacques Nadeau
Java > C++ has been available for a long time, I believe. With ARROW-3191, we now have a good story as well for C++ > Java. Reference management is the key thing you have to think about. Now that we have the ReferenceManager interface in Java, you can decide how this works between the two layers.

Re: Sharing Java Arrow Buffer with C++ in same process

2019-07-17 Thread Uwe L. Korn
Hello Hans, we sadly have no code for the C++<->Java interaction but a good example is the Python<->Java interaction code in https://github.com/apache/arrow/blob/master/python/pyarrow/jvm.py . This call Java from Python using the jpype1 module and then uses the memory pointers in the Java

Sharing Java Arrow Buffer with C++ in same process

2019-07-17 Thread hans-joachim . bothe
In my application I need to share Arrow buffers allocated in Java with C++ in the same process. Is there already some code in Arrow to pass the native address from Java to C++ or do I have to do my own JNI call? I do not want to go via the Plasma sockets and did not find any hint in docs and

[jira] [Created] (ARROW-5970) [Java] Provide pointer to Arrow buffer

2019-07-17 Thread Liya Fan (JIRA)
Liya Fan created ARROW-5970: --- Summary: [Java] Provide pointer to Arrow buffer Key: ARROW-5970 URL: https://issues.apache.org/jira/browse/ARROW-5970 Project: Apache Arrow Issue Type: New Feature

[jira] [Created] (ARROW-5969) [CI] [R] Lint failures

2019-07-17 Thread Antoine Pitrou (JIRA)
Antoine Pitrou created ARROW-5969: - Summary: [CI] [R] Lint failures Key: ARROW-5969 URL: https://issues.apache.org/jira/browse/ARROW-5969 Project: Apache Arrow Issue Type: Bug

[jira] [Created] (ARROW-5968) [Java] Remove duplicate Preconditions check in JDBC adapter

2019-07-17 Thread Ji Liu (JIRA)
Ji Liu created ARROW-5968: - Summary: [Java] Remove duplicate Preconditions check in JDBC adapter Key: ARROW-5968 URL: https://issues.apache.org/jira/browse/ARROW-5968 Project: Apache Arrow Issue

[jira] [Created] (ARROW-5967) [Java] DateUtility#timeZoneList is not correct

2019-07-17 Thread Ji Liu (JIRA)
Ji Liu created ARROW-5967: - Summary: [Java] DateUtility#timeZoneList is not correct Key: ARROW-5967 URL: https://issues.apache.org/jira/browse/ARROW-5967 Project: Apache Arrow Issue Type:

Re: [VOTE] Release Apache Arrow 0.14.1 - RC0

2019-07-17 Thread Antoine Pitrou
+1 (binding). Tested on Ubuntu 18.04.2 (x86-64) with CUDA enabled: - binaries verification worked fine - source verification worked until the npm step, which failed (I don't have npm installed) Regards Antoine. Le 17/07/2019 à 04:54, Krisztián Szűcs a écrit : > Hi, > > I would like to

[jira] [Created] (ARROW-5965) Regression: segfault when reading hive table with v0.14

2019-07-17 Thread H. Vetinari (JIRA)
H. Vetinari created ARROW-5965: -- Summary: Regression: segfault when reading hive table with v0.14 Key: ARROW-5965 URL: https://issues.apache.org/jira/browse/ARROW-5965 Project: Apache Arrow

[jira] [Created] (ARROW-5964) [C++][Gandiva] Cast double to decimal with rounding returns 0

2019-07-17 Thread Pindikura Ravindra (JIRA)
Pindikura Ravindra created ARROW-5964: - Summary: [C++][Gandiva] Cast double to decimal with rounding returns 0 Key: ARROW-5964 URL: https://issues.apache.org/jira/browse/ARROW-5964 Project: