[jira] [Created] (ARROW-4228) [GLib] Add garrow_list_data_type_get_field()

2019-01-09 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4228: --- Summary: [GLib] Add garrow_list_data_type_get_field() Key: ARROW-4228 URL: https://issues.apache.org/jira/browse/ARROW-4228 Project: Apache Arrow Issue Type:

[jira] [Created] (ARROW-4227) [GLib] Field in composite data type returns wrong data type

2019-01-09 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4227: --- Summary: [GLib] Field in composite data type returns wrong data type Key: ARROW-4227 URL: https://issues.apache.org/jira/browse/ARROW-4227 Project: Apache Arrow

[jira] [Created] (ARROW-4226) [C++] Add CSF sparse tensor support

2019-01-09 Thread Kenta Murata (JIRA)
Kenta Murata created ARROW-4226: --- Summary: [C++] Add CSF sparse tensor support Key: ARROW-4226 URL: https://issues.apache.org/jira/browse/ARROW-4226 Project: Apache Arrow Issue Type: New

[jira] [Created] (ARROW-4225) [C++] Add CSC sparse matrix support

2019-01-09 Thread Kenta Murata (JIRA)
Kenta Murata created ARROW-4225: --- Summary: [C++] Add CSC sparse matrix support Key: ARROW-4225 URL: https://issues.apache.org/jira/browse/ARROW-4225 Project: Apache Arrow Issue Type: New

[jira] [Created] (ARROW-4224) [Python] Support integration with pydata/sparse library

2019-01-09 Thread Kenta Murata (JIRA)
Kenta Murata created ARROW-4224: --- Summary: [Python] Support integration with pydata/sparse library Key: ARROW-4224 URL: https://issues.apache.org/jira/browse/ARROW-4224 Project: Apache Arrow

[jira] [Created] (ARROW-4223) [Python] Support scipy.sparse integration

2019-01-09 Thread Kenta Murata (JIRA)
Kenta Murata created ARROW-4223: --- Summary: [Python] Support scipy.sparse integration Key: ARROW-4223 URL: https://issues.apache.org/jira/browse/ARROW-4223 Project: Apache Arrow Issue Type:

[jira] [Created] (ARROW-4221) [Format] Add canonical flag in COO sparse index

2019-01-09 Thread Kenta Murata (JIRA)
Kenta Murata created ARROW-4221: --- Summary: [Format] Add canonical flag in COO sparse index Key: ARROW-4221 URL: https://issues.apache.org/jira/browse/ARROW-4221 Project: Apache Arrow Issue

[jira] [Created] (ARROW-4222) [C++] Support equality comparison between COO and CSR sparse tensors in SparseTensorEquals

2019-01-09 Thread Kenta Murata (JIRA)
Kenta Murata created ARROW-4222: --- Summary: [C++] Support equality comparison between COO and CSR sparse tensors in SparseTensorEquals Key: ARROW-4222 URL: https://issues.apache.org/jira/browse/ARROW-4222

Re: Arrow sync today 17:00 UTC / 12:00 US-Eastern

2019-01-09 Thread Wes McKinney
Notes from today's call - Wes McKinney (Ursa Labs) - 0.12 release - Gandiva on Windows - Ravindra (Dremio) - Bryan (IBM) - 0.12 release - Hatem Halel (Mathworks) - Release schedule - Kevin Gurney (Mathworks) - Rylan Dmello (Mathworks) - Neville Dipale (S Africa) - Shyam Singh (Dremio) -

[jira] [Created] (ARROW-4220) [Python] Add buffered input and output stream ASV benchmarks with simulated high latency IO

2019-01-09 Thread Wes McKinney (JIRA)
Wes McKinney created ARROW-4220: --- Summary: [Python] Add buffered input and output stream ASV benchmarks with simulated high latency IO Key: ARROW-4220 URL: https://issues.apache.org/jira/browse/ARROW-4220

[jira] [Created] (ARROW-4219) [Rust] [Parquet] Implement ArrowReader

2019-01-09 Thread Renjie Liu (JIRA)
Renjie Liu created ARROW-4219: - Summary: [Rust] [Parquet] Implement ArrowReader Key: ARROW-4219 URL: https://issues.apache.org/jira/browse/ARROW-4219 Project: Apache Arrow Issue Type: Sub-task

[jira] [Created] (ARROW-4218) [Rust][Parquet]Implement ColumnReader

2019-01-09 Thread Renjie Liu (JIRA)
Renjie Liu created ARROW-4218: - Summary: [Rust][Parquet]Implement ColumnReader Key: ARROW-4218 URL: https://issues.apache.org/jira/browse/ARROW-4218 Project: Apache Arrow Issue Type: Sub-task

Re: Timeline for Arrow 0.12.0 release

2019-01-09 Thread Wes McKinney
I'll connect with Krisztian tomorrow (Thursday) morning to make sure KEYS are in order and to give him the day to do a run through of the release process to make sure there are no problems. In the meantime I'll try to resolve as many of the remaining issues as possible, then move the leftovers to

[jira] [Created] (ARROW-4217) [Plasma] Remove custom object metadata

2019-01-09 Thread Philipp Moritz (JIRA)
Philipp Moritz created ARROW-4217: - Summary: [Plasma] Remove custom object metadata Key: ARROW-4217 URL: https://issues.apache.org/jira/browse/ARROW-4217 Project: Apache Arrow Issue Type:

[jira] [Created] (ARROW-4216) [Python] Add CUDA API docs

2019-01-09 Thread Antoine Pitrou (JIRA)
Antoine Pitrou created ARROW-4216: - Summary: [Python] Add CUDA API docs Key: ARROW-4216 URL: https://issues.apache.org/jira/browse/ARROW-4216 Project: Apache Arrow Issue Type: Improvement

[jira] [Created] (ARROW-4215) [GLib] Fix typos in documentation

2019-01-09 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4215: --- Summary: [GLib] Fix typos in documentation Key: ARROW-4215 URL: https://issues.apache.org/jira/browse/ARROW-4215 Project: Apache Arrow Issue Type: Bug

[jira] [Created] (ARROW-4214) [Ruby] Add support for building RecordBatch from raw Ruby objects

2019-01-09 Thread Kouhei Sutou (JIRA)
Kouhei Sutou created ARROW-4214: --- Summary: [Ruby] Add support for building RecordBatch from raw Ruby objects Key: ARROW-4214 URL: https://issues.apache.org/jira/browse/ARROW-4214 Project: Apache Arrow

Re: Datum API

2019-01-09 Thread Wes McKinney
Not a compelling reason -- it was perhaps a premature optimization to avoid boxing and unboxing costs as the kernel APIs are more likely to work with ArrayData consistently than the boxed types. The internal data representation of the default RecordBatch is unboxed ArrayData rather than boxed

[jira] [Created] (ARROW-4213) [Flight] C++ and Java implementations are incompatible

2019-01-09 Thread David Li (JIRA)
David Li created ARROW-4213: --- Summary: [Flight] C++ and Java implementations are incompatible Key: ARROW-4213 URL: https://issues.apache.org/jira/browse/ARROW-4213 Project: Apache Arrow Issue

Re: Compiling Arrow for RaspberryPi

2019-01-09 Thread Suvayu Ali
Hello Uwe, Wes, others On Wed, Jan 9, 2019 at 10:08 AM Uwe L. Korn wrote: > > for arrow-cpp it is definitely possible to cross-compile on the desktop as it > using standard CMake for the build. There are a lot of guides available for > doing cross compilation with CMake. This may work but I

Datum API

2019-01-09 Thread Francois Saint-Jacques
Is there a reason why Datum::ARRAY stores an ArrayData and not an Array? I'm aware there's the `make_array` method to obtain the equivalent, but was wondering if there was a deeper reason.

[jira] [Created] (ARROW-4212) [Python] [CUDA] Creating a CUDA buffer from Numba device array should be easier

2019-01-09 Thread Antoine Pitrou (JIRA)
Antoine Pitrou created ARROW-4212: - Summary: [Python] [CUDA] Creating a CUDA buffer from Numba device array should be easier Key: ARROW-4212 URL: https://issues.apache.org/jira/browse/ARROW-4212

Re: [DRAFT] Apache Arrow board report January 2019

2019-01-09 Thread Wes McKinney
Yes, definitely. On Wed, Jan 9, 2019 at 5:07 AM Antoine Pitrou wrote: > > > Should it mention the forthcoming 0.12.0 release? > > Regards > > Antoine. > > > Le 09/01/2019 à 06:54, Wes McKinney a écrit : > > ## Description: > > > > - Apache Arrow is a cross-language development platform for

Arrow sync today 17:00 UTC / 12:00 US-Eastern

2019-01-09 Thread Wes McKinney
The biweekly sync appoint crept up on me again today. We'll be convening briefly at https://meet.google.com/vtm-teks-phx All are welcome -- notes to be sent to the mailing list after

[jira] [Created] (ARROW-4211) [GLib] Add GArrowFixedSizeBinaryDataType

2019-01-09 Thread Yosuke Shiro (JIRA)
Yosuke Shiro created ARROW-4211: --- Summary: [GLib] Add GArrowFixedSizeBinaryDataType Key: ARROW-4211 URL: https://issues.apache.org/jira/browse/ARROW-4211 Project: Apache Arrow Issue Type:

Re: Compiling Arrow for RaspberryPi

2019-01-09 Thread Wes McKinney
We would definitely welcome PRs with documentation or advice for RPi users. I have compiled Arrow and Parquet on Raspberry Pi in the past but not very recently (the project has grown a lot in size since then). On Wed, Jan 9, 2019 at 10:08 AM Uwe L. Korn wrote: > > Hello Suvayu, > > for

Re: Compiling Arrow for RaspberryPi

2019-01-09 Thread Uwe L. Korn
Hello Suvayu, for arrow-cpp it is definitely possible to cross-compile on the desktop as it using standard CMake for the build. There are a lot of guides available for doing cross compilation with CMake. This may work but I would expect that in some places we're probably not passing all flags

Re: RecordBatchFile with no batches, Error: Pyarrow.lib.ArrowInvalid: File is smaller than indicated metadata size.

2019-01-09 Thread Uwe L. Korn
Hello Ryan, for CentOS and pip, I would recommend to use the docker scripts that we use to build the manylinux1 compatible wheels (the ones we also upload to PyPI): https://github.com/apache/arrow/tree/master/python/manylinux1 They will bootstrap an isolated environment in docker that is

[jira] [Created] (ARROW-4210) [Python] Mention boost-cpp directly in the conda meta.yaml for pyarrow

2019-01-09 Thread Uwe L. Korn (JIRA)
Uwe L. Korn created ARROW-4210: -- Summary: [Python] Mention boost-cpp directly in the conda meta.yaml for pyarrow Key: ARROW-4210 URL: https://issues.apache.org/jira/browse/ARROW-4210 Project: Apache

Re: [Rust] [DISCUSS] Donate DataFusion to Arrow project

2019-01-09 Thread Andy Grove
Wes, Thanks. This sounds great. Andy. On Tue, Jan 8, 2019 at 8:28 AM Wes McKinney wrote: > hi Andy -- I'm supportive of the code donation. I see building > in-memory, embeddable analytics and query processing as the natural > next stage of this project. As I have described on this mailing

Re: [Rust] [DISCUSS] Donate DataFusion to Arrow project

2019-01-09 Thread Andy Grove
Hi Neville, Thanks for the support. DataFrame and SQL are two different ways of building a logical query plan and it makes sense that they should both build the same type of plan without code duplication. It is also sometimes beneficial to mix and match DataFrame and SQL operations (as per

RE: RecordBatchFile with no batches, Error: Pyarrow.lib.ArrowInvalid: File is smaller than indicated metadata size.

2019-01-09 Thread White4, Ryan (STATCAN)
Thanks Wes, I confirmed this is fixed in master. In the future, I'll check against the master if we come across anything. We would be interested to use the nightly builds, for sure. We do not use conda as of now, so may be best to become more familiar with conda. I needed to get home to use my

Re: Timeline for Arrow 0.12.0 release

2019-01-09 Thread Antoine Pitrou
No preference from me. Regards Antoine. Le 09/01/2019 à 14:34, Krisztián Szűcs a écrit : > Should We aim for Friday (2019-01-11) or Thursday (2019-01-10)? > > On Wed, Jan 9, 2019 at 11:43 AM Antoine Pitrou wrote: > >> >> For the record, none of these issues seem release-critical to me,

Re: Timeline for Arrow 0.12.0 release

2019-01-09 Thread Krisztián Szűcs
Should We aim for Friday (2019-01-11) or Thursday (2019-01-10)? On Wed, Jan 9, 2019 at 11:43 AM Antoine Pitrou wrote: > > For the record, none of these issues seem release-critical to me, apart > from the RAT issue and perhaps the Windows git-describe issue. Though > perhaps the release

[jira] [Created] (ARROW-4209) [Gandiva] returning IR structs causes issues with windows

2019-01-09 Thread Pindikura Ravindra (JIRA)
Pindikura Ravindra created ARROW-4209: - Summary: [Gandiva] returning IR structs causes issues with windows Key: ARROW-4209 URL: https://issues.apache.org/jira/browse/ARROW-4209 Project: Apache

[jira] [Created] (ARROW-4208) [CI/Python] Have automatized tests for S3

2019-01-09 Thread Krisztian Szucs (JIRA)
Krisztian Szucs created ARROW-4208: -- Summary: [CI/Python] Have automatized tests for S3 Key: ARROW-4208 URL: https://issues.apache.org/jira/browse/ARROW-4208 Project: Apache Arrow Issue

Re: [DRAFT] Apache Arrow board report January 2019

2019-01-09 Thread Antoine Pitrou
Should it mention the forthcoming 0.12.0 release? Regards Antoine. Le 09/01/2019 à 06:54, Wes McKinney a écrit : > ## Description: > > - Apache Arrow is a cross-language development platform for in-memory data. It > specifies a standardized language-independent columnar memory format for

Re: Timeline for Arrow 0.12.0 release

2019-01-09 Thread Antoine Pitrou
For the record, none of these issues seem release-critical to me, apart from the RAT issue and perhaps the Windows git-describe issue. Though perhaps the release manager is best placed to evaluate them. Regards Antoine. Le 08/01/2019 à 23:07, Wes McKinney a écrit : > Well, a mid-week

[jira] [Created] (ARROW-4207) [Gandiva] [GLib] Add support for IfNode

2019-01-09 Thread Yosuke Shiro (JIRA)
Yosuke Shiro created ARROW-4207: --- Summary: [Gandiva] [GLib] Add support for IfNode Key: ARROW-4207 URL: https://issues.apache.org/jira/browse/ARROW-4207 Project: Apache Arrow Issue Type: