[jira] [Created] (ARROW-7845) [c++] reading list from parquet files

2020-02-12 Thread Mikhail Filimonov (Jira)
Mikhail Filimonov created ARROW-7845: Summary: [c++] reading list from parquet files Key: ARROW-7845 URL: https://issues.apache.org/jira/browse/ARROW-7845 Project: Apache Arrow Issue

Re: Arrow doesn't have a MapType

2020-02-12 Thread Shawn Yang
Thanks Wes. I was using 0.14 before. BTW, it seems the doc for data types didn't updated fully. I'll submit a PR for this. On Thu, Feb 13, 2020 at 12:28 AM Wes McKinney wrote: > It was added between 0.15.0 and 0.16.0. Any feedback from using it > would be welcome > > >

Re: [VOTE] Adopt Arrow in-process C Data Interface specification

2020-02-12 Thread Wes McKinney
+1 (binding) On Tue, Feb 11, 2020 at 4:29 PM Antoine Pitrou wrote: > > > Ah, you're right, it's PR 6040: > https://github.com/apache/arrow/pull/6040 > > Similarly, the C++ implementation is at PR 6026: > https://github.com/apache/arrow/pull/6026 > > Regards > > Antoine. > > > Le 11/02/2020 à

Re: PR Dashboard for Java?

2020-02-12 Thread Bryan Cutler
Works now, thanks! I added a page for Java open PRs https://cwiki.apache.org/confluence/display/ARROW/Java+Open+Patches On Tue, Feb 11, 2020 at 12:08 PM Wes McKinney wrote: > Weird. Try now > > On Tue, Feb 11, 2020 at 1:03 PM Bryan Cutler wrote: > > > > Wes, it doesn't seem to have worked.

[jira] [Created] (ARROW-7844) [R] Parquet list column test is flaky

2020-02-12 Thread Neal Richardson (Jira)
Neal Richardson created ARROW-7844: -- Summary: [R] Parquet list column test is flaky Key: ARROW-7844 URL: https://issues.apache.org/jira/browse/ARROW-7844 Project: Apache Arrow Issue Type:

Re: [ARROW-3329] Re: Decimal casting or scaling

2020-02-12 Thread Wes McKinney
On Wed, Feb 12, 2020 at 2:37 PM Jacek Pliszka wrote: > > Actually these options still make some sense - but not as much as before. > > The use case: unit conversion > > Data about prices exported from sql in Decimal(38,10) which uses 128 > bit but the numbers are actually prices which expressed

Re: [C++][Parquet] Is arrow::parquet::FileWriter::WriteColumnChunk intended to be public?

2020-02-12 Thread Wes McKinney
Having them be public was the intention, but it seems that column-wise writing is not yet fully baked. I think it would be OK to make these methods private until they can be appropriately tested On Sat, Feb 8, 2020 at 10:49 PM Micah Kornfield wrote: > > I'm asking because it doesn't seem to do

[jira] [Created] (ARROW-7843) [Ruby] MSYS2 packages needed for arrow-gandiva arrow-cuda

2020-02-12 Thread Dominic Sisneros (Jira)
Dominic Sisneros created ARROW-7843: --- Summary: [Ruby] MSYS2 packages needed for arrow-gandiva arrow-cuda Key: ARROW-7843 URL: https://issues.apache.org/jira/browse/ARROW-7843 Project: Apache Arrow

Re: [ARROW-3329] Re: Decimal casting or scaling

2020-02-12 Thread Jacek Pliszka
Actually these options still make some sense - but not as much as before. The use case: unit conversion Data about prices exported from sql in Decimal(38,10) which uses 128 bit but the numbers are actually prices which expressed in cents fit perfectly in uint32 Having scaling would reduce

[jira] [Created] (ARROW-7842) [Rust] [Parquet] Implement array reader for list type

2020-02-12 Thread Morgan Cassels (Jira)
Morgan Cassels created ARROW-7842: - Summary: [Rust] [Parquet] Implement array reader for list type Key: ARROW-7842 URL: https://issues.apache.org/jira/browse/ARROW-7842 Project: Apache Arrow

Re: [ARROW-3329] Re: Decimal casting or scaling

2020-02-12 Thread Jacek Pliszka
OK, then what I proposed does not make sense and I can just copy the solution you pointed out. Thank you, Jacek śr., 12 lut 2020 o 19:27 Wes McKinney napisał(a): > > On Wed, Feb 12, 2020 at 12:09 PM Jacek Pliszka > wrote: > > > > Hi! > > > > ARROW-3329 - we can discuss there. > > > > > It

Re: [ARROW-3329] Re: Decimal casting or scaling

2020-02-12 Thread Wes McKinney
On Wed, Feb 12, 2020 at 12:09 PM Jacek Pliszka wrote: > > Hi! > > ARROW-3329 - we can discuss there. > > > It seems like it makes sense to implement both lossless safe casts > > (when all zeros after the decimal point) and lossy casts (fractional > > part discarded) from decimal to integer, do I

[jira] [Created] (ARROW-7841) pyarrow release 0.16.0 breaks `libhdfs.so` loading mechanism

2020-02-12 Thread Jack Fan (Jira)
Jack Fan created ARROW-7841: --- Summary: pyarrow release 0.16.0 breaks `libhdfs.so` loading mechanism Key: ARROW-7841 URL: https://issues.apache.org/jira/browse/ARROW-7841 Project: Apache Arrow

[ARROW-3329] Re: Decimal casting or scaling

2020-02-12 Thread Jacek Pliszka
Hi! ARROW-3329 - we can discuss there. > It seems like it makes sense to implement both lossless safe casts > (when all zeros after the decimal point) and lossy casts (fractional > part discarded) from decimal to integer, do I have that right? Yes, though if I understood your examples are the

Re: Decimal casting or scaling

2020-02-12 Thread Wes McKinney
hi Jacek, What is the JIRA issue for this change? In the interest of organizing the discussion (may make sense to move some of this to that issue) There are no casts implemented DecimalType at all in [1], either to decimal or from decimal to anything else. It seems like it makes sense to

Re: Arrow doesn't have a MapType

2020-02-12 Thread Wes McKinney
It was added between 0.15.0 and 0.16.0. Any feedback from using it would be welcome https://github.com/apache/arrow/commit/e0c1ffe9c38d1759f1b5311f95864b0e2a406c51 On Wed, Feb 12, 2020 at 5:12 AM Shawn Yang wrote: > > Thanks François, I didn't find it in pyarrow. I'll check again. > > On Fri,

[jira] [Created] (ARROW-7840) [Java] [Integration] Java executables fail

2020-02-12 Thread Antoine Pitrou (Jira)
Antoine Pitrou created ARROW-7840: - Summary: [Java] [Integration] Java executables fail Key: ARROW-7840 URL: https://issues.apache.org/jira/browse/ARROW-7840 Project: Apache Arrow Issue

[NIGHTLY] Arrow Build Report for Job nightly-2020-02-12-0

2020-02-12 Thread Crossbow
Arrow Build Report for Job nightly-2020-02-12-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2020-02-12-0 Failed Tasks: - test-conda-python-3.7-turbodbc-latest: URL:

Decimal casting or scaling

2020-02-12 Thread Jacek Pliszka
Hi! I am interested in having cast from Decimal to Int in pyarrow. I have couple ideas but I am a newbie so I might be wrong: Do I understand correctly that the problem lies in the fact that CastFunctor knows nothing about decimal scale? Were there any ideas how to handle this properly? My

Re: Arrow doesn't have a MapType

2020-02-12 Thread Shawn Yang
Thanks François, I didn't find it in pyarrow. I'll check again. On Fri, Feb 7, 2020 at 9:18 PM Francois Saint-Jacques < fsaintjacq...@gmail.com> wrote: > Arrow does have a Map type [1][2][3]. It is represented as a list of pairs. > > François > > [1] >

[jira] [Created] (ARROW-7839) [Python][Dataset] Add IPC format to python bindings

2020-02-12 Thread Joris Van den Bossche (Jira)
Joris Van den Bossche created ARROW-7839: Summary: [Python][Dataset] Add IPC format to python bindings Key: ARROW-7839 URL: https://issues.apache.org/jira/browse/ARROW-7839 Project: Apache

[jira] [Created] (ARROW-7838) [C++] Installed plasma-store-server fails finding Boost

2020-02-12 Thread Antoine Pitrou (Jira)
Antoine Pitrou created ARROW-7838: - Summary: [C++] Installed plasma-store-server fails finding Boost Key: ARROW-7838 URL: https://issues.apache.org/jira/browse/ARROW-7838 Project: Apache Arrow