Re: [VOTE] Release Apache Parquet 1.11.0 RC6

2019-04-11 Thread Andy Grove
I'm curious if there is any update on this vote? The thread seems eerily quiet. Thanks. On 4/3/19, 10:38 AM, "Andy Grove" wrote: CAUTION – UNVERIFIED EXTERNAL EMAIL I have been able to run mvn verify and have also tested this RC against our internal systems, wit

Re: [VOTE] Release Apache Parquet 1.11.0 RC6

2019-04-03 Thread Andy Grove
I have been able to run mvn verify and have also tested this RC against our internal systems, with no issue. +1 (non-binding) I have raised the issue about Hadoop-lzo, but that is present in the 1.10.1 release also. Andy. On 3/20/19, 7:50 AM, "Zoltan Ivanfi" wrote: CAUTION –

[jira] [Commented] (PARQUET-1556) Add twitter maven repo to POM for hadoop-lzo dependency

2019-04-03 Thread Andy Grove (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-1556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808868#comment-16808868 ] Andy Grove commented on PARQUET-1556: - I'm also curious to understand the implications of adding

Re: [VOTE] Release Apache Parquet 1.11.0 RC6

2019-04-03 Thread Andy Grove
It works after configuring the twitter Maven repo. Here is the JIRA: https://issues.apache.org/jira/browse/PARQUET-1556 On 4/3/19, 9:07 AM, "Andy Grove" wrote: CAUTION – UNVERIFIED EXTERNAL EMAIL This artifact isn't available in maven central. It is in Twitter's m

[jira] [Created] (PARQUET-1556) Instructions are missing for configuring twitter maven repo for hadoop-lzo dependency

2019-04-03 Thread Andy Grove (JIRA)
Andy Grove created PARQUET-1556: --- Summary: Instructions are missing for configuring twitter maven repo for hadoop-lzo dependency Key: PARQUET-1556 URL: https://issues.apache.org/jira/browse/PARQUET-1556

Re: [VOTE] Release Apache Parquet 1.11.0 RC6

2019-04-03 Thread Andy Grove
wishes, > > Le mer. 3 avr. 2019 à 13:33, Andy Grove a écrit : > > > I've been watching the RC process here and would like to help verify the > > release, since I have co-workers wanting to upgrade to 1.11 > > > > When I follow the instruc

Re: [VOTE] Release Apache Parquet 1.11.0 RC6

2019-04-03 Thread Andy Grove
I've been watching the RC process here and would like to help verify the release, since I have co-workers wanting to upgrade to 1.11 When I follow the instructions to verify the release I am running into an issue resolving a dependency: Could not resolve dependencies for project

Re: [DISCUSS] Rust add adapter for parquet

2018-11-21 Thread Andy Grove
release if we go with nightly now. Thanks, Andy. On Wed, Nov 21, 2018 at 7:37 AM Renjie Liu wrote: > That sounds great. But parquet-rs currently relies on nightly rust, that > would be the first problem to resolve. > > On Wed, Nov 21, 2018 at 4:49 AM Andy Grove wrote: > >

Re: [DISCUSS] Rust add adapter for parquet

2018-11-20 Thread Andy Grove
This sounds like a great idea. With support for both CSV and Parquet in the Arrow crate, it would be nice to design a standard interface for Arrow data sources. Maybe this is as simple as implementing `Iterator`. Andy. On Tue, Nov 20, 2018 at 11:46 AM Chao Sun wrote: > Yes, we'd be interested

[jira] [Commented] (PARQUET-1390) [Java] Upgrade to Arrow 0.10.0

2018-08-18 Thread Andy Grove (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-1390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584793#comment-16584793 ] Andy Grove commented on PARQUET-1390: - PR: [https://github.com/apache/parquet-mr/pull/516

[jira] [Created] (PARQUET-1390) Upgrade to Arrow 0.10.0

2018-08-18 Thread Andy Grove (JIRA)
Andy Grove created PARQUET-1390: --- Summary: Upgrade to Arrow 0.10.0 Key: PARQUET-1390 URL: https://issues.apache.org/jira/browse/PARQUET-1390 Project: Parquet Issue Type: Improvement

Reading arrays from Java using ParquetFileReader

2018-07-23 Thread Andy Grove
I‘m using ParquetFileReader/ParquetPageReader to scan parquet files and apply a projection. This is working well for primitive column types but I’m running into an issue when trying at add support for arrays and could use some help. I’m retrieving the schema like this: val r = new

Re: [Java] Arrow/Parquet use different major Java versions

2018-04-23 Thread Andy Grove
This was intended for the Arrow mailing list, not the Parquet mailing list, of course, so will send there. I know there’s some overlap between the groups, so sorry for the extra noise. Andy. From: Andy Grove <andy.gr...@rms.com> Date: Monday, April 23, 2018 at 11:36 AM To

[jira] [Created] (PARQUET-1278) [Java] parquet-arrow is broken due to different JDK version

2018-04-23 Thread Andy Grove (JIRA)
Andy Grove created PARQUET-1278: --- Summary: [Java] parquet-arrow is broken due to different JDK version Key: PARQUET-1278 URL: https://issues.apache.org/jira/browse/PARQUET-1278 Project: Parquet

[Java] Arrow/Parquet use different major Java versions

2018-04-23 Thread Andy Grove
I’m trying to use the parquet-arrow library, which has just been updated to use Arrow 0.8.0 but unfortunately I am still running into this issue: java.lang.ClassNotFoundException: org.apache.arrow.vector.types.pojo.ArrowType$Struct_ The class in the arrow jar is actually `Struct` not

Brotli-codec dependency issue when building from source

2018-04-21 Thread Andy Grove
Hi, I’ve been following the instructions in the README to get parquet-mr building locally but I am running into this dependency issue: [ERROR] Failed to execute goal on project parquet-hadoop: Could not resolve dependencies for project org.apache.parquet:parquet-hadoop:jar:1.10.1-SNAPSHOT:

[jira] [Commented] (PARQUET-1128) [Java] Upgrade the Apache Arrow version to 0.8.0 for SchemaConverter

2018-04-20 Thread Andy Grove (JIRA)
[ https://issues.apache.org/jira/browse/PARQUET-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446412#comment-16446412 ] Andy Grove commented on PARQUET-1128: - What is the status of this? I am running into similar issues

Re: Specifying a projection in Java API

2018-04-13 Thread Andy Grove
OK sorry for all the messages but I have this working now: On 4/13/18, 12:59 PM, "Andy Grove" <andy.gr...@rms.com> wrote: Immediately after sending this I realized that I also needed to pass the projection message type in the following lines: va

Re: Specifying a projection in Java API

2018-04-13 Thread Andy Grove
up.java:97) at org.apache.parquet.example.data.simple.SimpleGroup.getInteger(SimpleGroup.java:129) at org.apache.parquet.example.data.GroupValueSource.getInteger(GroupValueSource.java:39) On 4/13/18, 12:56 PM, "Andy Grove" <andy.gr...@rms.com> wrote: Thanks. I tried this.

Re: Specifying a projection in Java API

2018-04-13 Thread Andy Grove
/13/18, 12:08 PM, "Ryan Blue" <rb...@netflix.com.INVALID> wrote: I'd suggest using the Types builders to create your projection schema (MessageType), then passing that schema to the ParquetFileReader.setRequestedSchema method you found. On Fri, Apr 13, 2018 a

Re: Specifying a projection in Java API

2018-04-13 Thread Andy Grove
or something. On Fri, Apr 13, 2018 at 10:31 AM, Andy Grove <andy.gr...@rms.com> wrote: > Hi, > > I’m trying to read a parquet file with a projection from Scala and I can’t > find docs or examples for the correct way to do this. > > I have the fi

Specifying a projection in Java API

2018-04-13 Thread Andy Grove
Hi, I’m trying to read a parquet file with a projection from Scala and I can’t find docs or examples for the correct way to do this. I have the file schema and have filtered for the list of columns I need, so I have a List of ColumnDescriptors. It looks like I should call