Fwd: Exposing low-level Parquet encryption to Python user (or, maybe not)

2021-03-09 Thread Gidon Gershinsky
Hi Antoine, All comments have been handled. Can we ask you to shepherd this PR for the reminder of its lifecycle? (hopefully, most of this is already behind us). https://github.com/apache/arrow/pull/8023 Cheers, Gidon -- Forwarded message - From: Gidon Gershinsky Date: Thu, Fe

Re: [Flight Extension] Request for Comments

2021-03-09 Thread Micah Kornfield
> > As for schema evolution, I agree with what Micah proposes as a first step. > That would again add some overhead, perhaps. As for feasibility, at least > on the C++/Python side, I think there would be a decent amount of > refactoring needed, and there's also the question of how to expose this in

Re: [DISCUSS] integration against parquet files?

2021-03-09 Thread Micah Kornfield
IIUC the only difference between 1 and 2 is whether to use the serialized schema from the parquet file in reconstruction? Maybe we can pass that as an additional flag through archery in the short term so we can already reuse the JSON description we have for arrow files? On Tue, Mar 9, 2021 at 3:5

Re: [C++] libarrow isolation

2021-03-09 Thread Tahsin Hassan
Hi Wes, Thanks for providing the feedback. 1. Regarding the symbol versioning change As a next step, should I open up a Jira issue and assign myself to see whether I can automate adding Macros BEGIN_ARROW_NS/ END_ARROW_NS ? using LLVM clang AST matchers. We can submit this as a one-time mecha

[C++] Kicking off some query execution buildout work in C++

2021-03-09 Thread Wes McKinney
hi all In light of https://github.com/apache/arrow/pull/9621 and some related issues, we're planning to start building out some query execution machinery in C++, with the goal of plumbing together the existing Datasets API with essential relational algorithms (projection, aggregation, filter, ev

Re: [Flight Extension] Request for Comments

2021-03-09 Thread David Li
There's not really any convention for the app_metadata field or any of the other application-defined fields (e.g. DoAction, Criteria). That said, I wouldn't necessarily worry about conflicting with other projects - if a client connects to a Barrage service, presumably it knows what to expect. An

Re: [JIRA Permissions] Requesting change to "Contributor" role

2021-03-09 Thread Fiona La
Hello Kou, Thank you! I am able to assign the issue to myself now. Best regards, Fiona From: Sutou Kouhei Date: Tuesday, March 9, 2021 at 3:40 PM To: dev@arrow.apache.org Subject: Re: [JIRA Permissions] Requesting change to "Contributor" role Done. Could you try it again? Thanks, -- kou In

Re: [JIRA Permissions] Requesting change to "Contributor" role

2021-03-09 Thread Sutou Kouhei
Done. Could you try it again? Thanks, -- kou In "[JIRA Permissions] Requesting change to "Contributor" role" on Tue, 9 Mar 2021 20:29:46 +, Fiona La wrote: > Hi all, > > I just submitted a Jira issue > (https://issues.apache.org/jira/browse/ARROW-11919) and I would like to > reque

[JIRA Permissions] Requesting change to "Contributor" role

2021-03-09 Thread Fiona La
Hi all, I just submitted a Jira issue (https://issues.apache.org/jira/browse/ARROW-11919) and I would like to request to be added as a Jira contributor to assign the issue to myself. >From the following Jira issue >(https://issues.apache.org/jira/browse/ARROW-11852), the steps for an Arrow >J

[DISCUSS] [Rust] Donate Ballista to Apache Arrow

2021-03-09 Thread Andy Grove
As many of you know, the reason that I got involved in Arrow back in 2018 was that I wanted to build a distributed compute platform in Rust, with capabilities similar to Apache Spark. This led to the creation of the DataFusion query engine, which is an in-memory query engine and is now part of the

Re: [DISCUSS] integration against parquet files?

2021-03-09 Thread Jorge Cardoso Leitão
Thanks a lot for your comments, and for the pointers. I think that we could separate this in 3 separate parts: 1. arrow implementation X (e.g. C++) reads a on-spec parquet file 2. arrow implementation X read a parquet file written by arrow implementation Y 3. arrow implementation X reads an parqu

Fwd: Call for Presentations for ApacheCon 2021 now open

2021-03-09 Thread Zoran Regvart
Hi folk, On the backs of the resounding success from last year's, ApacheCon@Home will be online, and again, we'll be featuring the (Software) Integration track. I'm the chair of the Integration track at ApacheCon@Home. And I'd like to invite folk from the Apache Arrow community to participate by su

[NIGHTLY] Arrow Build Report for Job nightly-2021-03-09-0

2021-03-09 Thread Crossbow
Arrow Build Report for Job nightly-2021-03-09-0 All tasks: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-09-0 Failed Tasks: - conda-linux-gcc-py37-aarch64: URL: https://github.com/ursacomputing/crossbow/branches/all?query=nightly-2021-03-09-0-drone-conda-linux

Fwd: Call for Presentations for ApacheCon 2021 now open

2021-03-09 Thread Zoran Regvart
Hi folk, On the backs of the resounding success from last year's, ApacheCon@Home will be online, and again, we'll be featuring the (Software) Integration track. I'm the chair of the Integration track at ApacheCon@Home. And I'd like to invite folk from the Apache Arrow community to participate by su