Re: [DISCUSS] Rotating the PMC Chair

2020-09-29 Thread Micah Kornfield
> > Re now vs Feb 17: I'm totally open to either. In general, I'm a do it now > kind of person so if others think a slightly longer tenure sounds good, we > could do it now. This makes sense to me as well, we can align on Feb 17th the next time around. Per my comments elsewhere, I'd like to

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-29 Thread Neville Dipale
+1 (non-binding) Rust support is behind, but we'll catch up at some point. On Wed, 30 Sep 2020 at 03:10, Holden Karau wrote: > +1 (non-binding) > > On Tue, Sep 29, 2020 at 6:08 PM Sutou Kouhei wrote: > > > +1 > > > > In > > "Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow > >

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-29 Thread Holden Karau
+1 (non-binding) On Tue, Sep 29, 2020 at 6:08 PM Sutou Kouhei wrote: > +1 > > In > "Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow > specification" on Tue, 29 Sep 2020 13:38:04 -0700, > Jacques Nadeau wrote: > > > +1 > > > > On Tue, Sep 29, 2020 at 11:19 AM Wes McKinney >

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-29 Thread Sutou Kouhei
+1 In "Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification" on Tue, 29 Sep 2020 13:38:04 -0700, Jacques Nadeau wrote: > +1 > > On Tue, Sep 29, 2020 at 11:19 AM Wes McKinney wrote: > >> +1 >> >> On Tue, Sep 29, 2020 at 4:07 AM Fan Liya wrote: >> > >> > +1 >> > >>

Re: [DISCUSS] Rethinking our approach to scheduling CPU and IO work in C++?

2020-09-29 Thread Pierre Belzile
Hi, Some thoughts: 1. For async IO, the system must have threads that quickly service the callback. Otherwise the S3/GCS end will close the connection. A single thread pool where all the threads are doing an expensive compute operation (like CSV decoding or regex matching) can starve the IO. 2.

Re: 2.0.0 release timeline: October 9

2020-09-29 Thread Neal Richardson
Thanks Neville. I also just bumped 60 C++/Python issues (new features, unassigned, not created recently) from 2.0. We're at 122 now, which still needs some pruning. I think something that more folks can do is review the issues assigned to them and make sure that they're planning to get all of

Re: MEAN Stack Use-case understanding

2020-09-29 Thread Paul Taylor
Hi Thomas, You can read CSVs in the browser using the browser's File input APis and an appropriate CSV library. The CSV library should be able to parse rows into JS objects, which can then be passed to the Arrow Struct Builder for serialization. In this example[1] I'm parsing the first row

Re: 2.0.0 release timeline: October 9

2020-09-29 Thread Neville Dipale
Hi Neal, I've pruned the Rust backlog a bit, but only changed PRs that I've either opened, or those that I presume nobody is currently working on. There's 2 major features I've been working on: 1. Writing Arrow data to Parquet (separate branch) 2. Integration testing I'll prioritise 2 as

Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC

2020-09-29 Thread Ryan Nicholson
Hello Andy and Wes, Thanks for reaching out. Later this week I have time freeing up to start on Flight-SQL. Would either of you mind examining the approach in the POC [1] as described in the "Proof of Concept" section of the proposal document [2]? I would certainly be happy to chat about this

2.0.0 release timeline: October 9

2020-09-29 Thread Neal Richardson
Hi folks, As has been discussed in the biweekly meetings (and in the notes from those meetings here on the mailing list), we're looking at an October timeline for our next release since we are going about 3 months between releases. So that we might get the release voted on and shipped by the

Arrow sync call September 30 at 12:00 US/Eastern, 16:00 UTC

2020-09-29 Thread Neal Richardson
Hi all, Reminder that our biweekly call is coming up tomorrow at https://meet.google.com/vtm-teks-phx. All are welcome to join. Notes will be sent out to the mailing list afterward. Neal

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-29 Thread Jacques Nadeau
+1 On Tue, Sep 29, 2020 at 11:19 AM Wes McKinney wrote: > +1 > > On Tue, Sep 29, 2020 at 4:07 AM Fan Liya wrote: > > > > +1 > > > > Best, > > Liya Fan > > > > On Tue, Sep 29, 2020 at 4:55 PM Antoine Pitrou > wrote: > > > > > > > > +1 (binding) > > > > > > I didn't look at the implementation.

Re: [DISCUSS] Rotating the PMC Chair

2020-09-29 Thread Jacques Nadeau
I'm super supportive of this, Julian. Thanks for bringing it up. Unlike some leaders, I'm even happy to guarantee a peaceful transition of power! Re now vs Feb 17: I'm totally open to either. In general, I'm a do it now kind of person so if others think a slightly longer tenure sounds good, we

Re: [DISCUSS] Rotating the PMC Chair

2020-09-29 Thread Julian Hyde
Feb 17 as a term end date sounds good. We don’t necessarily have to wait until Feb 17. If we want to move on this sooner, the next Chair could serve a slightly longer term, ending on Feb 17 2022. > On Sep 29, 2020, at 11:25 AM, Wes McKinney wrote: > > This sounds reasonable to me. We are

Re: [DISCUSS] Rotating the PMC Chair

2020-09-29 Thread Wes McKinney
This sounds reasonable to me. We are approaching our 5 year anniversary as an Apache project (on February 17, 2021) and I have thought that it would make sense to rotate PMC chairs at some point, but making it an annual thing makes things even simpler. For what it's worth, in this project the PMC

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-29 Thread Wes McKinney
+1 On Tue, Sep 29, 2020 at 4:07 AM Fan Liya wrote: > > +1 > > Best, > Liya Fan > > On Tue, Sep 29, 2020 at 4:55 PM Antoine Pitrou wrote: > > > > > +1 (binding) > > > > I didn't look at the implementation. > > > > Regards > > > > Antoine. > > > > > > Le 29/09/2020 à 06:54, Micah Kornfield a

Re: [FlightRPC] Add a "Flight SQL" extension on top of FlightRPC

2020-09-29 Thread Andy Grove
I am also interested in helping with this effort. Andy. On Mon, Sep 28, 2020, 1:58 PM Wes McKinney wrote: > hi Ryan, > > I'm interested in enabling this effort to move forward -- is there any > part where feedback or other help could be useful? > > Thanks, > Wes > > On Wed, Sep 2, 2020 at 4:45

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-29 Thread Fan Liya
+1 Best, Liya Fan On Tue, Sep 29, 2020 at 4:55 PM Antoine Pitrou wrote: > > +1 (binding) > > I didn't look at the implementation. > > Regards > > Antoine. > > > Le 29/09/2020 à 06:54, Micah Kornfield a écrit : > > I've opened a PR that updates the specification to allow for 256-bit > > Decimal

Re: [VOTE][Format] Allow for 256-bit Decimal's in the Arrow specification

2020-09-29 Thread Antoine Pitrou
+1 (binding) I didn't look at the implementation. Regards Antoine. Le 29/09/2020 à 06:54, Micah Kornfield a écrit : > I've opened a PR that updates the specification to allow for 256-bit > Decimal types [1]. It updates both schema.fbs and the C-ABI to document > this support. > > The

Re: [Rust] Arrow SQL Adapters/Connectors

2020-09-29 Thread Julian Hyde
ODBC and JDBC do not specify a wire protocol. So, while the client APIs are definitely row-based, any particular driver could use a protocol that is based on Arrow data. There is immense investment in ODBC and JDBC drivers, and they handle complex cases such as connection pooling, statement

Re: [DISCUSS] Rethinking our approach to scheduling CPU and IO work in C++?

2020-09-29 Thread Weston Pace
Antoine/Wes, thanks for the input. I will focus on the CSV reader and the minimal async needed to get I/O off the thread pool and support for a nested task group. This is just to focus on one small thing at a time. I'll avoid any scheduler work for now but maybe can look at that in the future.

Re: [Rust] Arrow SQL Adapters/Connectors

2020-09-29 Thread Sven Wagner-Boysen
I think this is a great initiative. If I understand correctly, it would open up Arrow for many more use cases and allow for example to connect BI Tools like PowerBi, Tableau, etc to DataFusion. I'll also try to make some time to support this. Thanks! Sven On Mon, Sep 28, 2020 at 3:49 AM Andy

[DISCUSS] Rotating the PMC Chair

2020-09-29 Thread Julian Hyde
There has been some discussion in the Arrow PMC about rotating the PMC Chair (also known as the project VP) every year. I wanted to raise the topic here for discussion among Arrow committers and within the broader Arrow community. Quite a few Apache projects have adopted a policy where they