Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-10 Thread Wes McKinney
I did my best to remove the class from the GLib bindings -- there are probably some conventions around API versions that I did not respect, so I will need some help from others on GLib and Ruby. MATLAB and R are also affected, but should be relatively simple to change. I'll wait to hear more feed

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Wes McKinney
Thanks for the feedback. I just posted a PR that removes the class in the C++ and Python libraries, hopefully this will help with the discussion. That I was able to do it in less than a day should be good evidence that the abstraction may be superfluous https://github.com/apache/arrow/pull/4841

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Tim Swast
FWIW, I found the Column class to be confusing in Python. It felt redundant / unneeded to actually create Tables. On Tue, Jul 9, 2019 at 11:19 AM Wes McKinney wrote: > On Tue, Jul 9, 2019 at 1:14 PM Antoine Pitrou wrote: > > > > > > Le 08/07/2019 à 23:17, Wes McKinney a écrit : > > > > > > I'm

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Wes McKinney
On Tue, Jul 9, 2019 at 1:14 PM Antoine Pitrou wrote: > > > Le 08/07/2019 à 23:17, Wes McKinney a écrit : > > > > I'm concerned about continuing to maintain the Column class as it's > > spilling complexity into computational libraries and bindings alike. > > > > The Python Column class for example

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Antoine Pitrou
Le 08/07/2019 à 23:17, Wes McKinney a écrit : > > I'm concerned about continuing to maintain the Column class as it's > spilling complexity into computational libraries and bindings alike. > > The Python Column class for example mostly forwards method calls to > the underlying ChunkedArray > >

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Francois Saint-Jacques
I'm also +1 on removing this class. François On Tue, Jul 9, 2019 at 10:57 AM Uwe L. Korn wrote: > > This sounds fine to me, thus I'm +1 on removing this class. > > On Tue, Jul 9, 2019, at 2:11 PM, Wes McKinney wrote: > > Yes, the schema would be the point of truth for the Field. The ChunkedArray

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Wes McKinney
I'll try to spend a little time soon refactoring to see how disruptive the change would be, and also to help persuade others about the benefits. On Tue, Jul 9, 2019 at 9:57 AM Uwe L. Korn wrote: > > This sounds fine to me, thus I'm +1 on removing this class. > > On Tue, Jul 9, 2019, at 2:11 PM, W

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Uwe L. Korn
This sounds fine to me, thus I'm +1 on removing this class. On Tue, Jul 9, 2019, at 2:11 PM, Wes McKinney wrote: > Yes, the schema would be the point of truth for the Field. The ChunkedArray > type would have to be validated against the schema types as with RecordBatch > > On Tue, Jul 9, 2019, 2:

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Wes McKinney
Yes, the schema would be the point of truth for the Field. The ChunkedArray type would have to be validated against the schema types as with RecordBatch On Tue, Jul 9, 2019, 2:54 AM Uwe L. Korn wrote: > Hello Wes, > > where do you intend the Field object living then? Would this be part of > the

Re: [DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-09 Thread Uwe L. Korn
Hello Wes, where do you intend the Field object living then? Would this be part of the schema of the Table object? Uwe On Mon, Jul 8, 2019, at 11:18 PM, Wes McKinney wrote: > hi folks, > > For some time now I have been uncertain about the utility provided by > the arrow::Column C++ class. Fund

[DISCUSS][C++] Evaluating the arrow::Column C++ class

2019-07-08 Thread Wes McKinney
hi folks, For some time now I have been uncertain about the utility provided by the arrow::Column C++ class. Fundamentally, it is a container for two things: * An arrow::Field object (name and data type) * An arrow::ChunkedArray object for the data It was added to the C++ library in ARROW-23 in