I agree that we should aim for time-based releases. Let's discuss a
time-based release schedule (my preference would be ~every 2 months)
for 2019 after we get 0.12 out.
On Wed, Dec 12, 2018 at 3:15 AM Antoine Pitrou <anto...@python.org> wrote:
>
>
> I think we should aim for time-based releases in general (rather than a
> specific set of features), but delaying this one sounds good to me.
>
> Regards
>
> Antoine.
>
>
> Le 12/12/2018 à 01:34, Wes McKinney a écrit :
> > hi all,
> >
> > I'm looking at the 0.12 backlog and I am not too comfortable with the
> > things that would have to be cut to get a release out next week.
> > Additionally, not a lot of developers are going to be working the week
> > of December 24 because of the Christmas and New Year's holidays, so
> > even if we did release, it might not get seen by a lot of people until
> > after the New Year.
> >
> > Based on this, I would suggest we push to complete as much work as
> > possible (from the 0.12 backlog and beyond) by the end of the year,
> > and release as soon as possible in 2019. Of course, anyone is welcome
> > to contribute work that is not found in the 0.12 milestone =)
> >
> > Any objections?
> >
> > Thanks
> > Wes
> > On Mon, Dec 10, 2018 at 8:04 AM Andy Grove <andygrov...@gmail.com> wrote:
> >>
> >> Cool. I will continue to add primitive operations but I am now adding this
> >> in a separate source file to keep it separate from the core array code.
> >>
> >> I'm not sure how important it will be to support Rust data sources with
> >> Gandiva. I can see that each language should be able to construct the
> >> logical query plan to submit to Gandiva and let Gandiva handle execution. I
> >> think the more interesting part is how do we support language-specific
> >> lambda functions as part of that logical query plan. Maybe it is possible
> >> to compile the lambda down to LLVM (I haven't started learning about LLVM
> >> in detail yet so this is wild speculation on my part). Another option is
> >> for Gandiva to support calling into shared libraries and that maybe is
> >> simpler for languages that support building C-native shared libraries (Rust
> >> supports this with zero overhead).
> >>
> >> Andy.
> >>
> >>
> >>
> >>
> >> On Sun, Dec 9, 2018 at 11:42 AM Wes McKinney <wesmck...@gmail.com> wrote:
> >>
> >>> hi Andy,
> >>>
> >>> I can see an argument for having some basic native function kernel
> >>> support in Rust. One of the things that Gandiva has begun is a
> >>> Protobuf-based serialized representation representation of projection
> >>> and filter expressions. In the long run I would like to see a more
> >>> complete relational algebra / logical query plan that can be submitted
> >>> for execution. There's complexities, though, such as bridging
> >>> iteration of data sources written in Rust, say, with a query engine
> >>> written in C++. You would need to provide some kind of a callback
> >>> mechanism for the query engine to request the next chunk of a dataset
> >>> to be materialized.
> >>>
> >>> It will be interested to see what contributors will be motivated
> >>> enough to build over the next few years. At the end of the day, Apache
> >>> projects are do-ocracies.
> >>>
> >>> - Wes
> >>> On Fri, Dec 7, 2018 at 6:22 AM Andy Grove <andygrov...@gmail.com> wrote:
> >>>>
> >>>> I've added one PR to the list (https://github.com/apache/arrow/pull/3119
> >>> )
> >>>> to update the project to use Rust 2018 Edition.
> >>>>
> >>>> I'm also considering removing one PR from the list and would like to get
> >>>> opinions here.
> >>>>
> >>>> I have a PR (https://github.com/apache/arrow/pull/3033) to add some
> >>> basic
> >>>> math and comparison operators to primitive arrays. These are baby steps
> >>>> towards implementing more query execution capabilities such as
> >>> projection,
> >>>> selection, etc but Chao made a good point that other Rust implementations
> >>>> don't have these kind of capabilities and I am now wondering if this is a
> >>>> distraction. We already have Gandiva and the new efforts in Ursa labs and
> >>>> it would probably make more sense to look at having Rust bindings for the
> >>>> query execution capabilities there rather than having a competing (and
> >>> less
> >>>> capable) implementation in Rust.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> Andy.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Dec 6, 2018 at 8:42 PM paddy horan <paddyho...@hotmail.com>
> >>> wrote:
> >>>>
> >>>>> Other than Andy’s PR below I’m going to try and find time to work on
> >>>>> ARROW-3827, I’ll bump it 0.13 if I can’t find the time early next week.
> >>>>> There is nothing else in the 0.12 backlog for Rust. It would be nice
> >>> to
> >>>>> get the parquet merge in though.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Paddy
> >>>>>
> >>>>>
> >>>>>
> >>>>> ________________________________
> >>>>> From: Andy Grove <andygrov...@gmail.com>
> >>>>> Sent: Thursday, December 6, 2018 10:20:48 AM
> >>>>> To: dev@arrow.apache.org
> >>>>> Subject: Re: Timeline for Arrow 0.12.0 release
> >>>>>
> >>>>> I have PRs pending for all the Rust issues that I want to get into
> >>> 0.12.0
> >>>>> and would appreciate some reviews so I can go ahead and merge:
> >>>>>
> >>>>> https://github.com/apache/arrow/pull/3033 (covers ARROW-3880 and
> >>>>> ARROW-3881
> >>>>> - add math and comparison operations to primitive arrays)
> >>>>> https://github.com/apache/arrow/pull/3096 (ARROW-3885 - Rust release
> >>>>> process)
> >>>>> https://github.com/apache/arrow/pull/3111 (ARROW-3838 - CSV Writer)
> >>>>>
> >>>>> With these in place I plan on writing a tutorial for reading a CSV
> >>> file,
> >>>>> performing some operations on primitive arrays and writing the output
> >>> to a
> >>>>> new CSV file.
> >>>>>
> >>>>> I am deferring ARROW-3882 (casting for primitive arrays) to 0.13.0
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Andy.
> >>>>>
> >>>>> On Tue, Dec 4, 2018 at 7:57 PM Andy Grove <andygrov...@gmail.com>
> >>> wrote:
> >>>>>
> >>>>>> I'd love to tackle the three related issues for supporting simple
> >>>>>> math/comparison operations on primitive arrays and casting primitive
> >>>>> arrays
> >>>>>> but since the change to use Rust specialization feature I'm a bit
> >>> stuck
> >>>>> and
> >>>>>> need some assistance applying the math operations to the numeric
> >>> types
> >>>>> and
> >>>>>> not the boolean primitives. I have added a comment to
> >>>>>> https://github.com/apache/arrow/pull/3033 ... if I can get help
> >>> solving
> >>>>>> for this PR then I should be able to handle the others. I'll also do
> >>> some
> >>>>>> research and try and figure this out myself.
> >>>>>>
> >>>>>> Andy.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Dec 4, 2018 at 7:03 PM Wes McKinney <wesmck...@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>>> Andy, Paddy, or other Rust developers -- could you review the 6
> >>> issues
> >>>>>>> in TODO in the 0.12 backlog and either assign them or move them to
> >>> the
> >>>>>>> next release if they aren't going to be completed this week or next?
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, Nov 30, 2018 at 4:34 PM Wes McKinney <wesmck...@gmail.com>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> hi folks,
> >>>>>>>>
> >>>>>>>> Tomorrow is December 1. The last major Arrow release (0.11.0) took
> >>>>>>>> place on October 8. Given how much work has happened in the
> >>> project in
> >>>>>>>> the last ~2 months, I think it would be great to complete the next
> >>>>>>>> major release before the end-of-year holidays set in.
> >>>>>>>>
> >>>>>>>> I've been curating the JIRA backlog the last couple of weeks, and
> >>> have
> >>>>>>>> just created a 0.12.0 release wiki page to help us stay organized
> >>>>>>>>
> >>>>>>>>
> >>>>> https://cwiki.apache.org/confluence/display/ARROW/Arrow+0.12.0+Release
> >>>>>>>>
> >>>>>>>> Given that there are only 3 full working weeks between now and
> >>>>>>>> Christmas, I think we should be in position to cut a release by
> >>> the
> >>>>>>>> end of the week of December 10, i.e. by Friday December 14. Not
> >>> all of
> >>>>>>>> the TODO issues have to be completed to make the release, but it
> >>> would
> >>>>>>>> be good to push to complete as much as possible. Please help by
> >>>>>>>> reviewing the backlog, and if possible, assigning issues to
> >>> yourself
> >>>>>>>> that you'd like to pursue in the next 2 weeks.
> >>>>>>>>
> >>>>>>>> Let me know if this sounds reasonable, or any concerns.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> Wes
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
