+1 for synchronizing to the main releases when possible. In the 0.12
thread we have discussed moving to time-based releases (e.g. every 2
months). Time-based releases are helpful to create urgency around
getting work completed, and making sure that the project is always
ready to release.
On Thu, Dec 13, 2018 at 10:39 AM Brian Hulette <hulet...@gmail.com> wrote:
>
> Sounds great Paul! Really excited that this refactor is wrapping up. My
> only concern with including this in 0.4.0 is that I'm not going to have the
> time to thoroughly review it for a few weeks, so gating on that would
> really delay it. But I can just manually test with some use-cases I care
> about in lieu of a thorough review in the interest of time.
>
> I think in the future (after 0.12?) it may behoove us to tie back in to the
> main Arrow release cycle. The idea with the separate JS release was to
> allow us to release faster, but in practice it has done the opposite. Since
> the fall of 2017 we've cut two major JS releases (0.2, 0.3) while there
> were four major main releases (0.8 - 0.11). Not to mention the disjoint
> version numbers can be confusing to users - perhaps not as much of a
> concern now that the format is pretty stable, but it can still be a
> friction point. And finally selfishly - if we had been on the main release
> cycle, the contributions I made in the summer would have been released in
> either 0.10 or 0.11 by now.
>
> Brian
>
> On Thu, Dec 13, 2018 at 3:29 AM Paul Taylor <ptay...@apache.org> wrote:
>
> > The ongoing JS refactor/upgrade branch
> > <https://github.com/trxcllnt/arrow/tree/js-data-refactor/js> is just
> > about done. It's passing all the integration tests, as well as a hundred
> > or so new unit tests. I have to update existing tests where the APIs
> > changed, battle with closure-compiler a bit, then it'll be ready to
> > merge in and ship out. I think I'll be able to wrap it up in the next
> > couple hours.
> >
> > I started this branch to clean up the Vector Data classes to make it
> > easier to add higher-level Table and Vector operators, but as the Data
> > classes are fairly embedded in the core, it lead to a larger refactor of
> > the DataTypes, Vectors, Visitors, and IPC readers and writers.
> >
> > While I was updating the IPC readers and writers, I took the opportunity
> > to back-port all the Node and WhatWG (browser) streams integration that
> > we've built for Graphistry. Putting it in the Arrow JS library means we
> > can better ensure zero-copy when possible, empowers library consumers to
> > easily build streaming applications in both server and browser
> > environments, and (selfishly) reduces complexity in my code base. It
> > also advances a longer term personal goal to more closely adhere to the
> > structure and organization of ArrowCPP when reasonable.
> >
> > A non-exhaustive list of updates includes:
> >
> > * Updates the Table, Schema, RecordBatch, Visitor, Vector, Data, and
> > DataTypes to ensure the generic type signatures cascade recursively
> > through the type declarations
> > * New io primitives that abstract over the (mutually exclusive) file and
> > stream APIs in both node and browser environments
> > * New RecordBatchReaders and RecordBatchWriters that directly use the
> > zero-copy node and browser io primitives
> > * A consolidated reflective Visitor implementation that supports late
> > binding to shortcut traversal, provides an easy API for building higher
> > level Vector operators
> > * Fixed bugs/added support for reading and writing DictionaryBatch
> > deltas (tricky)
> > * Updated all the dependencies and did some config file gardening to
> > make debugging tests easier
> > * Added a bunch of new tests
> >
> > I'd be more than happy to help shepherd a 0.4.0 release of what's in
> > arrow/master if that's what everyone wants to do. But in the interest of
> > cutting a more feature-rich release and preventing customers paying the
> > cost of updating twice in a short time span, I vote we hold off for
> > another day or two and merge + release the work in the refactor branch.
> >
> > Paul
> >
> > On 12/9/18 10:51 AM, Wes McKinney wrote:
> > > I agree that we should cut a JavaScript release.
> > >
> > > With the amount of maintenance work on my plate I have to declare
> > > bankruptcy on doing any more than I am right now. Can another PMC
> > > volunteer to be the RM for the 0.4.0 JavaScript release?
> > >
> > > Thanks
> > > Wes
> > > On Tue, Dec 4, 2018 at 10:07 PM Brian Hulette<hulet...@gmail.com>
> > wrote:
> > >> Hi all,
> > >> It's been quite a while since our last major Arrow JS release (0.3.0 on
> > >> February 22!), and since then we've added several new features that will
> > >> make Arrow JS much easier to adopt. We've added convenience functions
> > for
> > >> creating Arrow vectors and tables natively in JavaScript, an IPC writer,
> > >> and a row proxy interface that will make integrating with existing JS
> > >> libraries much simpler.
> > >>
> > >> I think it's time we cut 0.4.0, so I spent some time closing out or
> > >> postponing the last few JIRAs in JS-0.4.0. I got it down to just one
> > JIRA
> > >> which involves documenting the release process - hopefully we can close
> > >> that out as we go through it again.
> > >>
> > >> Please let me know if you think it makes sense to cut JS-0.4.0 now, or
> > if
> > >> you have any concerns.
> > >>
> > >> Brian
> >
