Hi, On Friday, 2012-09-14, Josh Wills wrote: > I like the idea of having themes for releases. In my head, the theme of > this release could be either > > a) Hacking the new MSCRPlanner code, esp. to add the ability to fuse > different MSCR jobs into a single instance that it enables, or > b) data access/integration points-- things like solr, hcatalog, hbase, > cassandra, jdbc, etc. as input and output sources for Crunch pipelines, or > c) API refactoring-- the crunch-api/crunch-impl/crunch-lib split, or
I would see c) as part of a larger mission for improving documentation and usability. An immediate benefit would be that we don't have to provide javadoc for each and every class, only for those packages that are client-facing. Higher perceived quality with less work for us. I wouldn't make it a separate release though, perhaps we can do this in a series of smaller steps, starting with the crunch-api split. Refactorings like this usually turn into long frustrating monster tasks that prevent other progress. I'd really like to avoid that. But, before spending any more time on this, I think we should all agree that this is what we really want. Somehow I got the impression that you're not fully convinced that this refactoring is necessary or even a good idea. To me it would feel like people trying to rearrange the furniture in my living room. Let's discuss this here before we produce any more patches. > d) working on a PStream API that would let people apply DoFns to streams > and would build on top of things like WalMart's mupd8 or Storm or whatever. > > Of course, this is in addition to whatever fixes and new lib functions we > want to add over time. I don't want anything heavyweight, but those are > some of the larger-scale things that we'll need to tackle as a community, > and I would think of completing each of those big things as corresponding > to a release. Sounds good to me. Regards, Matthias
