On Tue, Oct 5, 2010 at 8:17 PM, Andrew Purtell <[email protected]> wrote:
> > It's not a matter of viewing it as second-class, but more > > that it's not necessarily within scope for this exact > > release -- in general we have always talked about 0.90 as > > a release where we focus on stability and durability. > > What of replication? > > Or HFOF? > > Both of these went into our very first 0.89 dev release this summer, and have been tried out on clusters for multiple dev releases since. In replication's case, it's only been used so far at StumbleUpon AFAIK, but the new HFOF support has been used in real workloads at at least 5 companies off the top of my head, three of which do not employ any contributors. Like i said above, I think new APIs and new big features should go in early in a DR series, not right before the release itself. If we could hold up the 0.90 release for enough time to do a dev release or two with coprocessors, I'd be all for it. But given the aims of the release, I don't think coprocessors are a blocking feature. > Or the recent Result changes? > > Honestly I've been too swamped to look at these, but if it's a substantial new API I also think it should be held for 0.92 as well. If it's a few new function calls to existing classes, it seems OK to put it in late. > None are in scope for a stability and durability release. > > HBASE-2001 has been little changed for 6 months. We've made incremental > changes to the server side framework but it's been basically as stable as > possible given the changes around it. Then, Gary added a big push with a lot > of work on client side support. This part is newer, I grant you that. > > That it has not received as much review or experience as perhaps you would > like means mainly that other HBase committers have had other priorities. > That's true - I've been wanting to review it for months but haven't had time to do so. Apologies for that. I think, though, that for this kind of large new framework, we want to have some experience actually trying to use it - having at least one other person/company outside of TrendMicro use the framework to build a non-toy app is the requirement in my mind. I'm pretty sure I said exactly that on the very first coprocessor patch, and my opinion is the same- it's an awful lot of code that has the primary purpose of extensibility, and if we don't have some examples of real extensions, it's hard to know if it's all good. For example, I imagine the Lily people could make good use of this framework. Similarly, Sam Pullara's havrobase might provide some good use cases. The secondary indexing situation in HBase right now leaves a lot to be desired - maybe we could try implementing indexing based on CPs? > That does not mean we do not deserve due consideration. > > Any thoughts on releasing an 0.90-coprocessors-preview next to 0.90, off the branch (ie without merging)? I think if we did this, in conjunction with a few good blog posts on the hbase blog about example use cases, we could drum up some interest and get some feedback, and then merge for 0.92. -Todd > - Andy > > > From: Todd Lipcon <[email protected]> > > Subject: Re: dynamic RPC and coprocessor changes > > To: [email protected] > > Cc: "[email protected]" <[email protected]> > > Date: Tuesday, October 5, 2010, 7:57 PM > > I agree with Jon here. > > > > It's not a matter of viewing it as second-class, but more > > that it's not > > necessarily within scope for this exact release -- in > > general we have always > > talked about 0.90 as a release where we focus on stability > > and durability. A > > giant new framework (awesome though it is) seems less > > important for this > > specific dot release. I also agree that we need to have a > > few more use cases > > developed to flesh out the API and get more exposure before > > merging. > > > > The idea of the development release series was to give > > people exposure to > > stuff (and get testing) before it ended up in a so-called > > "stable" release. > > A large merge should definitely go through at least one or > > two dev releases > > before showing up in an even-numbered .0. I would also > > support a > > "coprocessors preview" release done in parallel with 0.90, > > available on the > > Apache download page, that would let interested people get > > involved and try > > the branch before it's part of trunk. > > > > For the same reasons, I also wanted to release > > 0.89.20100924 as 0.90 and > > push off the new master to 92, but I think I lost that one > > :) > > > > I also agree strongly with Jon that we should immediately > > start an 0.91 DR > > series with coprocessors merged as soon as 0.90 is > > released. We have some > > good momentum, and moving quickly towards the next bit will > > help keep it up. > > > > -Todd > > > > On Tue, Oct 5, 2010 at 7:30 PM, Jonathan Gray <[email protected]> > > wrote: > > > > > Committing this stuff onto trunk the day before a > > feature freeze and as we > > > move towards stability gives me a little bit of the > > fear. > > > > > > I completely understand that it's been worked on for > > quite some time and > > > think this stuff is awesome. But I still wish I > > had more time to give it > > > further review and to actually take this stuff for a > > test spin. Right now > > > that's just not possible with all the work to be done > > stabilizing and > > > testing trunk (and I think time is best spent right > > now stabilizing the > > > master/rs versus new features). > > > > > > Personally, I have a bunch of fairly sweeping > > compaction, flush, and split > > > changes that have been partially reviewed but I > > haven't had the time to push > > > them over the finish line. I hope to get them in > > immediately after an 0.90 > > > branch is cut and trunk becomes 0.92. This > > should be rather soon. I'd like > > > to see 0.92 get released in short order after 0.90 as > > there are lots of > > > changes that just have not made it into 0.90 > > yet. So even if we waited to > > > commit, we could work hard towards the next major > > release (I don't think > > > there are any major rewrites planned) :) > > > > > > With this kind of big change and addition of new APIs, > > my preference would > > > be to have it committed early in a trunk cycle (versus > > immediately before a > > > release is cut). That way, we can play with it > > ourselves, see how it works, > > > try out some of our own ideas for usage of CPs, etc... > > In doing that, I > > > would expect there would be suggestions for changes in > > the API, additions to > > > the API, naming changes to be more clear from a > > non-implementer POV, etc... > > > > > > By putting it in immediately before a release, the > > original API can't be > > > broken, class names can't be changed, all would need > > to be deprecated, > > > etc... > > > > > > The other issue I have with the current CP code that I > > have read is the > > > number of new classes introduced into existing core > > packages (especially > > > clent). Looking at some of their names, they are > > generic sounding and it's > > > not clear to me without digging in that they are part > > of coprocessors. Is > > > it possible to move CP stuff into client.cp packages > > and the like? Or > > > prefix with CoProc or some way to keep this stuff > > separated? > > > > > > I apologize for dropping this on you now and if no one > > else feels the same, > > > I am not going to spoil the party. > > > > > > But is it critical that this gets into 0.90 and into a > > release now? Could > > > we wait for 0.92 and get this committed in a couple > > weeks and allowed to > > > mature a bit on trunk before it's released? > > > > > > JG > > > > > > > -----Original Message----- > > > > From: Andrew Purtell [mailto:[email protected]] > > > > Sent: Tuesday, October 05, 2010 7:14 PM > > > > To: [email protected] > > > > Subject: dynamic RPC and coprocessor changes > > > > > > > > We've been nursing dynamic RPC and coprocessor > > changes for a couple of > > > > weeks now and don't see any failures beyond what > > are already on trunk > > > > with them applied. > > > > > > > > I was going to apply them tonight but will wait > > until tomorrow given > > > > that trunk is already disrupted a little. > > > > > > > > Best regards, > > > > > > > > - Andy > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > > > > > > > -- Todd Lipcon Software Engineer, Cloudera
