For what it's worth, the Result changes are just a few functions and represent a big bang for the buck in terms of lines of code added. I fixed the build - sorry about that!
>From what I've seen, the coprocessor stuff interacts but isnt as crazy as other things. I'd say go for it. We only get 1 chance a year to release apparently. -ryan On Tue, Oct 5, 2010 at 8:49 PM, Todd Lipcon <[email protected]> wrote: > On Tue, Oct 5, 2010 at 8:17 PM, Andrew Purtell <[email protected]> wrote: > >> > It's not a matter of viewing it as second-class, but more >> > that it's not necessarily within scope for this exact >> > release -- in general we have always talked about 0.90 as >> > a release where we focus on stability and durability. >> >> What of replication? >> >> Or HFOF? >> >> > Both of these went into our very first 0.89 dev release this summer, and > have been tried out on clusters for multiple dev releases since. In > replication's case, it's only been used so far at StumbleUpon AFAIK, but the > new HFOF support has been used in real workloads at at least 5 companies off > the top of my head, three of which do not employ any contributors. Like i > said above, I think new APIs and new big features should go in early in a DR > series, not right before the release itself. > > If we could hold up the 0.90 release for enough time to do a dev release or > two with coprocessors, I'd be all for it. But given the aims of the release, > I don't think coprocessors are a blocking feature. > > >> Or the recent Result changes? >> >> > Honestly I've been too swamped to look at these, but if it's a substantial > new API I also think it should be held for 0.92 as well. If it's a few new > function calls to existing classes, it seems OK to put it in late. > > >> None are in scope for a stability and durability release. >> >> HBASE-2001 has been little changed for 6 months. We've made incremental >> changes to the server side framework but it's been basically as stable as >> possible given the changes around it. Then, Gary added a big push with a lot >> of work on client side support. This part is newer, I grant you that. >> >> That it has not received as much review or experience as perhaps you would >> like means mainly that other HBase committers have had other priorities. >> > > That's true - I've been wanting to review it for months but haven't had time > to do so. Apologies for that. I think, though, that for this kind of large > new framework, we want to have some experience actually trying to use it - > having at least one other person/company outside of TrendMicro use the > framework to build a non-toy app is the requirement in my mind. I'm pretty > sure I said exactly that on the very first coprocessor patch, and my opinion > is the same- it's an awful lot of code that has the primary purpose of > extensibility, and if we don't have some examples of real extensions, it's > hard to know if it's all good. > > For example, I imagine the Lily people could make good use of this > framework. Similarly, Sam Pullara's havrobase might provide some good use > cases. The secondary indexing situation in HBase right now leaves a lot to > be desired - maybe we could try implementing indexing based on CPs? > > >> That does not mean we do not deserve due consideration. >> >> > Any thoughts on releasing an 0.90-coprocessors-preview next to 0.90, off the > branch (ie without merging)? I think if we did this, in conjunction with a > few good blog posts on the hbase blog about example use cases, we could drum > up some interest and get some feedback, and then merge for 0.92. > > -Todd > > >> - Andy >> >> > From: Todd Lipcon <[email protected]> >> > Subject: Re: dynamic RPC and coprocessor changes >> > To: [email protected] >> > Cc: "[email protected]" <[email protected]> >> > Date: Tuesday, October 5, 2010, 7:57 PM >> > I agree with Jon here. >> > >> > It's not a matter of viewing it as second-class, but more >> > that it's not >> > necessarily within scope for this exact release -- in >> > general we have always >> > talked about 0.90 as a release where we focus on stability >> > and durability. A >> > giant new framework (awesome though it is) seems less >> > important for this >> > specific dot release. I also agree that we need to have a >> > few more use cases >> > developed to flesh out the API and get more exposure before >> > merging. >> > >> > The idea of the development release series was to give >> > people exposure to >> > stuff (and get testing) before it ended up in a so-called >> > "stable" release. >> > A large merge should definitely go through at least one or >> > two dev releases >> > before showing up in an even-numbered .0. I would also >> > support a >> > "coprocessors preview" release done in parallel with 0.90, >> > available on the >> > Apache download page, that would let interested people get >> > involved and try >> > the branch before it's part of trunk. >> > >> > For the same reasons, I also wanted to release >> > 0.89.20100924 as 0.90 and >> > push off the new master to 92, but I think I lost that one >> > :) >> > >> > I also agree strongly with Jon that we should immediately >> > start an 0.91 DR >> > series with coprocessors merged as soon as 0.90 is >> > released. We have some >> > good momentum, and moving quickly towards the next bit will >> > help keep it up. >> > >> > -Todd >> > >> > On Tue, Oct 5, 2010 at 7:30 PM, Jonathan Gray <[email protected]> >> > wrote: >> > >> > > Committing this stuff onto trunk the day before a >> > feature freeze and as we >> > > move towards stability gives me a little bit of the >> > fear. >> > > >> > > I completely understand that it's been worked on for >> > quite some time and >> > > think this stuff is awesome. But I still wish I >> > had more time to give it >> > > further review and to actually take this stuff for a >> > test spin. Right now >> > > that's just not possible with all the work to be done >> > stabilizing and >> > > testing trunk (and I think time is best spent right >> > now stabilizing the >> > > master/rs versus new features). >> > > >> > > Personally, I have a bunch of fairly sweeping >> > compaction, flush, and split >> > > changes that have been partially reviewed but I >> > haven't had the time to push >> > > them over the finish line. I hope to get them in >> > immediately after an 0.90 >> > > branch is cut and trunk becomes 0.92. This >> > should be rather soon. I'd like >> > > to see 0.92 get released in short order after 0.90 as >> > there are lots of >> > > changes that just have not made it into 0.90 >> > yet. So even if we waited to >> > > commit, we could work hard towards the next major >> > release (I don't think >> > > there are any major rewrites planned) :) >> > > >> > > With this kind of big change and addition of new APIs, >> > my preference would >> > > be to have it committed early in a trunk cycle (versus >> > immediately before a >> > > release is cut). That way, we can play with it >> > ourselves, see how it works, >> > > try out some of our own ideas for usage of CPs, etc... >> > In doing that, I >> > > would expect there would be suggestions for changes in >> > the API, additions to >> > > the API, naming changes to be more clear from a >> > non-implementer POV, etc... >> > > >> > > By putting it in immediately before a release, the >> > original API can't be >> > > broken, class names can't be changed, all would need >> > to be deprecated, >> > > etc... >> > > >> > > The other issue I have with the current CP code that I >> > have read is the >> > > number of new classes introduced into existing core >> > packages (especially >> > > clent). Looking at some of their names, they are >> > generic sounding and it's >> > > not clear to me without digging in that they are part >> > of coprocessors. Is >> > > it possible to move CP stuff into client.cp packages >> > and the like? Or >> > > prefix with CoProc or some way to keep this stuff >> > separated? >> > > >> > > I apologize for dropping this on you now and if no one >> > else feels the same, >> > > I am not going to spoil the party. >> > > >> > > But is it critical that this gets into 0.90 and into a >> > release now? Could >> > > we wait for 0.92 and get this committed in a couple >> > weeks and allowed to >> > > mature a bit on trunk before it's released? >> > > >> > > JG >> > > >> > > > -----Original Message----- >> > > > From: Andrew Purtell [mailto:[email protected]] >> > > > Sent: Tuesday, October 05, 2010 7:14 PM >> > > > To: [email protected] >> > > > Subject: dynamic RPC and coprocessor changes >> > > > >> > > > We've been nursing dynamic RPC and coprocessor >> > changes for a couple of >> > > > weeks now and don't see any failures beyond what >> > are already on trunk >> > > > with them applied. >> > > > >> > > > I was going to apply them tonight but will wait >> > until tomorrow given >> > > > that trunk is already disrupted a little. >> > > > >> > > > Best regards, >> > > > >> > > > - Andy >> > > > >> > > > >> > > > >> > > > >> > > >> > > >> > >> > >> > -- >> > Todd Lipcon >> > Software Engineer, Cloudera >> > >> >> >> >> >> > > > -- > Todd Lipcon > Software Engineer, Cloudera >
