This is not at all about seeing CP as "second class". As you know, I'm extremely excited about them and I think it's absolutely awesome how much work you guys have been putting in to them. I have a bunch of plans for what I'd like to try to get done utilizing coprocessors.
My issues are about the timing and the nature of coprocessors. I'd like to see this done in the best way possible. IMO two weeks is not that much time to have something like this up for review, because it's not just about the code review (though I think that's important and there's been great back-and-forth). If we were at the start of a new trunk/release cycle, I would actually be fine with committing it this week. I (and others) would have time to play with it over the coming weeks and changes could still be made as it is out in the wild for a reasonable amount of time. I just don't think this is the kind of thing that gets nailed with two weeks of code reviews. It needs to be used, exercised, and different use cases need to be applied to it. For example, the initial CP implementation underwent a huge amount of change once you guys started to make security work with it. That's just one use case. Others will push it in new directions and will have different requirements. Of course it can still change after the initial release, but it gets much harder to make big changes and you have to worry about breaking compatibility. We already dealt with this with filters and those are far simpler than coprocessors. And I disagree strongly that we are going to continue to do yearly releases. Our previous release rewrote the RS in a major way, changed our on-disk format, etc. This release introduces appends/no data loss for the first time and the master has been completely redesigned. And we've only just-now broken from the hadoop release cycle with 0.90. The 0.92 release does not currently have any big sweeping changes like that planned. It will/should be a feature release which adds a lot of the stuff we've all been working on in recent months but have not yet finished or polished, plus another month or so of additional work. I'd like to push to get it out just a couple months after the 0.90 release. JG > -----Original Message----- > From: Andrew Purtell [mailto:[email protected]] > Sent: Tuesday, October 05, 2010 7:39 PM > To: [email protected] > Subject: RE: dynamic RPC and coprocessor changes > > We have been working on this for a while. It represents a significant > investment. I see no reason why this work should be viewed as second > class compared with some of the other work that has already gone in for > 0.90. The patches have been up on reviewboard for over two weeks now, > ample time for review. > > > - Andy > > > > From: Jonathan Gray <[email protected]> > > Subject: RE: dynamic RPC and coprocessor changes > > To: "[email protected]" <[email protected]>, > "[email protected]" <[email protected]> > > Date: Tuesday, October 5, 2010, 7:30 PM > > Committing this stuff onto trunk the > > day before a feature freeze and as we move towards stability > > gives me a little bit of the fear. > > > > I completely understand that it's been worked on for quite > > some time and think this stuff is awesome. But I still > > wish I had more time to give it further review and to > > actually take this stuff for a test spin. Right now > > that's just not possible with all the work to be done > > stabilizing and testing trunk (and I think time is best > > spent right now stabilizing the master/rs versus new > > features). > > > > Personally, I have a bunch of fairly sweeping compaction, > > flush, and split changes that have been partially reviewed > > but I haven't had the time to push them over the finish > > line. I hope to get them in immediately after an 0.90 > > branch is cut and trunk becomes 0.92. This should be > > rather soon. I'd like to see 0.92 get released in > > short order after 0.90 as there are lots of changes that > > just have not made it into 0.90 yet. So even if we > > waited to commit, we could work hard towards the next major > > release (I don't think there are any major rewrites > > planned) :) > > > > With this kind of big change and addition of new APIs, my > > preference would be to have it committed early in a trunk > > cycle (versus immediately before a release is cut). > > That way, we can play with it ourselves, see how it works, > > try out some of our own ideas for usage of CPs, etc... In > > doing that, I would expect there would be suggestions for > > changes in the API, additions to the API, naming changes to > > be more clear from a non-implementer POV, etc... > > > > By putting it in immediately before a release, the original > > API can't be broken, class names can't be changed, all would > > need to be deprecated, etc... > > > > The other issue I have with the current CP code that I have > > read is the number of new classes introduced into existing > > core packages (especially clent). Looking at some of > > their names, they are generic sounding and it's not clear to > > me without digging in that they are part of > > coprocessors. Is it possible to move CP stuff into > > client.cp packages and the like? Or prefix with CoProc > > or some way to keep this stuff separated? > > > > I apologize for dropping this on you now and if no one else > > feels the same, I am not going to spoil the party. > > > > But is it critical that this gets into 0.90 and into a > > release now? Could we wait for 0.92 and get this > > committed in a couple weeks and allowed to mature a bit on > > trunk before it's released? > > > > JG > > > > > -----Original Message----- > > > From: Andrew Purtell [mailto:[email protected]] > > > Sent: Tuesday, October 05, 2010 7:14 PM > > > To: [email protected] > > > Subject: dynamic RPC and coprocessor changes > > > > > > We've been nursing dynamic RPC and coprocessor changes > > for a couple of > > > weeks now and don't see any failures beyond what are > > already on trunk > > > with them applied. > > > > > > I was going to apply them tonight but will wait until > > tomorrow given > > > that trunk is already disrupted a little. > > > > > > Best regards, > > > > > > - Andy > > > > > > > > > > > > > > > > > > >
