On Thu, Jan 28, 2010 at 3:08 PM, Joydeep Sarma <jsensa...@gmail.com> wrote: > fwiw - i agree with Dhruba's take. i can't imagine (for example) > hadoop shipping without hadoop-streaming (which is contrib). and it's > awesome that hadoop changes that break streaming are caught early and > often.
I suggest that contribs we deem critical should be pulled up out of contrib and moved up to core. Regards changes in core breaking contrib early, contribs moved out to other repositories can retain this 'benefit' by building against snapshots of the core TRUNK via built-in ivy/maven facility, only now, the onus for updating the moved contrib is no longer inline with core dev and instead has been shifted to the moved contrib maintainer. It is up to him or her to update to match core changes at their pleasure or to report on core changes that cause fundamental distress in contrib. > it makes sense this analog doesn't apply to all contrib projects - > perhaps they shouldn't be in contrib to begin with? Yes. There is some of this going on. IMO, all currently have sufficient character living apart outside contrib. alternatively - to > extend Dhruba's idea - one could break up contrib projects into those > that are critical (like streaming would be to hadoop and i imagine the > replication stuff would be to hbase) - and those that are non-critical > (and not build these by default). > We could do this but to my mind, I can't help but think that we are just piling the work integrating contribs to the period that just precedes a release. > the only reason to have non-critical contrib projects in the codebase > seems to be to give young/new projects exposure. i am not sure whether > this is a good enough reason - perhaps those with some historical > perspective have better idea on this. > You have it right. I think it fair to say that that explains how we ended up with our current set of contribs. At least one contrib has been in hbase more than a year now. Its time we held a graduation for it at least? St.Ack > > On Thu, Jan 28, 2010 at 2:35 PM, Ryan Rawson <ryano...@gmail.com> wrote: >> Part of the problem with contrib is fixing the deeper problems during >> major version changes (eg: new file format in 0.20) requires >> specialized knowledge that only the contrib author has. Then they must >> write a patch, submit it to a hbase committer, etc. If the contrib >> was in a non-ASF repo, the author could fix it much faster. >> >> That's my theory at least? >> >> On Thu, Jan 28, 2010 at 2:27 PM, Dhruba Borthakur <dhr...@gmail.com> wrote: >>> Hi stack, >>> >>> I have seen this issue crop up in the HDFS repository too. >>> >>> The issue that you have brought up is that code changes to core sometimes >>> needs lots of changes to contrib to make it compile. This is a big problem, >>> especially when all developers are not intricately familiar with each and >>> every contrib module. However, the big plus is that when a HBase release is >>> made, it has all the contrib modules in it, and it makes it easier for a >>> user to start using any of those contrib modules right away. Can we get the >>> best of both worlds if we do something like this: >>> >>> Contribs are not first-class citizens. The build script can be setup to >>> ignore build failures in contrib modules. The hope is that when a release is >>> about to be cut, contrib owners will submit fixes to any build problems if >>> they want their contrib to be shipped as part of the Hbase release. will >>> this solve your problem? >>> >>> thanks, >>> dhruba >>> >>> >>> >>> On Thu, Jan 28, 2010 at 12:42 PM, Ryan Rawson <ryano...@gmail.com> wrote: >>> >>>> +1 >>>> >>>> I think when we are ready, we should publish the replication as a >>>> contrib up on github. I really like github, its way better than any >>>> other system I've seen (in part due to git of course). >>>> >>>> I do like the notion of officially released contribs that are included >>>> at specific version numbers with the hbase release. Hbase core 0.21.0 >>>> with TH, ITH at version 0.1 or whatever. >>>> >>>> -ryan >>>> >>>> On Thu, Jan 28, 2010 at 12:38 PM, Stack <st...@duboce.net> wrote: >>>> > I'd like to start a discussion on moving src/contrib out of hbase. >>>> > Keep reading if you have an opinion. >>>> > >>>> > I'd like to suggest that we undo the notion of hbase contribs for the >>>> > following reasons: >>>> > >>>> > + In my experience, they present a friction on changes to core as any >>>> > significant core change tends to ripple down into the contribs whether >>>> > its code or infrastructure/build changes. Usually what happens then >>>> > is a non-expert in the contrib code is making edits -- often radical >>>> > -- to code they are not completely up on and are a little frustrated >>>> > that they have to do it. Bad. >>>> > + A few of our contribs are maintained by non-committers. This means >>>> > it takes a committers time getting in updates. The owner is at mercy >>>> > of the committer making wanted changes. The committer is consumed >>>> > reviewing and making update. This indirection hurts at both ends (We >>>> > could discuss making contrib owners committers on their contrib only >>>> > but that'd be a bit of bureaucratic nightmare and a burden on the >>>> > hadoop pmc to vote on granting access to a subprojects, contrib. Its >>>> > tough enough getting hadoop pmc to vote on hbase committers. There is >>>> > no precedent in other project, to my knowledge). >>>> > + Contribs and core evolve at different rates. They should not be >>>> > constrained by core release schedule (or the opposite, core should not >>>> > be held up because fixes in contrib are wanting). >>>> > >>>> > I suggest that current contribs be moved out of hbase up to standalone >>>> > github or google code projects (witness how hbql does it). Previous >>>> > to our move to Ivy (and possibly soon, Maven), asking contribs be >>>> > standalone was a pain as they'd have to check in hbase jars and all of >>>> > dependencies and then move these forward over time. Now that we are >>>> > Ivy-ized, contribs just need write a bit of ivy.xml and it'll take >>>> > care of pulling dependencies. >>>> > >>>> > I'd imagine support for contribs would go on as it does now with >>>> > queries up on hbase mailing list and help out on IRC. We'd give >>>> > contribs first-class billing up on home page. Popular contribs might >>>> > run their own mailing lists, etc... >>>> > >>>> > We should chat but some contribs should be pulled up into core. >>>> > Thrift was core. Talk was to move current thrift update out to >>>> > contrib. I think now it should just stay in core. Stargate perhaps >>>> > should come up into core? >>>> > >>>> > What do folks think? >>>> > St.Ack >>>> > >>>> > P.S. I've fostered the contrib notion in the past. I've since had a >>>> > change of heart. Please pardon my flip. >>>> > >>>> >>> >>> >>> >>> -- >>> Connect to me at http://www.facebook.com/dhruba >>> >> >
