On Thu, Feb 3, 2011 at 12:16 AM, Alan Gates <ga...@yahoo-inc.com> wrote: > Edward, > > I understand your concern with having a copy of the metastore code in Howl. > However, let's separate code from governance. The reason Howl has a copy > of Hive's metastore is not because we're proposing it for the Incubator, it > is because in the course of developing it over the last six months we've > found that Howl development needs to move much faster than Hive development > can. This is appropriate, since Hive is a mature product and has at least > one large customer that runs code in production very soon after it is > checked in. Thus the Hive community is rightly cautious about checking in > changes to the metastore. Howl, on the other hand, is new and innovating > quickly, so it likes to get things checked in quickly. Over the last six > months every patch Howl has made to the Hive metastore code has made it back > into Hive code. But it generally takes a few weeks or more to get in. > > Whether Howl is a Hive subproject or an Incubator project it faces the same > dilemma. The only other alternative that was suggested was to have Howl > extern the metastore code from Hive and keep its patches in its build and > apply them at build time. But this is very fragile, since any changes in > the Hive metastore code could invalidate all those patches. We know that > this is not sustainable in the long run, which is why the proposal calls out > the need to resolve this one way or another as the project matures. > > As far as reaching an end state where Hive and Howl are not compatible, we > would view that as a failure for Howl. The goal for Howl is to be a > metastore for Pig, MapReduce, and Hive, not just 2 out 3. So we have a > strong motivation to maintain that compatibility. > > In terms of governance, given that we have significant contributions coming > from members of the Pig team, the Hive team, and the core Hadoop team it > seemed that giving Howl its own space in the Incubator made more sense than > adding it as a subproject of any one of those teams. > > Alan. > > On Feb 2, 2011, at 3:11 PM, Edward Capriolo wrote: > >> On Wed, Feb 2, 2011 at 5:08 PM, Jeff Hammerbacher <ham...@cloudera.com> >> wrote: >>> >>> Awesome! Huge +1. >>> >>> On Wed, Feb 2, 2011 at 1:18 PM, Alan Gates <ga...@yahoo-inc.com> wrote: >>> >>>> Howl is a table management system built to provide metadata and storage >>>> management across data processing tools in Hadoop (Pig, Hive, MapReduce, >>>> ...). You can learn more details at http://wiki.apache.org/pig/Howl. >>>> For >>>> the last six months the code has been hosted at github. The Howl team >>>> would >>>> like to move the project into the Apache Incubator. You can see the >>>> proposal for the project at >>>> http://wiki.apache.org/incubator/HowlProposal. >>>> >>>> In order to be accepted as an Incubator project Howl needs a Sponsoring >>>> project. I propose that we, the Pig project, sponsor Howl. By >>>> sponsoring >>>> Howl we are saying that we believe it is a good fit for the ASF and that >>>> we >>>> will assist the Howl project to succeed. You can read full details of >>>> sponsoring a project at >>>> >>>> http://incubator.apache.org/incubation/Roles_and_Responsibilities.html#Sponsor >>>> . >>>> >>>> Our bylaws don't explicitly cover such a vote, but I think lazy majority >>>> should be reasonable. All votes are welcome, PMC member votes will be >>>> binding. >>>> >>>> Clearly I'm +1. >>>> >>>> Alan. >>>> >>> >> >> I do think it is a great idea that hive/pig/ and map reduce share a >> meta store. However I am not sure I agree with the approach. IMHO Howl >> should be a hive sub project. >> >> "The initial release of Howl will allow interoperability of data >> between Pig, Map Reduce, and Hive" >> I believe the "The initial release of Howl should support hive" >> at this point hive should remove the /metastore code from inside hive >> and depend on howl. >> >> I say this because hive is very actively reworking the metastore right >> now for security, a new type of views, and indexes. I feel if the >> metastore branches from the hive as howl getting the two entities back >> together will be difficult. Having 99% of the same code base shared >> between hive and howl but not having compatibility between the two is >> my fear. > >
Alan, I see your points. I agree with you and I am +1. (incubator/subproject is not important to me) You mentioned that hive is cautious about checking changes into the meta-store. I would not say we (hive) are cautious. Hive is getting pulled in many people in many directions (this is a good thing). But the number of people that can technically review patches might be burdened at times by the number of them. Ideally, I would think hive committers are going to be active (and probably would have commit) on howl or is it going to be the burden of howl track pig and hive until hive drops /metastore and begins using howl? I am just curious about what you think the time line looks like (IE how long howl will be in the incubator for) (rought guess of course) Thank you, Edward
