Re: [DISCUSS] HCatalog becoming a subproject of Hive
Changes made. Alan. On Jan 21, 2013, at 12:39 PM, Carl Steinbach wrote: > Hi Alan, > > Overall this looks good to me. I have a couple small suggestions: > > * Replace occurrences of "Hive's subversion repository" with "Hive's > source code repository". > * In the "Actions" table the sentence "This also covers the creation of new > sub-projects within the project" should be changed to "This also covers the > creation of new sub-projects and sub-modules within the project." > > Thanks. > > Carl > > > On Fri, Jan 18, 2013 at 4:42 PM, Alan Gates wrote: > >> I've created a wiki page for my proposed changes at >> https://cwiki.apache.org/confluence/display/Hive/Proposed+Changes+to+Hive+Bylaws+for+Submodule+Committers >> >> Text to be removed is struck through. Text to be added is in italics. >> >> Any recommended changes before we vote? >> >> Alan. >> >> >> On Jan 17, 2013, at 2:08 PM, Carl Steinbach wrote: >> >>> Sounds like a good plan to me. Since Ashutosh is a member of both the >> Hive >>> and HCatalog PMCs it probably makes more sense for him to call the vote, >>> but I'm willing to do it too. >>> >>> On Wed, Jan 16, 2013 at 8:24 AM, Alan Gates >> wrote: >>> If you think that's the best path forward that's fine. I can't call a vote I don't think, since I'm not part of the Hive PMC. But I'm happy >> to draft a resolution for you and then let you call the vote. Should I do that? Alan. On Jan 11, 2013, at 4:34 PM, Carl Steinbach wrote: > Hi Alan, > > I agree that submitting this for a vote is the best option. > > If anyone has additional proposed modifications please make them. Otherwise I propose that the Hive PMC vote on this proposal. > > In order for the Hive PMC to be able to vote on these changes they need to be expressed in terms of one or more of the "actions" listed at the >> end of the Hive project bylaws: > > https://cwiki.apache.org/confluence/display/Hive/Bylaws > > So I think we first need to amend to the bylaws in order to define the rights and privileges of a submodule committer, and then separately vote the HCatalog committers in as Hive submodule committers. Does this make sense? > > Thanks. > > Carl > >> >>
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Hi Alan, Overall this looks good to me. I have a couple small suggestions: * Replace occurrences of "Hive's subversion repository" with "Hive's source code repository". * In the "Actions" table the sentence "This also covers the creation of new sub-projects within the project" should be changed to "This also covers the creation of new sub-projects and sub-modules within the project." Thanks. Carl On Fri, Jan 18, 2013 at 4:42 PM, Alan Gates wrote: > I've created a wiki page for my proposed changes at > https://cwiki.apache.org/confluence/display/Hive/Proposed+Changes+to+Hive+Bylaws+for+Submodule+Committers > > Text to be removed is struck through. Text to be added is in italics. > > Any recommended changes before we vote? > > Alan. > > > On Jan 17, 2013, at 2:08 PM, Carl Steinbach wrote: > > > Sounds like a good plan to me. Since Ashutosh is a member of both the > Hive > > and HCatalog PMCs it probably makes more sense for him to call the vote, > > but I'm willing to do it too. > > > > On Wed, Jan 16, 2013 at 8:24 AM, Alan Gates > wrote: > > > >> If you think that's the best path forward that's fine. I can't call a > >> vote I don't think, since I'm not part of the Hive PMC. But I'm happy > to > >> draft a resolution for you and then let you call the vote. Should I do > >> that? > >> > >> Alan. > >> > >> On Jan 11, 2013, at 4:34 PM, Carl Steinbach wrote: > >> > >>> Hi Alan, > >>> > >>> I agree that submitting this for a vote is the best option. > >>> > >>> If anyone has additional proposed modifications please make them. > >> Otherwise I propose that the Hive PMC vote on this proposal. > >>> > >>> In order for the Hive PMC to be able to vote on these changes they need > >> to be expressed in terms of one or more of the "actions" listed at the > end > >> of the Hive project bylaws: > >>> > >>> https://cwiki.apache.org/confluence/display/Hive/Bylaws > >>> > >>> So I think we first need to amend to the bylaws in order to define the > >> rights and privileges of a submodule committer, and then separately vote > >> the HCatalog committers in as Hive submodule committers. Does this make > >> sense? > >>> > >>> Thanks. > >>> > >>> Carl > >>> > >> > >> > >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I've created a wiki page for my proposed changes at https://cwiki.apache.org/confluence/display/Hive/Proposed+Changes+to+Hive+Bylaws+for+Submodule+Committers Text to be removed is struck through. Text to be added is in italics. Any recommended changes before we vote? Alan. On Jan 17, 2013, at 2:08 PM, Carl Steinbach wrote: > Sounds like a good plan to me. Since Ashutosh is a member of both the Hive > and HCatalog PMCs it probably makes more sense for him to call the vote, > but I'm willing to do it too. > > On Wed, Jan 16, 2013 at 8:24 AM, Alan Gates wrote: > >> If you think that's the best path forward that's fine. I can't call a >> vote I don't think, since I'm not part of the Hive PMC. But I'm happy to >> draft a resolution for you and then let you call the vote. Should I do >> that? >> >> Alan. >> >> On Jan 11, 2013, at 4:34 PM, Carl Steinbach wrote: >> >>> Hi Alan, >>> >>> I agree that submitting this for a vote is the best option. >>> >>> If anyone has additional proposed modifications please make them. >> Otherwise I propose that the Hive PMC vote on this proposal. >>> >>> In order for the Hive PMC to be able to vote on these changes they need >> to be expressed in terms of one or more of the "actions" listed at the end >> of the Hive project bylaws: >>> >>> https://cwiki.apache.org/confluence/display/Hive/Bylaws >>> >>> So I think we first need to amend to the bylaws in order to define the >> rights and privileges of a submodule committer, and then separately vote >> the HCatalog committers in as Hive submodule committers. Does this make >> sense? >>> >>> Thanks. >>> >>> Carl >>> >> >>
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Sounds like a good plan to me. Since Ashutosh is a member of both the Hive and HCatalog PMCs it probably makes more sense for him to call the vote, but I'm willing to do it too. On Wed, Jan 16, 2013 at 8:24 AM, Alan Gates wrote: > If you think that's the best path forward that's fine. I can't call a > vote I don't think, since I'm not part of the Hive PMC. But I'm happy to > draft a resolution for you and then let you call the vote. Should I do > that? > > Alan. > > On Jan 11, 2013, at 4:34 PM, Carl Steinbach wrote: > > > Hi Alan, > > > > I agree that submitting this for a vote is the best option. > > > > If anyone has additional proposed modifications please make them. > Otherwise I propose that the Hive PMC vote on this proposal. > > > > In order for the Hive PMC to be able to vote on these changes they need > to be expressed in terms of one or more of the "actions" listed at the end > of the Hive project bylaws: > > > > https://cwiki.apache.org/confluence/display/Hive/Bylaws > > > > So I think we first need to amend to the bylaws in order to define the > rights and privileges of a submodule committer, and then separately vote > the HCatalog committers in as Hive submodule committers. Does this make > sense? > > > > Thanks. > > > > Carl > > > >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
If you think that's the best path forward that's fine. I can't call a vote I don't think, since I'm not part of the Hive PMC. But I'm happy to draft a resolution for you and then let you call the vote. Should I do that? Alan. On Jan 11, 2013, at 4:34 PM, Carl Steinbach wrote: > Hi Alan, > > I agree that submitting this for a vote is the best option. > > If anyone has additional proposed modifications please make them. Otherwise > I propose that the Hive PMC vote on this proposal. > > In order for the Hive PMC to be able to vote on these changes they need to be > expressed in terms of one or more of the "actions" listed at the end of the > Hive project bylaws: > > https://cwiki.apache.org/confluence/display/Hive/Bylaws > > So I think we first need to amend to the bylaws in order to define the rights > and privileges of a submodule committer, and then separately vote the > HCatalog committers in as Hive submodule committers. Does this make sense? > > Thanks. > > Carl >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Hi Alan, I agree that submitting this for a vote is the best option. > If anyone has additional proposed modifications please make them. > Otherwise I propose that the Hive PMC vote on this proposal. > In order for the Hive PMC to be able to vote on these changes they need to be expressed in terms of one or more of the "actions" listed at the end of the Hive project bylaws: https://cwiki.apache.org/confluence/display/Hive/Bylaws So I think we first need to amend to the bylaws in order to define the rights and privileges of a submodule committer, and then separately vote the HCatalog committers in as Hive submodule committers. Does this make sense? Thanks. Carl
Re: [DISCUSS] HCatalog becoming a subproject of Hive
So what is the next step here? As far as I can tell all agree that moving HCat into Hive is the right thing to do, but there is a fairly polarized division on how to handle the commit rights. Given that I re-propose my compromise. Like all good compromises most everyone hates it. But I think it gives us a path forward to what we agree is a common goal. Here is the slightly modified version of my proposed compromise, which makes clear that promotion from HCat committer to Hive committer is not automatic. >>> All active HCat committers (those who have contributed or committed a patch in the last 6 months) will be made committers in the HCat portion only of Hive. In addition those committers will be assigned a particular shepherd who is a current Hive committer and who will be responsible for mentoring them towards full Hive committership. As a part of this mentorship the HCat committer will review patches of other contributors, contribute patches to Hive (both inside and outside of HCatalog), respond to user issues on the mailing lists, etc. It is intended that as a result of this mentorship program HCat committers can become full Hive committers in 6-9 months. Becoming a full Hive committer in this time frame is not guaranteed. It will require the vote of the Hive PMC to move the HCat committer to a full Hive committer. No new HCat only committers will be elected in Hive after this. All Hive committers will automatically also have commit rights on HCatalog. <<< If anyone has additional proposed modifications please make them. Otherwise I propose that the Hive PMC vote on this proposal. Alan. On Dec 20, 2012, at 3:53 PM, Ashish Thusoo wrote: > hmm... why is this considered "preferential treatment"? > > All the work for HCat is in the public domain so we can really evaluate > whether they have been following apache practices - the fact that they are > graduating from the incubator would seem to indicate that they have been > doing so. If this code base is contributed back to Hive, is that not counted > as a significant contribution to Hive? I am failing to understand on what > count they don't qualify to be committers. > > Plus if it is too onerous to enforce committer privileges on selective parts > (is there a way?) of the project, then what do terms like Hive committer, > HCat committer mean? Also should Hive committers have privileges to commit > into HCat part of the code once it becomes a subproject. I think we are just > creating walls and the problem with walls is that they just impede cross > pollination and community expansion. > > Ashish > > > On Thu, Dec 20, 2012 at 1:59 PM, Carl Steinbach wrote: > I agree with Namit on this issue. I don't think it's fair to the > existing group of Hive contributors to give preferential > treatment to HCat committers, or to automatically promote them to > full committer status on the Hive project. > > On Thu, Dec 20, 2012 at 1:10 PM, Bhandarkar, Milind < > milind.bhandar...@emc.com> wrote: > > > I agree with Ashish. > > > > When Hcat becomes a subproject of Hive, all Hcat committers should > > immediately become Hive committers. > > > > After all, that worked well for Hadoop, where all Hadoop committers can > > commit to all Hadoop code (common/HDFS/MapReduce), but not all do, instead > > focusing only on their area of expertise, and familiarity with portions of > > codebase. > > > > - milind > > > > --- > > Milind Bhandarkar > > Chief Scientist, > > Machine Learning Platforms, > > Greenplum, A Division of EMC > > +1-650-523-3858 (W) > > +1-408-666-8483 (C) > > > > > > > > > > > > On 12/20/12 5:58 AM, "Ashish Thusoo" wrote: > > > > >Actually I don't understand why getting Hcat folks as committers on Hive > > >is > > >a problem. Hive itself became a subproject of Hadoop when it started with > > >all the Hive committers becoming Hadoop committers. And of course everyone > > >maintained the discipline that they commit in parts of the code that they > > >understand and that they have worked on. Some of the committers from Hive > > >ended up becoming Hadoop committers - others who worked only on Hive ended > > >up leaving the Hadoop committers list once Hive became a TLP. So why put > > >in > > >these arguments about process when the end result would be beneficial to > > >the community and to the project. Would Hive not benefit if some folks > > >from > > >Hcat start working on Hive proper as well - of course under the guidance > > >of > > >Hive mentors etc. Would the project not benefit in the long run if Hcat is > > >brought in and some day becomes the default metastore for Hive. I mean if > > >there are so many long term benefits from this then why focus on control > > >and code safety which I think any responsible committer knows how to > > >navigate and there are well understood best practices for that. And why > > >can't a committer be booted out if he/she is breaking the discipline and > > >really nosin
Re: [DISCUSS] HCatalog becoming a subproject of Hive
+1. Putting my marketing hat on (I don¹t think marketeers is a four-letter word), touting Hive metastore as a sufficient metadata solution for other platforms such as MR and Pig, is nothing to be ashamed of. - Milind --- Milind Bhandarkar Chief Scientist, Machine Learning Platforms, Greenplum, A Division of EMC +1-650-523-3858 (W) +1-408-666-8483 (C) On 12/20/12 3:56 PM, "Ashish Thusoo" wrote: >We are certainly not marketeers and no one as far as I know is teeing up >for such a campaign. The intent here is certainly not to claim how great >HCatalog is or how great Hive is. The intent here is to see what is best >for the project and how great both are together. > >Ashish > > >On Thu, Dec 20, 2012 at 2:02 PM, Carl Steinbach wrote: > >> > Would the project not benefit in the long run if Hcat is >> > brought in and some day becomes the default metastore for Hive. >> >> >> Folks, can we please try to keep this straight? HCatalog on its own does >> not provide any support for metadata. It is a set of wrapper APIs that >>make >> Hive's metastore and serdes accessible to Pig and MR. I think these >> wrappers provide a lot of value, and I'm eager to see them merged into >> Hive, but I'm dreading the marketing campaign that I suspect will >>follow: >> "Hive now supports metadata thanks to HCatalog!". >>
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I am really looking forward to the headline that "Hive Metastore now provides metadata for platforms other than Hive". I think that would demonstrate the flexibility of Hive metastore to the world. Even if some people consider this a purely "marketing message", the flexibility of the APIs do appeal to developers. Right ? - milind --- Milind Bhandarkar Chief Scientist, Machine Learning Platforms, Greenplum, A Division of EMC +1-650-523-3858 (W) +1-408-666-8483 (C) On 12/20/12 2:02 PM, "Carl Steinbach" wrote: >> Would the project not benefit in the long run if Hcat is >> brought in and some day becomes the default metastore for Hive. > > >Folks, can we please try to keep this straight? HCatalog on its own does >not provide any support for metadata. It is a set of wrapper APIs that >make >Hive's metastore and serdes accessible to Pig and MR. I think these >wrappers provide a lot of value, and I'm eager to see them merged into >Hive, but I'm dreading the marketing campaign that I suspect will follow: >"Hive now supports metadata thanks to HCatalog!".
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Carl, I think Ashish is trying to avoid exactly this issue of any preferential treatment to Hcat committers *becoming* Hive committers. If Hcatalog as a subproject of Hive is accepted, both of us are advocating Hcat committers becoming Hive committers immediately. If you read Ashish's comment, it would become clear that when Hive, Pig etc were subprojects of Hadoop, they had full Hadoop committership. But none of them committed to hadoop core. And that worked well. So, why change the rules now ? - Milind --- Milind Bhandarkar Chief Scientist, Machine Learning Platforms, Greenplum, A Division of EMC +1-650-523-3858 (W) +1-408-666-8483 (C) On 12/20/12 1:59 PM, "Carl Steinbach" wrote: >I agree with Namit on this issue. I don't think it's fair to the >existing group of Hive contributors to give preferential >treatment to HCat committers, or to automatically promote them to >full committer status on the Hive project. > >On Thu, Dec 20, 2012 at 1:10 PM, Bhandarkar, Milind < >milind.bhandar...@emc.com> wrote: > >> I agree with Ashish. >> >> When Hcat becomes a subproject of Hive, all Hcat committers should >> immediately become Hive committers. >> >> After all, that worked well for Hadoop, where all Hadoop committers can >> commit to all Hadoop code (common/HDFS/MapReduce), but not all do, >>instead >> focusing only on their area of expertise, and familiarity with portions >>of >> codebase. >> >> - milind >> >> --- >> Milind Bhandarkar >> Chief Scientist, >> Machine Learning Platforms, >> Greenplum, A Division of EMC >> +1-650-523-3858 (W) >> +1-408-666-8483 (C) >> >> >> >> >> >> On 12/20/12 5:58 AM, "Ashish Thusoo" wrote: >> >> >Actually I don't understand why getting Hcat folks as committers on >>Hive >> >is >> >a problem. Hive itself became a subproject of Hadoop when it started >>with >> >all the Hive committers becoming Hadoop committers. And of course >>everyone >> >maintained the discipline that they commit in parts of the code that >>they >> >understand and that they have worked on. Some of the committers from >>Hive >> >ended up becoming Hadoop committers - others who worked only on Hive >>ended >> >up leaving the Hadoop committers list once Hive became a TLP. So why >>put >> >in >> >these arguments about process when the end result would be beneficial >>to >> >the community and to the project. Would Hive not benefit if some folks >> >from >> >Hcat start working on Hive proper as well - of course under the >>guidance >> >of >> >Hive mentors etc. Would the project not benefit in the long run if >>Hcat is >> >brought in and some day becomes the default metastore for Hive. I mean >>if >> >there are so many long term benefits from this then why focus on >>control >> >and code safety which I think any responsible committer knows how to >> >navigate and there are well understood best practices for that. And why >> >can't a committer be booted out if he/she is breaking the discipline >>and >> >really nosing in places which he/she does not understand. >> > >> >I mean if we agree that directionally Hcat being a part of Hive makes >> >sense >> >then why don't we try to get rid of the procedural elements that would >> >only >> >slow down that transition? If there is angst about specific people on >>Hcat >> >committers list on the Hive committers side (are there any?), then I >>think >> >that should be addressed on a case by case basis but why enforce a >>general >> >rule. In the same vein why have a rule saying in 6-9 months a Hcat >> >committer becomes a Hive committer - how is that helpful? If they are >> >changing the Hcat subproject in Hive are they not already Hive >>committers? >> >And if they gain the expertise to review and commit code in the >> >SemanticAnalyzer in a few months should they not be able to do that >>before >> >9 months are over? And if they don't get that expertise in 9 months >>would >> >they really review and commit anything in the SemanticAnalyzer - I mean >> >there are Hive committers who don't touch that piece of code today. no? >> > >> >Ashish >> > >> > >> >On Wed, Dec 19, 2012 at 8:23 PM, Namit Jain wrote: >> > >> >> I don’t agree with the proposal. It is impractical to have a Hcat >> >>committer >> >> with commit access to Hcat only portions of Hive. We cannot guarantee >> >>that >> >> a Hcat >> >> committer will become a Hive committer in 6-9 months, that depends on >> >>what >> >> they do >> >> in the next 6-9 months. >> >> >> >> The current Hcat committers should spend more time in reviewing >>patches, >> >> work on non-Hcat areas in Hive, and then gradually become a hive >> >> committer. They should not be given any preferential treatment, and >>the >> >> process should be same as it would be for any other hive contributor >> >> currently. Given that the expertise of the Hcat committers, they >>should >> >> be inline for becoming a hive committer if they continue to work in >> >>hive, >> >> but that cannot be guaranteed. I agree that some Hive committers >>shoul
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Namit, I was not proposing that promotion to full committership would be automatic. I assume it would still be done via a vote by the PMC. I agree that we cannot _guarantee_ committership for HCat committers in 6-9 months. But I am trying to lay out a clear path they can follow. If they don't follow the path then they won't be committers. I am also trying to make it non-preferential in that I am setting the criteria to be what I believe the Hive PMC would expect any prospective Hive committer to do. The only intended preferential part of the proposal is the Hive shepherds, which we have all agreed is a good idea. Alan. On Dec 19, 2012, at 8:23 PM, Namit Jain wrote: > I don’t agree with the proposal. It is impractical to have a Hcat committer > with commit access to Hcat only portions of Hive. We cannot guarantee that > a Hcat > committer will become a Hive committer in 6-9 months, that depends on what > they do > in the next 6-9 months. > > The current Hcat committers should spend more time in reviewing patches, > work on non-Hcat areas in Hive, and then gradually become a hive > committer. They should not be given any preferential treatment, and the > process should be same as it would be for any other hive contributor > currently. Given that the expertise of the Hcat committers, they should > be inline for becoming a hive committer if they continue to work in hive, > but that cannot be guaranteed. I agree that some Hive committers should try > and help the existing Hcat patches, and again that is voluntary and > different > committers cannot be assigned to different parts of the code. > > Thanks, > -namit > > > > > > > > On 12/20/12 1:03 AM, "Carl Steinbach" wrote: > >> Alan's proposal sounds like a good idea to me. >> >> +1 >> >> On Dec 18, 2012 5:36 PM, "Travis Crawford" >> wrote: >> >>> Alan, I think your proposal sounds great. >>> >>> --travis >>> >>> On Tue, Dec 18, 2012 at 1:13 PM, Alan Gates >>> wrote: Carl, speaking just for myself and not as a representative of the HCat >>> PPMC at this point, I am coming to agree with you that HCat integrating >>> with Hive fully makes more sense. However, this makes the committer question even thornier. Travis and >>> Namit, I think the shepherd proposal needs to lay out a clear and time >>> bounded path to committership for HCat committers. Having HCat >>> committers >>> as second class Hive citizens for the long run will not be healthy. I >>> propose the following as a starting point for discussion: All active HCat committers (those who have contributed or committed a >>> patch in the last 6 months) will be made committers in the HCat portion >>> only of Hive. In addition those committers will be assigned a >>> particular >>> shepherd who is a current Hive committer and who will be responsible for >>> mentoring them towards full Hive committership. As a part of this >>> mentorship the HCat committer will review patches of other contributors, >>> contribute patches to Hive (both inside and outside of HCatalog), >>> respond >>> to user issues on the mailing lists, etc. It is intended that as a >>> result >>> of this mentorship program HCat committers can become full Hive >>> committers >>> in 6-9 months. No new HCat only committers will be elected in Hive >>> after >>> this. All Hive committers will automatically also have commit rights on >>> HCatalog. Alan. On Dec 14, 2012, at 10:05 AM, Carl Steinbach wrote: > On a functional level I don't think there is going to be much of a > difference between the subproject option proposed by Travis and the >>> other > option where HCatalog becomes a TLP. In both cases HCatalog and Hive >>> will > have separate committers, separate code repositories, separate >>> release > cycles, and separate project roadmaps. Aside from ASF bureaucracy, I >>> think > the only major difference between the two options is that the >>> subproject > route will give the rest of the community the false impression that >>> the >>> two > projects have coordinated roadmaps and a process to prevent >>> overlapping > functionality from appearing in both projects. Consequently, If these >>> are > the only two options then I would prefer that HCatalog become a TLP. > > On the other hand, I also agree with many of the sentiments that have > already been expressed in this thread, namely that the two projects >>> are > closely related and that it would benefit the community at large if >>> the >>> two > projects could be brought closer together. Up to this point the major > source of pain for the HCatalog team has been the frequent necessity >>> of > making changes on both the Hive and HCatalog sides when implementing >>> new > features in HCatalog. This situation is compounded by the ASF >>> requirement > that release artifacts may not depend on snapshot artifacts from >>> other
Re: [DISCUSS] HCatalog becoming a subproject of Hive
We are certainly not marketeers and no one as far as I know is teeing up for such a campaign. The intent here is certainly not to claim how great HCatalog is or how great Hive is. The intent here is to see what is best for the project and how great both are together. Ashish On Thu, Dec 20, 2012 at 2:02 PM, Carl Steinbach wrote: > > Would the project not benefit in the long run if Hcat is > > brought in and some day becomes the default metastore for Hive. > > > Folks, can we please try to keep this straight? HCatalog on its own does > not provide any support for metadata. It is a set of wrapper APIs that make > Hive's metastore and serdes accessible to Pig and MR. I think these > wrappers provide a lot of value, and I'm eager to see them merged into > Hive, but I'm dreading the marketing campaign that I suspect will follow: > "Hive now supports metadata thanks to HCatalog!". >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
hmm... why is this considered "preferential treatment"? All the work for HCat is in the public domain so we can really evaluate whether they have been following apache practices - the fact that they are graduating from the incubator would seem to indicate that they have been doing so. If this code base is contributed back to Hive, is that not counted as a significant contribution to Hive? I am failing to understand on what count they don't qualify to be committers. Plus if it is too onerous to enforce committer privileges on selective parts (is there a way?) of the project, then what do terms like Hive committer, HCat committer mean? Also should Hive committers have privileges to commit into HCat part of the code once it becomes a subproject. I think we are just creating walls and the problem with walls is that they just impede cross pollination and community expansion. Ashish On Thu, Dec 20, 2012 at 1:59 PM, Carl Steinbach wrote: > I agree with Namit on this issue. I don't think it's fair to the > existing group of Hive contributors to give preferential > treatment to HCat committers, or to automatically promote them to > full committer status on the Hive project. > > On Thu, Dec 20, 2012 at 1:10 PM, Bhandarkar, Milind < > milind.bhandar...@emc.com> wrote: > > > I agree with Ashish. > > > > When Hcat becomes a subproject of Hive, all Hcat committers should > > immediately become Hive committers. > > > > After all, that worked well for Hadoop, where all Hadoop committers can > > commit to all Hadoop code (common/HDFS/MapReduce), but not all do, > instead > > focusing only on their area of expertise, and familiarity with portions > of > > codebase. > > > > - milind > > > > --- > > Milind Bhandarkar > > Chief Scientist, > > Machine Learning Platforms, > > Greenplum, A Division of EMC > > +1-650-523-3858 (W) > > +1-408-666-8483 (C) > > > > > > > > > > > > On 12/20/12 5:58 AM, "Ashish Thusoo" wrote: > > > > >Actually I don't understand why getting Hcat folks as committers on Hive > > >is > > >a problem. Hive itself became a subproject of Hadoop when it started > with > > >all the Hive committers becoming Hadoop committers. And of course > everyone > > >maintained the discipline that they commit in parts of the code that > they > > >understand and that they have worked on. Some of the committers from > Hive > > >ended up becoming Hadoop committers - others who worked only on Hive > ended > > >up leaving the Hadoop committers list once Hive became a TLP. So why put > > >in > > >these arguments about process when the end result would be beneficial to > > >the community and to the project. Would Hive not benefit if some folks > > >from > > >Hcat start working on Hive proper as well - of course under the guidance > > >of > > >Hive mentors etc. Would the project not benefit in the long run if Hcat > is > > >brought in and some day becomes the default metastore for Hive. I mean > if > > >there are so many long term benefits from this then why focus on control > > >and code safety which I think any responsible committer knows how to > > >navigate and there are well understood best practices for that. And why > > >can't a committer be booted out if he/she is breaking the discipline and > > >really nosing in places which he/she does not understand. > > > > > >I mean if we agree that directionally Hcat being a part of Hive makes > > >sense > > >then why don't we try to get rid of the procedural elements that would > > >only > > >slow down that transition? If there is angst about specific people on > Hcat > > >committers list on the Hive committers side (are there any?), then I > think > > >that should be addressed on a case by case basis but why enforce a > general > > >rule. In the same vein why have a rule saying in 6-9 months a Hcat > > >committer becomes a Hive committer - how is that helpful? If they are > > >changing the Hcat subproject in Hive are they not already Hive > committers? > > >And if they gain the expertise to review and commit code in the > > >SemanticAnalyzer in a few months should they not be able to do that > before > > >9 months are over? And if they don't get that expertise in 9 months > would > > >they really review and commit anything in the SemanticAnalyzer - I mean > > >there are Hive committers who don't touch that piece of code today. no? > > > > > >Ashish > > > > > > > > >On Wed, Dec 19, 2012 at 8:23 PM, Namit Jain wrote: > > > > > >> I don’t agree with the proposal. It is impractical to have a Hcat > > >>committer > > >> with commit access to Hcat only portions of Hive. We cannot guarantee > > >>that > > >> a Hcat > > >> committer will become a Hive committer in 6-9 months, that depends on > > >>what > > >> they do > > >> in the next 6-9 months. > > >> > > >> The current Hcat committers should spend more time in reviewing > patches, > > >> work on non-Hcat areas in Hive, and then gradually become a hive > > >> committer. They should not be given any preferential treatment, and > t
Re: [DISCUSS] HCatalog becoming a subproject of Hive
> Would the project not benefit in the long run if Hcat is > brought in and some day becomes the default metastore for Hive. Folks, can we please try to keep this straight? HCatalog on its own does not provide any support for metadata. It is a set of wrapper APIs that make Hive's metastore and serdes accessible to Pig and MR. I think these wrappers provide a lot of value, and I'm eager to see them merged into Hive, but I'm dreading the marketing campaign that I suspect will follow: "Hive now supports metadata thanks to HCatalog!".
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I agree with Namit on this issue. I don't think it's fair to the existing group of Hive contributors to give preferential treatment to HCat committers, or to automatically promote them to full committer status on the Hive project. On Thu, Dec 20, 2012 at 1:10 PM, Bhandarkar, Milind < milind.bhandar...@emc.com> wrote: > I agree with Ashish. > > When Hcat becomes a subproject of Hive, all Hcat committers should > immediately become Hive committers. > > After all, that worked well for Hadoop, where all Hadoop committers can > commit to all Hadoop code (common/HDFS/MapReduce), but not all do, instead > focusing only on their area of expertise, and familiarity with portions of > codebase. > > - milind > > --- > Milind Bhandarkar > Chief Scientist, > Machine Learning Platforms, > Greenplum, A Division of EMC > +1-650-523-3858 (W) > +1-408-666-8483 (C) > > > > > > On 12/20/12 5:58 AM, "Ashish Thusoo" wrote: > > >Actually I don't understand why getting Hcat folks as committers on Hive > >is > >a problem. Hive itself became a subproject of Hadoop when it started with > >all the Hive committers becoming Hadoop committers. And of course everyone > >maintained the discipline that they commit in parts of the code that they > >understand and that they have worked on. Some of the committers from Hive > >ended up becoming Hadoop committers - others who worked only on Hive ended > >up leaving the Hadoop committers list once Hive became a TLP. So why put > >in > >these arguments about process when the end result would be beneficial to > >the community and to the project. Would Hive not benefit if some folks > >from > >Hcat start working on Hive proper as well - of course under the guidance > >of > >Hive mentors etc. Would the project not benefit in the long run if Hcat is > >brought in and some day becomes the default metastore for Hive. I mean if > >there are so many long term benefits from this then why focus on control > >and code safety which I think any responsible committer knows how to > >navigate and there are well understood best practices for that. And why > >can't a committer be booted out if he/she is breaking the discipline and > >really nosing in places which he/she does not understand. > > > >I mean if we agree that directionally Hcat being a part of Hive makes > >sense > >then why don't we try to get rid of the procedural elements that would > >only > >slow down that transition? If there is angst about specific people on Hcat > >committers list on the Hive committers side (are there any?), then I think > >that should be addressed on a case by case basis but why enforce a general > >rule. In the same vein why have a rule saying in 6-9 months a Hcat > >committer becomes a Hive committer - how is that helpful? If they are > >changing the Hcat subproject in Hive are they not already Hive committers? > >And if they gain the expertise to review and commit code in the > >SemanticAnalyzer in a few months should they not be able to do that before > >9 months are over? And if they don't get that expertise in 9 months would > >they really review and commit anything in the SemanticAnalyzer - I mean > >there are Hive committers who don't touch that piece of code today. no? > > > >Ashish > > > > > >On Wed, Dec 19, 2012 at 8:23 PM, Namit Jain wrote: > > > >> I don’t agree with the proposal. It is impractical to have a Hcat > >>committer > >> with commit access to Hcat only portions of Hive. We cannot guarantee > >>that > >> a Hcat > >> committer will become a Hive committer in 6-9 months, that depends on > >>what > >> they do > >> in the next 6-9 months. > >> > >> The current Hcat committers should spend more time in reviewing patches, > >> work on non-Hcat areas in Hive, and then gradually become a hive > >> committer. They should not be given any preferential treatment, and the > >> process should be same as it would be for any other hive contributor > >> currently. Given that the expertise of the Hcat committers, they should > >> be inline for becoming a hive committer if they continue to work in > >>hive, > >> but that cannot be guaranteed. I agree that some Hive committers should > >>try > >> and help the existing Hcat patches, and again that is voluntary and > >> different > >> committers cannot be assigned to different parts of the code. > >> > >> Thanks, > >> -namit > >> > >> > >> > >> > >> > >> > >> > >> On 12/20/12 1:03 AM, "Carl Steinbach" wrote: > >> > >> >Alan's proposal sounds like a good idea to me. > >> > > >> >+1 > >> > > >> >On Dec 18, 2012 5:36 PM, "Travis Crawford" > >> >wrote: > >> > > >> >> Alan, I think your proposal sounds great. > >> >> > >> >> --travis > >> >> > >> >> On Tue, Dec 18, 2012 at 1:13 PM, Alan Gates > >> >>wrote: > >> >> > Carl, speaking just for myself and not as a representative of the > >>HCat > >> >> PPMC at this point, I am coming to agree with you that HCat > >>integrating > >> >> with Hive fully makes more sense. > >> >> > > >> >> > However, this makes the committer quest
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I agree with Ashish. When Hcat becomes a subproject of Hive, all Hcat committers should immediately become Hive committers. After all, that worked well for Hadoop, where all Hadoop committers can commit to all Hadoop code (common/HDFS/MapReduce), but not all do, instead focusing only on their area of expertise, and familiarity with portions of codebase. - milind --- Milind Bhandarkar Chief Scientist, Machine Learning Platforms, Greenplum, A Division of EMC +1-650-523-3858 (W) +1-408-666-8483 (C) On 12/20/12 5:58 AM, "Ashish Thusoo" wrote: >Actually I don't understand why getting Hcat folks as committers on Hive >is >a problem. Hive itself became a subproject of Hadoop when it started with >all the Hive committers becoming Hadoop committers. And of course everyone >maintained the discipline that they commit in parts of the code that they >understand and that they have worked on. Some of the committers from Hive >ended up becoming Hadoop committers - others who worked only on Hive ended >up leaving the Hadoop committers list once Hive became a TLP. So why put >in >these arguments about process when the end result would be beneficial to >the community and to the project. Would Hive not benefit if some folks >from >Hcat start working on Hive proper as well - of course under the guidance >of >Hive mentors etc. Would the project not benefit in the long run if Hcat is >brought in and some day becomes the default metastore for Hive. I mean if >there are so many long term benefits from this then why focus on control >and code safety which I think any responsible committer knows how to >navigate and there are well understood best practices for that. And why >can't a committer be booted out if he/she is breaking the discipline and >really nosing in places which he/she does not understand. > >I mean if we agree that directionally Hcat being a part of Hive makes >sense >then why don't we try to get rid of the procedural elements that would >only >slow down that transition? If there is angst about specific people on Hcat >committers list on the Hive committers side (are there any?), then I think >that should be addressed on a case by case basis but why enforce a general >rule. In the same vein why have a rule saying in 6-9 months a Hcat >committer becomes a Hive committer - how is that helpful? If they are >changing the Hcat subproject in Hive are they not already Hive committers? >And if they gain the expertise to review and commit code in the >SemanticAnalyzer in a few months should they not be able to do that before >9 months are over? And if they don't get that expertise in 9 months would >they really review and commit anything in the SemanticAnalyzer - I mean >there are Hive committers who don't touch that piece of code today. no? > >Ashish > > >On Wed, Dec 19, 2012 at 8:23 PM, Namit Jain wrote: > >> I don’t agree with the proposal. It is impractical to have a Hcat >>committer >> with commit access to Hcat only portions of Hive. We cannot guarantee >>that >> a Hcat >> committer will become a Hive committer in 6-9 months, that depends on >>what >> they do >> in the next 6-9 months. >> >> The current Hcat committers should spend more time in reviewing patches, >> work on non-Hcat areas in Hive, and then gradually become a hive >> committer. They should not be given any preferential treatment, and the >> process should be same as it would be for any other hive contributor >> currently. Given that the expertise of the Hcat committers, they should >> be inline for becoming a hive committer if they continue to work in >>hive, >> but that cannot be guaranteed. I agree that some Hive committers should >>try >> and help the existing Hcat patches, and again that is voluntary and >> different >> committers cannot be assigned to different parts of the code. >> >> Thanks, >> -namit >> >> >> >> >> >> >> >> On 12/20/12 1:03 AM, "Carl Steinbach" wrote: >> >> >Alan's proposal sounds like a good idea to me. >> > >> >+1 >> > >> >On Dec 18, 2012 5:36 PM, "Travis Crawford" >> >wrote: >> > >> >> Alan, I think your proposal sounds great. >> >> >> >> --travis >> >> >> >> On Tue, Dec 18, 2012 at 1:13 PM, Alan Gates >> >>wrote: >> >> > Carl, speaking just for myself and not as a representative of the >>HCat >> >> PPMC at this point, I am coming to agree with you that HCat >>integrating >> >> with Hive fully makes more sense. >> >> > >> >> > However, this makes the committer question even thornier. Travis >>and >> >> Namit, I think the shepherd proposal needs to lay out a clear and >>time >> >> bounded path to committership for HCat committers. Having HCat >> >>committers >> >> as second class Hive citizens for the long run will not be healthy. >>I >> >> propose the following as a starting point for discussion: >> >> > >> >> > All active HCat committers (those who have contributed or >>committed a >> >> patch in the last 6 months) will be made committers in the HCat >>portion >> >> only of Hive. In addition those committers
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Actually I don't understand why getting Hcat folks as committers on Hive is a problem. Hive itself became a subproject of Hadoop when it started with all the Hive committers becoming Hadoop committers. And of course everyone maintained the discipline that they commit in parts of the code that they understand and that they have worked on. Some of the committers from Hive ended up becoming Hadoop committers - others who worked only on Hive ended up leaving the Hadoop committers list once Hive became a TLP. So why put in these arguments about process when the end result would be beneficial to the community and to the project. Would Hive not benefit if some folks from Hcat start working on Hive proper as well - of course under the guidance of Hive mentors etc. Would the project not benefit in the long run if Hcat is brought in and some day becomes the default metastore for Hive. I mean if there are so many long term benefits from this then why focus on control and code safety which I think any responsible committer knows how to navigate and there are well understood best practices for that. And why can't a committer be booted out if he/she is breaking the discipline and really nosing in places which he/she does not understand. I mean if we agree that directionally Hcat being a part of Hive makes sense then why don't we try to get rid of the procedural elements that would only slow down that transition? If there is angst about specific people on Hcat committers list on the Hive committers side (are there any?), then I think that should be addressed on a case by case basis but why enforce a general rule. In the same vein why have a rule saying in 6-9 months a Hcat committer becomes a Hive committer - how is that helpful? If they are changing the Hcat subproject in Hive are they not already Hive committers? And if they gain the expertise to review and commit code in the SemanticAnalyzer in a few months should they not be able to do that before 9 months are over? And if they don't get that expertise in 9 months would they really review and commit anything in the SemanticAnalyzer - I mean there are Hive committers who don't touch that piece of code today. no? Ashish On Wed, Dec 19, 2012 at 8:23 PM, Namit Jain wrote: > I don’t agree with the proposal. It is impractical to have a Hcat committer > with commit access to Hcat only portions of Hive. We cannot guarantee that > a Hcat > committer will become a Hive committer in 6-9 months, that depends on what > they do > in the next 6-9 months. > > The current Hcat committers should spend more time in reviewing patches, > work on non-Hcat areas in Hive, and then gradually become a hive > committer. They should not be given any preferential treatment, and the > process should be same as it would be for any other hive contributor > currently. Given that the expertise of the Hcat committers, they should > be inline for becoming a hive committer if they continue to work in hive, > but that cannot be guaranteed. I agree that some Hive committers should try > and help the existing Hcat patches, and again that is voluntary and > different > committers cannot be assigned to different parts of the code. > > Thanks, > -namit > > > > > > > > On 12/20/12 1:03 AM, "Carl Steinbach" wrote: > > >Alan's proposal sounds like a good idea to me. > > > >+1 > > > >On Dec 18, 2012 5:36 PM, "Travis Crawford" > >wrote: > > > >> Alan, I think your proposal sounds great. > >> > >> --travis > >> > >> On Tue, Dec 18, 2012 at 1:13 PM, Alan Gates > >>wrote: > >> > Carl, speaking just for myself and not as a representative of the HCat > >> PPMC at this point, I am coming to agree with you that HCat integrating > >> with Hive fully makes more sense. > >> > > >> > However, this makes the committer question even thornier. Travis and > >> Namit, I think the shepherd proposal needs to lay out a clear and time > >> bounded path to committership for HCat committers. Having HCat > >>committers > >> as second class Hive citizens for the long run will not be healthy. I > >> propose the following as a starting point for discussion: > >> > > >> > All active HCat committers (those who have contributed or committed a > >> patch in the last 6 months) will be made committers in the HCat portion > >> only of Hive. In addition those committers will be assigned a > >>particular > >> shepherd who is a current Hive committer and who will be responsible for > >> mentoring them towards full Hive committership. As a part of this > >> mentorship the HCat committer will review patches of other contributors, > >> contribute patches to Hive (both inside and outside of HCatalog), > >>respond > >> to user issues on the mailing lists, etc. It is intended that as a > >>result > >> of this mentorship program HCat committers can become full Hive > >>committers > >> in 6-9 months. No new HCat only committers will be elected in Hive > >>after > >> this. All Hive committers will automatically also have commit rights on > >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I don’t agree with the proposal. It is impractical to have a Hcat committer with commit access to Hcat only portions of Hive. We cannot guarantee that a Hcat committer will become a Hive committer in 6-9 months, that depends on what they do in the next 6-9 months. The current Hcat committers should spend more time in reviewing patches, work on non-Hcat areas in Hive, and then gradually become a hive committer. They should not be given any preferential treatment, and the process should be same as it would be for any other hive contributor currently. Given that the expertise of the Hcat committers, they should be inline for becoming a hive committer if they continue to work in hive, but that cannot be guaranteed. I agree that some Hive committers should try and help the existing Hcat patches, and again that is voluntary and different committers cannot be assigned to different parts of the code. Thanks, -namit On 12/20/12 1:03 AM, "Carl Steinbach" wrote: >Alan's proposal sounds like a good idea to me. > >+1 > >On Dec 18, 2012 5:36 PM, "Travis Crawford" >wrote: > >> Alan, I think your proposal sounds great. >> >> --travis >> >> On Tue, Dec 18, 2012 at 1:13 PM, Alan Gates >>wrote: >> > Carl, speaking just for myself and not as a representative of the HCat >> PPMC at this point, I am coming to agree with you that HCat integrating >> with Hive fully makes more sense. >> > >> > However, this makes the committer question even thornier. Travis and >> Namit, I think the shepherd proposal needs to lay out a clear and time >> bounded path to committership for HCat committers. Having HCat >>committers >> as second class Hive citizens for the long run will not be healthy. I >> propose the following as a starting point for discussion: >> > >> > All active HCat committers (those who have contributed or committed a >> patch in the last 6 months) will be made committers in the HCat portion >> only of Hive. In addition those committers will be assigned a >>particular >> shepherd who is a current Hive committer and who will be responsible for >> mentoring them towards full Hive committership. As a part of this >> mentorship the HCat committer will review patches of other contributors, >> contribute patches to Hive (both inside and outside of HCatalog), >>respond >> to user issues on the mailing lists, etc. It is intended that as a >>result >> of this mentorship program HCat committers can become full Hive >>committers >> in 6-9 months. No new HCat only committers will be elected in Hive >>after >> this. All Hive committers will automatically also have commit rights on >> HCatalog. >> > >> > Alan. >> > >> > On Dec 14, 2012, at 10:05 AM, Carl Steinbach wrote: >> > >> >> On a functional level I don't think there is going to be much of a >> >> difference between the subproject option proposed by Travis and the >> other >> >> option where HCatalog becomes a TLP. In both cases HCatalog and Hive >> will >> >> have separate committers, separate code repositories, separate >>release >> >> cycles, and separate project roadmaps. Aside from ASF bureaucracy, I >> think >> >> the only major difference between the two options is that the >>subproject >> >> route will give the rest of the community the false impression that >>the >> two >> >> projects have coordinated roadmaps and a process to prevent >>overlapping >> >> functionality from appearing in both projects. Consequently, If these >> are >> >> the only two options then I would prefer that HCatalog become a TLP. >> >> >> >> On the other hand, I also agree with many of the sentiments that have >> >> already been expressed in this thread, namely that the two projects >>are >> >> closely related and that it would benefit the community at large if >>the >> two >> >> projects could be brought closer together. Up to this point the major >> >> source of pain for the HCatalog team has been the frequent necessity >>of >> >> making changes on both the Hive and HCatalog sides when implementing >>new >> >> features in HCatalog. This situation is compounded by the ASF >> requirement >> >> that release artifacts may not depend on snapshot artifacts from >>other >> ASF >> >> projects. Furthermore, if Hive adds a dependency on HCatalog then it >> will >> >> be subject to these same problems (in addition to the gross circular >> >> dependency!). >> >> >> >> I think the best way to avoid these problems is for HCatalog to >>become a >> >> Hive submodule. In this scenario HCatalog would exist as a >>subdirectory >> in >> >> the Hive repository and would be distributed as a Hive artifact in >> future >> >> Hive releases. In addition to solving the problems I mentioned >>earlier, >> I >> >> think this would also help to assuage the concerns of many Hive >> committers >> >> who don't want to see the MetaStore split out into a separate >>project. >> >> >> >> Thanks. >> >> >> >> Carl >> >> >> >> On Thu, Dec 13, 2012 at 7:59 PM, Namit Jain wrote: >> >> >> >>> I am fine with this. Any hive
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Alan's proposal sounds like a good idea to me. +1 On Dec 18, 2012 5:36 PM, "Travis Crawford" wrote: > Alan, I think your proposal sounds great. > > --travis > > On Tue, Dec 18, 2012 at 1:13 PM, Alan Gates wrote: > > Carl, speaking just for myself and not as a representative of the HCat > PPMC at this point, I am coming to agree with you that HCat integrating > with Hive fully makes more sense. > > > > However, this makes the committer question even thornier. Travis and > Namit, I think the shepherd proposal needs to lay out a clear and time > bounded path to committership for HCat committers. Having HCat committers > as second class Hive citizens for the long run will not be healthy. I > propose the following as a starting point for discussion: > > > > All active HCat committers (those who have contributed or committed a > patch in the last 6 months) will be made committers in the HCat portion > only of Hive. In addition those committers will be assigned a particular > shepherd who is a current Hive committer and who will be responsible for > mentoring them towards full Hive committership. As a part of this > mentorship the HCat committer will review patches of other contributors, > contribute patches to Hive (both inside and outside of HCatalog), respond > to user issues on the mailing lists, etc. It is intended that as a result > of this mentorship program HCat committers can become full Hive committers > in 6-9 months. No new HCat only committers will be elected in Hive after > this. All Hive committers will automatically also have commit rights on > HCatalog. > > > > Alan. > > > > On Dec 14, 2012, at 10:05 AM, Carl Steinbach wrote: > > > >> On a functional level I don't think there is going to be much of a > >> difference between the subproject option proposed by Travis and the > other > >> option where HCatalog becomes a TLP. In both cases HCatalog and Hive > will > >> have separate committers, separate code repositories, separate release > >> cycles, and separate project roadmaps. Aside from ASF bureaucracy, I > think > >> the only major difference between the two options is that the subproject > >> route will give the rest of the community the false impression that the > two > >> projects have coordinated roadmaps and a process to prevent overlapping > >> functionality from appearing in both projects. Consequently, If these > are > >> the only two options then I would prefer that HCatalog become a TLP. > >> > >> On the other hand, I also agree with many of the sentiments that have > >> already been expressed in this thread, namely that the two projects are > >> closely related and that it would benefit the community at large if the > two > >> projects could be brought closer together. Up to this point the major > >> source of pain for the HCatalog team has been the frequent necessity of > >> making changes on both the Hive and HCatalog sides when implementing new > >> features in HCatalog. This situation is compounded by the ASF > requirement > >> that release artifacts may not depend on snapshot artifacts from other > ASF > >> projects. Furthermore, if Hive adds a dependency on HCatalog then it > will > >> be subject to these same problems (in addition to the gross circular > >> dependency!). > >> > >> I think the best way to avoid these problems is for HCatalog to become a > >> Hive submodule. In this scenario HCatalog would exist as a subdirectory > in > >> the Hive repository and would be distributed as a Hive artifact in > future > >> Hive releases. In addition to solving the problems I mentioned earlier, > I > >> think this would also help to assuage the concerns of many Hive > committers > >> who don't want to see the MetaStore split out into a separate project. > >> > >> Thanks. > >> > >> Carl > >> > >> On Thu, Dec 13, 2012 at 7:59 PM, Namit Jain wrote: > >> > >>> I am fine with this. Any hive committers who wants to volunteer to be > >>> a hcat shepherd is welcome. > >>> > >>> > >>> > >>> On 12/14/12 7:01 AM, "Travis Crawford" > wrote: > >>> > Thanks for reviving this thread. Reviewing the comments everyone seems > to agree HCatalog makes sense as a Hive subproject. I think that's > great news for the Hadoop community. > > The discussion seems to have turned to one of committer permissions. I > agree with the Hive folks sentiment that its something that must be > earned. That said, I've found it challenging at times getting patches > into Hive that would help earn taking on a hive committer > responsibility. > > Proposal: if a couple hive committers can volunteer to be hcat > shepherds, we can work with the shepherds when making hive changes in > a timely manor. Conversely, we can help shepherd any hive committers > who are interested in working more with hcat. There are certainly > benefits to cross-committership, and this approach could help each > other build a history of meaningful contributio
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Alan, I think your proposal sounds great. --travis On Tue, Dec 18, 2012 at 1:13 PM, Alan Gates wrote: > Carl, speaking just for myself and not as a representative of the HCat PPMC > at this point, I am coming to agree with you that HCat integrating with Hive > fully makes more sense. > > However, this makes the committer question even thornier. Travis and Namit, > I think the shepherd proposal needs to lay out a clear and time bounded path > to committership for HCat committers. Having HCat committers as second class > Hive citizens for the long run will not be healthy. I propose the following > as a starting point for discussion: > > All active HCat committers (those who have contributed or committed a patch > in the last 6 months) will be made committers in the HCat portion only of > Hive. In addition those committers will be assigned a particular shepherd > who is a current Hive committer and who will be responsible for mentoring > them towards full Hive committership. As a part of this mentorship the HCat > committer will review patches of other contributors, contribute patches to > Hive (both inside and outside of HCatalog), respond to user issues on the > mailing lists, etc. It is intended that as a result of this mentorship > program HCat committers can become full Hive committers in 6-9 months. No > new HCat only committers will be elected in Hive after this. All Hive > committers will automatically also have commit rights on HCatalog. > > Alan. > > On Dec 14, 2012, at 10:05 AM, Carl Steinbach wrote: > >> On a functional level I don't think there is going to be much of a >> difference between the subproject option proposed by Travis and the other >> option where HCatalog becomes a TLP. In both cases HCatalog and Hive will >> have separate committers, separate code repositories, separate release >> cycles, and separate project roadmaps. Aside from ASF bureaucracy, I think >> the only major difference between the two options is that the subproject >> route will give the rest of the community the false impression that the two >> projects have coordinated roadmaps and a process to prevent overlapping >> functionality from appearing in both projects. Consequently, If these are >> the only two options then I would prefer that HCatalog become a TLP. >> >> On the other hand, I also agree with many of the sentiments that have >> already been expressed in this thread, namely that the two projects are >> closely related and that it would benefit the community at large if the two >> projects could be brought closer together. Up to this point the major >> source of pain for the HCatalog team has been the frequent necessity of >> making changes on both the Hive and HCatalog sides when implementing new >> features in HCatalog. This situation is compounded by the ASF requirement >> that release artifacts may not depend on snapshot artifacts from other ASF >> projects. Furthermore, if Hive adds a dependency on HCatalog then it will >> be subject to these same problems (in addition to the gross circular >> dependency!). >> >> I think the best way to avoid these problems is for HCatalog to become a >> Hive submodule. In this scenario HCatalog would exist as a subdirectory in >> the Hive repository and would be distributed as a Hive artifact in future >> Hive releases. In addition to solving the problems I mentioned earlier, I >> think this would also help to assuage the concerns of many Hive committers >> who don't want to see the MetaStore split out into a separate project. >> >> Thanks. >> >> Carl >> >> On Thu, Dec 13, 2012 at 7:59 PM, Namit Jain wrote: >> >>> I am fine with this. Any hive committers who wants to volunteer to be >>> a hcat shepherd is welcome. >>> >>> >>> >>> On 12/14/12 7:01 AM, "Travis Crawford" wrote: >>> Thanks for reviving this thread. Reviewing the comments everyone seems to agree HCatalog makes sense as a Hive subproject. I think that's great news for the Hadoop community. The discussion seems to have turned to one of committer permissions. I agree with the Hive folks sentiment that its something that must be earned. That said, I've found it challenging at times getting patches into Hive that would help earn taking on a hive committer responsibility. Proposal: if a couple hive committers can volunteer to be hcat shepherds, we can work with the shepherds when making hive changes in a timely manor. Conversely, we can help shepherd any hive committers who are interested in working more with hcat. There are certainly benefits to cross-committership, and this approach could help each other build a history of meaningful contributions and earn the privilege & responsibility of being committers. Thoughts? --travis On Thu, Dec 13, 2012 at 11:59 AM, Edward Capriolo wrote: > I initially was a hesitant of hcatalog mostly because I imagined we
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Carl, speaking just for myself and not as a representative of the HCat PPMC at this point, I am coming to agree with you that HCat integrating with Hive fully makes more sense. However, this makes the committer question even thornier. Travis and Namit, I think the shepherd proposal needs to lay out a clear and time bounded path to committership for HCat committers. Having HCat committers as second class Hive citizens for the long run will not be healthy. I propose the following as a starting point for discussion: All active HCat committers (those who have contributed or committed a patch in the last 6 months) will be made committers in the HCat portion only of Hive. In addition those committers will be assigned a particular shepherd who is a current Hive committer and who will be responsible for mentoring them towards full Hive committership. As a part of this mentorship the HCat committer will review patches of other contributors, contribute patches to Hive (both inside and outside of HCatalog), respond to user issues on the mailing lists, etc. It is intended that as a result of this mentorship program HCat committers can become full Hive committers in 6-9 months. No new HCat only committers will be elected in Hive after this. All Hive committers will automatically also have commit rights on HCatalog. Alan. On Dec 14, 2012, at 10:05 AM, Carl Steinbach wrote: > On a functional level I don't think there is going to be much of a > difference between the subproject option proposed by Travis and the other > option where HCatalog becomes a TLP. In both cases HCatalog and Hive will > have separate committers, separate code repositories, separate release > cycles, and separate project roadmaps. Aside from ASF bureaucracy, I think > the only major difference between the two options is that the subproject > route will give the rest of the community the false impression that the two > projects have coordinated roadmaps and a process to prevent overlapping > functionality from appearing in both projects. Consequently, If these are > the only two options then I would prefer that HCatalog become a TLP. > > On the other hand, I also agree with many of the sentiments that have > already been expressed in this thread, namely that the two projects are > closely related and that it would benefit the community at large if the two > projects could be brought closer together. Up to this point the major > source of pain for the HCatalog team has been the frequent necessity of > making changes on both the Hive and HCatalog sides when implementing new > features in HCatalog. This situation is compounded by the ASF requirement > that release artifacts may not depend on snapshot artifacts from other ASF > projects. Furthermore, if Hive adds a dependency on HCatalog then it will > be subject to these same problems (in addition to the gross circular > dependency!). > > I think the best way to avoid these problems is for HCatalog to become a > Hive submodule. In this scenario HCatalog would exist as a subdirectory in > the Hive repository and would be distributed as a Hive artifact in future > Hive releases. In addition to solving the problems I mentioned earlier, I > think this would also help to assuage the concerns of many Hive committers > who don't want to see the MetaStore split out into a separate project. > > Thanks. > > Carl > > On Thu, Dec 13, 2012 at 7:59 PM, Namit Jain wrote: > >> I am fine with this. Any hive committers who wants to volunteer to be >> a hcat shepherd is welcome. >> >> >> >> On 12/14/12 7:01 AM, "Travis Crawford" wrote: >> >>> Thanks for reviving this thread. Reviewing the comments everyone seems >>> to agree HCatalog makes sense as a Hive subproject. I think that's >>> great news for the Hadoop community. >>> >>> The discussion seems to have turned to one of committer permissions. I >>> agree with the Hive folks sentiment that its something that must be >>> earned. That said, I've found it challenging at times getting patches >>> into Hive that would help earn taking on a hive committer >>> responsibility. >>> >>> Proposal: if a couple hive committers can volunteer to be hcat >>> shepherds, we can work with the shepherds when making hive changes in >>> a timely manor. Conversely, we can help shepherd any hive committers >>> who are interested in working more with hcat. There are certainly >>> benefits to cross-committership, and this approach could help each >>> other build a history of meaningful contributions and earn the >>> privilege & responsibility of being committers. >>> >>> Thoughts? >>> >>> --travis >>> >>> >>> >>> On Thu, Dec 13, 2012 at 11:59 AM, Edward Capriolo >>> wrote: I initially was a hesitant of hcatalog mostly because I imagined we would end up in a spot very similar to this. Namely the hcatlog folks are interested in making a metastore to support pig, hive, and map reduce. However I get the impression th
Re: [DISCUSS] HCatalog becoming a subproject of Hive
On a functional level I don't think there is going to be much of a difference between the subproject option proposed by Travis and the other option where HCatalog becomes a TLP. In both cases HCatalog and Hive will have separate committers, separate code repositories, separate release cycles, and separate project roadmaps. Aside from ASF bureaucracy, I think the only major difference between the two options is that the subproject route will give the rest of the community the false impression that the two projects have coordinated roadmaps and a process to prevent overlapping functionality from appearing in both projects. Consequently, If these are the only two options then I would prefer that HCatalog become a TLP. On the other hand, I also agree with many of the sentiments that have already been expressed in this thread, namely that the two projects are closely related and that it would benefit the community at large if the two projects could be brought closer together. Up to this point the major source of pain for the HCatalog team has been the frequent necessity of making changes on both the Hive and HCatalog sides when implementing new features in HCatalog. This situation is compounded by the ASF requirement that release artifacts may not depend on snapshot artifacts from other ASF projects. Furthermore, if Hive adds a dependency on HCatalog then it will be subject to these same problems (in addition to the gross circular dependency!). I think the best way to avoid these problems is for HCatalog to become a Hive submodule. In this scenario HCatalog would exist as a subdirectory in the Hive repository and would be distributed as a Hive artifact in future Hive releases. In addition to solving the problems I mentioned earlier, I think this would also help to assuage the concerns of many Hive committers who don't want to see the MetaStore split out into a separate project. Thanks. Carl On Thu, Dec 13, 2012 at 7:59 PM, Namit Jain wrote: > I am fine with this. Any hive committers who wants to volunteer to be > a hcat shepherd is welcome. > > > > On 12/14/12 7:01 AM, "Travis Crawford" wrote: > > >Thanks for reviving this thread. Reviewing the comments everyone seems > >to agree HCatalog makes sense as a Hive subproject. I think that's > >great news for the Hadoop community. > > > >The discussion seems to have turned to one of committer permissions. I > >agree with the Hive folks sentiment that its something that must be > >earned. That said, I've found it challenging at times getting patches > >into Hive that would help earn taking on a hive committer > >responsibility. > > > >Proposal: if a couple hive committers can volunteer to be hcat > >shepherds, we can work with the shepherds when making hive changes in > >a timely manor. Conversely, we can help shepherd any hive committers > >who are interested in working more with hcat. There are certainly > >benefits to cross-committership, and this approach could help each > >other build a history of meaningful contributions and earn the > >privilege & responsibility of being committers. > > > >Thoughts? > > > >--travis > > > > > > > >On Thu, Dec 13, 2012 at 11:59 AM, Edward Capriolo > >wrote: > >> I initially was a hesitant of hcatalog mostly because I imagined we > >>would > >> end up in a spot very similar to this. > >> > >> Namely the hcatlog folks are interested in making a metastore to support > >> pig, hive, and map reduce. However I get the impression that many in > >>hive > >> do not care much to have a metastore that caters to everyone. Their > >>needs > >> are only based on what hive needs. Which I believe is the wrong way to > >>look > >> at this situation. > >> > >> I though to reply to this thread because I have been following this > >>Jira: > >> https://issues.apache.org/jira/browse/HIVE-3752 > >> > >> On a high level I do not like this duplication of effort and code. If > >>hive > >> is compatible with hcatalog I do not see why we put off merging the two > >>at > >> all. Hive users would get an immediate benefit if Hive used hcatalog > >>with > >> no apparent downside. Meanwhile we are putting this off and staying in > >>this > >> awkward transition phase. > >> > >> Personally, I do not have a problem being a hive committer and not > >>having > >> hcatalog commit. None of the hive work I have done has ever touched the > >> metastore. Also of the thousands of jiras and features we have added > >>only a > >> small portion require metastore changes. > >> > >> As long as a couple active users have commit on hive and the suggested > >> hcatalog subproject I do not think not having commit will be a > >>roadblock in > >> moving hive forward. > >> > >> > >> On Mon, Dec 3, 2012 at 6:22 PM, Alan Gates > >>wrote: > >> > >>> I am not sure where we are on this discussion. So far those who have > >>> chimed in seemed generally positive (Namit, Edward, Clark, Alexander). > >>> Namit and I have different visions for what the committership might > >>>loo
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I am fine with this. Any hive committers who wants to volunteer to be a hcat shepherd is welcome. On 12/14/12 7:01 AM, "Travis Crawford" wrote: >Thanks for reviving this thread. Reviewing the comments everyone seems >to agree HCatalog makes sense as a Hive subproject. I think that's >great news for the Hadoop community. > >The discussion seems to have turned to one of committer permissions. I >agree with the Hive folks sentiment that its something that must be >earned. That said, I've found it challenging at times getting patches >into Hive that would help earn taking on a hive committer >responsibility. > >Proposal: if a couple hive committers can volunteer to be hcat >shepherds, we can work with the shepherds when making hive changes in >a timely manor. Conversely, we can help shepherd any hive committers >who are interested in working more with hcat. There are certainly >benefits to cross-committership, and this approach could help each >other build a history of meaningful contributions and earn the >privilege & responsibility of being committers. > >Thoughts? > >--travis > > > >On Thu, Dec 13, 2012 at 11:59 AM, Edward Capriolo >wrote: >> I initially was a hesitant of hcatalog mostly because I imagined we >>would >> end up in a spot very similar to this. >> >> Namely the hcatlog folks are interested in making a metastore to support >> pig, hive, and map reduce. However I get the impression that many in >>hive >> do not care much to have a metastore that caters to everyone. Their >>needs >> are only based on what hive needs. Which I believe is the wrong way to >>look >> at this situation. >> >> I though to reply to this thread because I have been following this >>Jira: >> https://issues.apache.org/jira/browse/HIVE-3752 >> >> On a high level I do not like this duplication of effort and code. If >>hive >> is compatible with hcatalog I do not see why we put off merging the two >>at >> all. Hive users would get an immediate benefit if Hive used hcatalog >>with >> no apparent downside. Meanwhile we are putting this off and staying in >>this >> awkward transition phase. >> >> Personally, I do not have a problem being a hive committer and not >>having >> hcatalog commit. None of the hive work I have done has ever touched the >> metastore. Also of the thousands of jiras and features we have added >>only a >> small portion require metastore changes. >> >> As long as a couple active users have commit on hive and the suggested >> hcatalog subproject I do not think not having commit will be a >>roadblock in >> moving hive forward. >> >> >> On Mon, Dec 3, 2012 at 6:22 PM, Alan Gates >>wrote: >> >>> I am not sure where we are on this discussion. So far those who have >>> chimed in seemed generally positive (Namit, Edward, Clark, Alexander). >>> Namit and I have different visions for what the committership might >>>look >>> like, so I'd like to hear from other Hive PMC members what their view >>>is on >>> this. I have to say from an HCatalog perspective the proposition is >>>much >>> less attractive without some commit rights. >>> >>> On a related note, people should be aware of these threads in the >>> Incubator list: >>> >>> >>>http://mail-archives.apache.org/mod_mbox/incubator-general/201211.mbox/% >>>3CCAGU5spdWHNtJxgQ8f%3DnPEXx9xNLjyjOYaFfnSw4EyAjgm1c46w%40mail.gmail.com >>>%3E >>> >>> >>> >>>http://mail-archives.apache.org/mod_mbox/incubator-general/201211.mbox/% >>>3CCAKQbXgDZj_zMj4qSodXjMHV7xQZxpcY1-35cvq959YKLNd6tJQ%40mail.gmail.com%3 >>>E >>> >>> For those not inclined to read all the mails in the threads I will >>> summarize (though I urge all PMC members of Hive and PPMC members of >>>HCat >>> to read both mail threads because this is highly relevant to what we >>>are >>> discussing). There are two salient points in these threads: >>> >>> 1) It is not wise to build a subproject that is distinct from the main >>> project in the sense that it has separate community members interested >>>in >>> it. Bertrand, Arun, Chris Mattman, and Greg Stein all spoke against >>>this, >>> and all are long time Apache contributors with a lot of experience. >>>They >>> were all of the opinion that it was reasonable for one project to >>>release >>> separate products. >>> >>> 2) It is not wise to have committers that have access to parts of a >>> project but not others. Greg and Bertrand argued (and Arun seemed to >>> imply) that splitting up committer lists by sections of the code did >>>not >>> work out well. >>> >>> These insights cause me to question what we mean by subproject. I had >>> originally envisioned something that looked like Pig and Hive did when >>>they >>> were subprojects of Hadoop. But this violates both 1 and 2 above. >>>Given >>> this input from many of the "wise old timers" of Apache I think we >>>should >>> consider what we mean when we say subproject and how tightly we are >>>willing >>> to integrate these projects. Personally I think it makes sense to >>>continue >>> to pursue integrat
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Thanks for reviving this thread. Reviewing the comments everyone seems to agree HCatalog makes sense as a Hive subproject. I think that's great news for the Hadoop community. The discussion seems to have turned to one of committer permissions. I agree with the Hive folks sentiment that its something that must be earned. That said, I've found it challenging at times getting patches into Hive that would help earn taking on a hive committer responsibility. Proposal: if a couple hive committers can volunteer to be hcat shepherds, we can work with the shepherds when making hive changes in a timely manor. Conversely, we can help shepherd any hive committers who are interested in working more with hcat. There are certainly benefits to cross-committership, and this approach could help each other build a history of meaningful contributions and earn the privilege & responsibility of being committers. Thoughts? --travis On Thu, Dec 13, 2012 at 11:59 AM, Edward Capriolo wrote: > I initially was a hesitant of hcatalog mostly because I imagined we would > end up in a spot very similar to this. > > Namely the hcatlog folks are interested in making a metastore to support > pig, hive, and map reduce. However I get the impression that many in hive > do not care much to have a metastore that caters to everyone. Their needs > are only based on what hive needs. Which I believe is the wrong way to look > at this situation. > > I though to reply to this thread because I have been following this Jira: > https://issues.apache.org/jira/browse/HIVE-3752 > > On a high level I do not like this duplication of effort and code. If hive > is compatible with hcatalog I do not see why we put off merging the two at > all. Hive users would get an immediate benefit if Hive used hcatalog with > no apparent downside. Meanwhile we are putting this off and staying in this > awkward transition phase. > > Personally, I do not have a problem being a hive committer and not having > hcatalog commit. None of the hive work I have done has ever touched the > metastore. Also of the thousands of jiras and features we have added only a > small portion require metastore changes. > > As long as a couple active users have commit on hive and the suggested > hcatalog subproject I do not think not having commit will be a roadblock in > moving hive forward. > > > On Mon, Dec 3, 2012 at 6:22 PM, Alan Gates wrote: > >> I am not sure where we are on this discussion. So far those who have >> chimed in seemed generally positive (Namit, Edward, Clark, Alexander). >> Namit and I have different visions for what the committership might look >> like, so I'd like to hear from other Hive PMC members what their view is on >> this. I have to say from an HCatalog perspective the proposition is much >> less attractive without some commit rights. >> >> On a related note, people should be aware of these threads in the >> Incubator list: >> >> http://mail-archives.apache.org/mod_mbox/incubator-general/201211.mbox/%3CCAGU5spdWHNtJxgQ8f%3DnPEXx9xNLjyjOYaFfnSw4EyAjgm1c46w%40mail.gmail.com%3E >> >> >> http://mail-archives.apache.org/mod_mbox/incubator-general/201211.mbox/%3CCAKQbXgDZj_zMj4qSodXjMHV7xQZxpcY1-35cvq959YKLNd6tJQ%40mail.gmail.com%3E >> >> For those not inclined to read all the mails in the threads I will >> summarize (though I urge all PMC members of Hive and PPMC members of HCat >> to read both mail threads because this is highly relevant to what we are >> discussing). There are two salient points in these threads: >> >> 1) It is not wise to build a subproject that is distinct from the main >> project in the sense that it has separate community members interested in >> it. Bertrand, Arun, Chris Mattman, and Greg Stein all spoke against this, >> and all are long time Apache contributors with a lot of experience. They >> were all of the opinion that it was reasonable for one project to release >> separate products. >> >> 2) It is not wise to have committers that have access to parts of a >> project but not others. Greg and Bertrand argued (and Arun seemed to >> imply) that splitting up committer lists by sections of the code did not >> work out well. >> >> These insights cause me to question what we mean by subproject. I had >> originally envisioned something that looked like Pig and Hive did when they >> were subprojects of Hadoop. But this violates both 1 and 2 above. Given >> this input from many of the "wise old timers" of Apache I think we should >> consider what we mean when we say subproject and how tightly we are willing >> to integrate these projects. Personally I think it makes sense to continue >> to pursue integration, as I think HCat is really a set of interfaces on top >> of Hive and it makes sense to coalesce those into one project. I guess >> this would mean HCat becomes just another set of jars that Hive releases >> when it releases, rather than a stand alone entity. But I'm curious to >> hear what others think. >> >> Alan. >> >> On Nov 1
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I initially was a hesitant of hcatalog mostly because I imagined we would end up in a spot very similar to this. Namely the hcatlog folks are interested in making a metastore to support pig, hive, and map reduce. However I get the impression that many in hive do not care much to have a metastore that caters to everyone. Their needs are only based on what hive needs. Which I believe is the wrong way to look at this situation. I though to reply to this thread because I have been following this Jira: https://issues.apache.org/jira/browse/HIVE-3752 On a high level I do not like this duplication of effort and code. If hive is compatible with hcatalog I do not see why we put off merging the two at all. Hive users would get an immediate benefit if Hive used hcatalog with no apparent downside. Meanwhile we are putting this off and staying in this awkward transition phase. Personally, I do not have a problem being a hive committer and not having hcatalog commit. None of the hive work I have done has ever touched the metastore. Also of the thousands of jiras and features we have added only a small portion require metastore changes. As long as a couple active users have commit on hive and the suggested hcatalog subproject I do not think not having commit will be a roadblock in moving hive forward. On Mon, Dec 3, 2012 at 6:22 PM, Alan Gates wrote: > I am not sure where we are on this discussion. So far those who have > chimed in seemed generally positive (Namit, Edward, Clark, Alexander). > Namit and I have different visions for what the committership might look > like, so I'd like to hear from other Hive PMC members what their view is on > this. I have to say from an HCatalog perspective the proposition is much > less attractive without some commit rights. > > On a related note, people should be aware of these threads in the > Incubator list: > > http://mail-archives.apache.org/mod_mbox/incubator-general/201211.mbox/%3CCAGU5spdWHNtJxgQ8f%3DnPEXx9xNLjyjOYaFfnSw4EyAjgm1c46w%40mail.gmail.com%3E > > > http://mail-archives.apache.org/mod_mbox/incubator-general/201211.mbox/%3CCAKQbXgDZj_zMj4qSodXjMHV7xQZxpcY1-35cvq959YKLNd6tJQ%40mail.gmail.com%3E > > For those not inclined to read all the mails in the threads I will > summarize (though I urge all PMC members of Hive and PPMC members of HCat > to read both mail threads because this is highly relevant to what we are > discussing). There are two salient points in these threads: > > 1) It is not wise to build a subproject that is distinct from the main > project in the sense that it has separate community members interested in > it. Bertrand, Arun, Chris Mattman, and Greg Stein all spoke against this, > and all are long time Apache contributors with a lot of experience. They > were all of the opinion that it was reasonable for one project to release > separate products. > > 2) It is not wise to have committers that have access to parts of a > project but not others. Greg and Bertrand argued (and Arun seemed to > imply) that splitting up committer lists by sections of the code did not > work out well. > > These insights cause me to question what we mean by subproject. I had > originally envisioned something that looked like Pig and Hive did when they > were subprojects of Hadoop. But this violates both 1 and 2 above. Given > this input from many of the "wise old timers" of Apache I think we should > consider what we mean when we say subproject and how tightly we are willing > to integrate these projects. Personally I think it makes sense to continue > to pursue integration, as I think HCat is really a set of interfaces on top > of Hive and it makes sense to coalesce those into one project. I guess > this would mean HCat becomes just another set of jars that Hive releases > when it releases, rather than a stand alone entity. But I'm curious to > hear what others think. > > Alan. > > On Nov 14, 2012, at 10:22 PM, Namit Jain wrote: > > > The same criteria should be applied to all Hive committers. Only a > > committer should be able to commit code. > > I don¹t think we should bend this rule. Metastore is not a separate > > project, but a integral part of hive. > > > > -namit > > > > > > On 11/12/12 10:32 PM, "Alan Gates" wrote: > > > >> I would suggest looking over the patch history of HCat committers. I > >> think most of them have already contributed a number of patches to the > >> metastore. All are certainly aware of how to run Hive unit tests and > >> have an understanding of how Hive works. So I don't think it's fair to > >> say they would be unsafe with access to the metastore. And the Hive PMC > >> is there to assure this does not happen. If there are issues I am sure > >> they can deal with them. > >> > >> Alan. > >> > >> > >> On Nov 6, 2012, at 8:06 PM, Namit Jain wrote: > >> > >>> Alan, that would not be a good idea. Metastore code is part of hive > >>> code, > >>> and it > >>> would be safer if only Hive committers had commit access to that.
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I am not sure where we are on this discussion. So far those who have chimed in seemed generally positive (Namit, Edward, Clark, Alexander). Namit and I have different visions for what the committership might look like, so I'd like to hear from other Hive PMC members what their view is on this. I have to say from an HCatalog perspective the proposition is much less attractive without some commit rights. On a related note, people should be aware of these threads in the Incubator list: http://mail-archives.apache.org/mod_mbox/incubator-general/201211.mbox/%3CCAGU5spdWHNtJxgQ8f%3DnPEXx9xNLjyjOYaFfnSw4EyAjgm1c46w%40mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/incubator-general/201211.mbox/%3CCAKQbXgDZj_zMj4qSodXjMHV7xQZxpcY1-35cvq959YKLNd6tJQ%40mail.gmail.com%3E For those not inclined to read all the mails in the threads I will summarize (though I urge all PMC members of Hive and PPMC members of HCat to read both mail threads because this is highly relevant to what we are discussing). There are two salient points in these threads: 1) It is not wise to build a subproject that is distinct from the main project in the sense that it has separate community members interested in it. Bertrand, Arun, Chris Mattman, and Greg Stein all spoke against this, and all are long time Apache contributors with a lot of experience. They were all of the opinion that it was reasonable for one project to release separate products. 2) It is not wise to have committers that have access to parts of a project but not others. Greg and Bertrand argued (and Arun seemed to imply) that splitting up committer lists by sections of the code did not work out well. These insights cause me to question what we mean by subproject. I had originally envisioned something that looked like Pig and Hive did when they were subprojects of Hadoop. But this violates both 1 and 2 above. Given this input from many of the "wise old timers" of Apache I think we should consider what we mean when we say subproject and how tightly we are willing to integrate these projects. Personally I think it makes sense to continue to pursue integration, as I think HCat is really a set of interfaces on top of Hive and it makes sense to coalesce those into one project. I guess this would mean HCat becomes just another set of jars that Hive releases when it releases, rather than a stand alone entity. But I'm curious to hear what others think. Alan. On Nov 14, 2012, at 10:22 PM, Namit Jain wrote: > The same criteria should be applied to all Hive committers. Only a > committer should be able to commit code. > I don¹t think we should bend this rule. Metastore is not a separate > project, but a integral part of hive. > > -namit > > > On 11/12/12 10:32 PM, "Alan Gates" wrote: > >> I would suggest looking over the patch history of HCat committers. I >> think most of them have already contributed a number of patches to the >> metastore. All are certainly aware of how to run Hive unit tests and >> have an understanding of how Hive works. So I don't think it's fair to >> say they would be unsafe with access to the metastore. And the Hive PMC >> is there to assure this does not happen. If there are issues I am sure >> they can deal with them. >> >> Alan. >> >> >> On Nov 6, 2012, at 8:06 PM, Namit Jain wrote: >> >>> Alan, that would not be a good idea. Metastore code is part of hive >>> code, >>> and it >>> would be safer if only Hive committers had commit access to that. >>> >>> >>> On 11/6/12 11:25 PM, "Alan Gates" wrote: >>> On Nov 4, 2012, at 8:35 PM, Namit Jain wrote: > I like the idea of Hcatalog becoming a Hive sub-project. The > enhancements/bugs in the serde/metastore areas can indirectly > benefit the hive community, and it will be easier for the fix to be in > one > place. Having said that, I don't see serde/metastore > moving out of hive into a separate component. Things are tied too > closely > together. I am assuming that no new committers would > be automatically added to Hive as part of this, and both Hive and > HCatalog > will continue to have its own committers. One thing in this we'd like to discuss is the HCatalog committers having commit access to the metastore sections of Hive code. That doesn't mean it has to move into HCatalog's code base. But more and more the fixes and changes we're doing in HCatalog are really in Hive's metastore. So we believe it would make sense to give HCat committers access to that component as well as HCat. Alan. > > Thanks, > -namit > > > On 11/3/12 2:22 AM, "Alan Gates" wrote: > >> Hello Hive community. It is time for HCatalog to graduate from the >> Apache Incubator. Given the heavy dependence of HCatalog on Hive the >> HCatalog community agreed it made sense to explore graduating from >>
Re: [DISCUSS] HCatalog becoming a subproject of Hive
The same criteria should be applied to all Hive committers. Only a committer should be able to commit code. I don¹t think we should bend this rule. Metastore is not a separate project, but a integral part of hive. -namit On 11/12/12 10:32 PM, "Alan Gates" wrote: >I would suggest looking over the patch history of HCat committers. I >think most of them have already contributed a number of patches to the >metastore. All are certainly aware of how to run Hive unit tests and >have an understanding of how Hive works. So I don't think it's fair to >say they would be unsafe with access to the metastore. And the Hive PMC >is there to assure this does not happen. If there are issues I am sure >they can deal with them. > >Alan. > > >On Nov 6, 2012, at 8:06 PM, Namit Jain wrote: > >> Alan, that would not be a good idea. Metastore code is part of hive >>code, >> and it >> would be safer if only Hive committers had commit access to that. >> >> >> On 11/6/12 11:25 PM, "Alan Gates" wrote: >> >>> >>> On Nov 4, 2012, at 8:35 PM, Namit Jain wrote: >>> I like the idea of Hcatalog becoming a Hive sub-project. The enhancements/bugs in the serde/metastore areas can indirectly benefit the hive community, and it will be easier for the fix to be in one place. Having said that, I don't see serde/metastore moving out of hive into a separate component. Things are tied too closely together. I am assuming that no new committers would be automatically added to Hive as part of this, and both Hive and HCatalog will continue to have its own committers. >>> >>> One thing in this we'd like to discuss is the HCatalog committers >>>having >>> commit access to the metastore sections of Hive code. That doesn't >>>mean >>> it has to move into HCatalog's code base. But more and more the fixes >>> and changes we're doing in HCatalog are really in Hive's metastore. So >>> we believe it would make sense to give HCat committers access to that >>> component as well as HCat. >>> >>> Alan. >>> Thanks, -namit On 11/3/12 2:22 AM, "Alan Gates" wrote: > Hello Hive community. It is time for HCatalog to graduate from the > Apache Incubator. Given the heavy dependence of HCatalog on Hive the > HCatalog community agreed it made sense to explore graduating from >the > Incubator to become a subproject of Hive (see > > >http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/20120 >9. > mb > ox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and > > >http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/20121 >0. > mb > > >ox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gma >il > .c > om%3E ). To help both communities understand what HCatalog is and > hopes > to become we also developed a roadmap that summarizes HCatalog's > current > features, planned features, and other possible features under > discussion: > https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap > > So we are now approaching you to see if there is agreement in the >Hive > community that HCatalog graduating into Hive would make sense. > > Alan. >>> >> >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I would suggest looking over the patch history of HCat committers. I think most of them have already contributed a number of patches to the metastore. All are certainly aware of how to run Hive unit tests and have an understanding of how Hive works. So I don't think it's fair to say they would be unsafe with access to the metastore. And the Hive PMC is there to assure this does not happen. If there are issues I am sure they can deal with them. Alan. On Nov 6, 2012, at 8:06 PM, Namit Jain wrote: > Alan, that would not be a good idea. Metastore code is part of hive code, > and it > would be safer if only Hive committers had commit access to that. > > > On 11/6/12 11:25 PM, "Alan Gates" wrote: > >> >> On Nov 4, 2012, at 8:35 PM, Namit Jain wrote: >> >>> I like the idea of Hcatalog becoming a Hive sub-project. The >>> enhancements/bugs in the serde/metastore areas can indirectly >>> benefit the hive community, and it will be easier for the fix to be in >>> one >>> place. Having said that, I don't see serde/metastore >>> moving out of hive into a separate component. Things are tied too >>> closely >>> together. I am assuming that no new committers would >>> be automatically added to Hive as part of this, and both Hive and >>> HCatalog >>> will continue to have its own committers. >> >> One thing in this we'd like to discuss is the HCatalog committers having >> commit access to the metastore sections of Hive code. That doesn't mean >> it has to move into HCatalog's code base. But more and more the fixes >> and changes we're doing in HCatalog are really in Hive's metastore. So >> we believe it would make sense to give HCat committers access to that >> component as well as HCat. >> >> Alan. >> >>> >>> Thanks, >>> -namit >>> >>> >>> On 11/3/12 2:22 AM, "Alan Gates" wrote: >>> Hello Hive community. It is time for HCatalog to graduate from the Apache Incubator. Given the heavy dependence of HCatalog on Hive the HCatalog community agreed it made sense to explore graduating from the Incubator to become a subproject of Hive (see http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201209. mb ox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201210. mb ox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gmail .c om%3E ). To help both communities understand what HCatalog is and hopes to become we also developed a roadmap that summarizes HCatalog's current features, planned features, and other possible features under discussion: https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap So we are now approaching you to see if there is agreement in the Hive community that HCatalog graduating into Hive would make sense. Alan. >>> >> >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Alan, that would not be a good idea. Metastore code is part of hive code, and it would be safer if only Hive committers had commit access to that. On 11/6/12 11:25 PM, "Alan Gates" wrote: > >On Nov 4, 2012, at 8:35 PM, Namit Jain wrote: > >> I like the idea of Hcatalog becoming a Hive sub-project. The >> enhancements/bugs in the serde/metastore areas can indirectly >> benefit the hive community, and it will be easier for the fix to be in >>one >> place. Having said that, I don't see serde/metastore >> moving out of hive into a separate component. Things are tied too >>closely >> together. I am assuming that no new committers would >> be automatically added to Hive as part of this, and both Hive and >>HCatalog >> will continue to have its own committers. > >One thing in this we'd like to discuss is the HCatalog committers having >commit access to the metastore sections of Hive code. That doesn't mean >it has to move into HCatalog's code base. But more and more the fixes >and changes we're doing in HCatalog are really in Hive's metastore. So >we believe it would make sense to give HCat committers access to that >component as well as HCat. > >Alan. > >> >> Thanks, >> -namit >> >> >> On 11/3/12 2:22 AM, "Alan Gates" wrote: >> >>> Hello Hive community. It is time for HCatalog to graduate from the >>> Apache Incubator. Given the heavy dependence of HCatalog on Hive the >>> HCatalog community agreed it made sense to explore graduating from the >>> Incubator to become a subproject of Hive (see >>> >>>http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201209. >>>mb >>> ox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and >>> >>>http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201210. >>>mb >>> >>>ox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gmail >>>.c >>> om%3E ). To help both communities understand what HCatalog is and >>>hopes >>> to become we also developed a roadmap that summarizes HCatalog's >>>current >>> features, planned features, and other possible features under >>>discussion: >>> https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap >>> >>> So we are now approaching you to see if there is agreement in the Hive >>> community that HCatalog graduating into Hive would make sense. >>> >>> Alan. >> >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
On Nov 4, 2012, at 8:35 PM, Namit Jain wrote: > I like the idea of Hcatalog becoming a Hive sub-project. The > enhancements/bugs in the serde/metastore areas can indirectly > benefit the hive community, and it will be easier for the fix to be in one > place. Having said that, I don't see serde/metastore > moving out of hive into a separate component. Things are tied too closely > together. I am assuming that no new committers would > be automatically added to Hive as part of this, and both Hive and HCatalog > will continue to have its own committers. One thing in this we'd like to discuss is the HCatalog committers having commit access to the metastore sections of Hive code. That doesn't mean it has to move into HCatalog's code base. But more and more the fixes and changes we're doing in HCatalog are really in Hive's metastore. So we believe it would make sense to give HCat committers access to that component as well as HCat. Alan. > > Thanks, > -namit > > > On 11/3/12 2:22 AM, "Alan Gates" wrote: > >> Hello Hive community. It is time for HCatalog to graduate from the >> Apache Incubator. Given the heavy dependence of HCatalog on Hive the >> HCatalog community agreed it made sense to explore graduating from the >> Incubator to become a subproject of Hive (see >> http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201209.mb >> ox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and >> http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201210.mb >> ox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gmail.c >> om%3E ). To help both communities understand what HCatalog is and hopes >> to become we also developed a roadmap that summarizes HCatalog's current >> features, planned features, and other possible features under discussion: >> https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap >> >> So we are now approaching you to see if there is agreement in the Hive >> community that HCatalog graduating into Hive would make sense. >> >> Alan. >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
That is a good idea. I would really like to see hcatalog be a sub project and then have hive use remove it's metastore code in favour of hcatalog. How should we coordinate this? Edward On Mon, Nov 5, 2012 at 6:41 AM, Clark Yang (杨卓荦) wrote: > +1 for that. I like it. > > 2012/11/5 Alexander Lorenz : >> Like it too. >> +1 >> >> Thanks, >> Alex >> >> On Nov 5, 2012, at 5:35 AM, Namit Jain wrote: >> >>> I like the idea of Hcatalog becoming a Hive sub-project. The >>> enhancements/bugs in the serde/metastore areas can indirectly >>> benefit the hive community, and it will be easier for the fix to be in one >>> place. Having said that, I don't see serde/metastore >>> moving out of hive into a separate component. Things are tied too closely >>> together. I am assuming that no new committers would >>> be automatically added to Hive as part of this, and both Hive and HCatalog >>> will continue to have its own committers. >>> >>> Thanks, >>> -namit >>> >>> >>> On 11/3/12 2:22 AM, "Alan Gates" wrote: >>> Hello Hive community. It is time for HCatalog to graduate from the Apache Incubator. Given the heavy dependence of HCatalog on Hive the HCatalog community agreed it made sense to explore graduating from the Incubator to become a subproject of Hive (see http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201209.mb ox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201210.mb ox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gmail.c om%3E ). To help both communities understand what HCatalog is and hopes to become we also developed a roadmap that summarizes HCatalog's current features, planned features, and other possible features under discussion: https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap So we are now approaching you to see if there is agreement in the Hive community that HCatalog graduating into Hive would make sense. Alan. >>> >> >> -- >> Alexander Alten-Lorenz >> http://mapredit.blogspot.com >> German Hadoop LinkedIn Group: http://goo.gl/N8pCF >>
Re: [DISCUSS] HCatalog becoming a subproject of Hive
+1 for that. I like it. 2012/11/5 Alexander Lorenz : > Like it too. > +1 > > Thanks, > Alex > > On Nov 5, 2012, at 5:35 AM, Namit Jain wrote: > >> I like the idea of Hcatalog becoming a Hive sub-project. The >> enhancements/bugs in the serde/metastore areas can indirectly >> benefit the hive community, and it will be easier for the fix to be in one >> place. Having said that, I don't see serde/metastore >> moving out of hive into a separate component. Things are tied too closely >> together. I am assuming that no new committers would >> be automatically added to Hive as part of this, and both Hive and HCatalog >> will continue to have its own committers. >> >> Thanks, >> -namit >> >> >> On 11/3/12 2:22 AM, "Alan Gates" wrote: >> >>> Hello Hive community. It is time for HCatalog to graduate from the >>> Apache Incubator. Given the heavy dependence of HCatalog on Hive the >>> HCatalog community agreed it made sense to explore graduating from the >>> Incubator to become a subproject of Hive (see >>> http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201209.mb >>> ox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and >>> http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201210.mb >>> ox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gmail.c >>> om%3E ). To help both communities understand what HCatalog is and hopes >>> to become we also developed a roadmap that summarizes HCatalog's current >>> features, planned features, and other possible features under discussion: >>> https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap >>> >>> So we are now approaching you to see if there is agreement in the Hive >>> community that HCatalog graduating into Hive would make sense. >>> >>> Alan. >> > > -- > Alexander Alten-Lorenz > http://mapredit.blogspot.com > German Hadoop LinkedIn Group: http://goo.gl/N8pCF >
Re: [DISCUSS] HCatalog becoming a subproject of Hive
Like it too. +1 Thanks, Alex On Nov 5, 2012, at 5:35 AM, Namit Jain wrote: > I like the idea of Hcatalog becoming a Hive sub-project. The > enhancements/bugs in the serde/metastore areas can indirectly > benefit the hive community, and it will be easier for the fix to be in one > place. Having said that, I don't see serde/metastore > moving out of hive into a separate component. Things are tied too closely > together. I am assuming that no new committers would > be automatically added to Hive as part of this, and both Hive and HCatalog > will continue to have its own committers. > > Thanks, > -namit > > > On 11/3/12 2:22 AM, "Alan Gates" wrote: > >> Hello Hive community. It is time for HCatalog to graduate from the >> Apache Incubator. Given the heavy dependence of HCatalog on Hive the >> HCatalog community agreed it made sense to explore graduating from the >> Incubator to become a subproject of Hive (see >> http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201209.mb >> ox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and >> http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201210.mb >> ox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gmail.c >> om%3E ). To help both communities understand what HCatalog is and hopes >> to become we also developed a roadmap that summarizes HCatalog's current >> features, planned features, and other possible features under discussion: >> https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap >> >> So we are now approaching you to see if there is agreement in the Hive >> community that HCatalog graduating into Hive would make sense. >> >> Alan. > -- Alexander Alten-Lorenz http://mapredit.blogspot.com German Hadoop LinkedIn Group: http://goo.gl/N8pCF
Re: [DISCUSS] HCatalog becoming a subproject of Hive
I like the idea of Hcatalog becoming a Hive sub-project. The enhancements/bugs in the serde/metastore areas can indirectly benefit the hive community, and it will be easier for the fix to be in one place. Having said that, I don't see serde/metastore moving out of hive into a separate component. Things are tied too closely together. I am assuming that no new committers would be automatically added to Hive as part of this, and both Hive and HCatalog will continue to have its own committers. Thanks, -namit On 11/3/12 2:22 AM, "Alan Gates" wrote: >Hello Hive community. It is time for HCatalog to graduate from the >Apache Incubator. Given the heavy dependence of HCatalog on Hive the >HCatalog community agreed it made sense to explore graduating from the >Incubator to become a subproject of Hive (see >http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201209.mb >ox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and >http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201210.mb >ox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gmail.c >om%3E ). To help both communities understand what HCatalog is and hopes >to become we also developed a roadmap that summarizes HCatalog's current >features, planned features, and other possible features under discussion: > https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap > >So we are now approaching you to see if there is agreement in the Hive >community that HCatalog graduating into Hive would make sense. > >Alan.
[DISCUSS] HCatalog becoming a subproject of Hive
Hello Hive community. It is time for HCatalog to graduate from the Apache Incubator. Given the heavy dependence of HCatalog on Hive the HCatalog community agreed it made sense to explore graduating from the Incubator to become a subproject of Hive (see http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201209.mbox/%3C08C40723-8D4D-48EB-942B-8EE4327DD84A%40hortonworks.com%3E and http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201210.mbox/%3CCABN7xTCRM5wXGgJKEko0PmqDXhuAYpK%2BD-H57T29zcSGhkwGQw%40mail.gmail.com%3E ). To help both communities understand what HCatalog is and hopes to become we also developed a roadmap that summarizes HCatalog's current features, planned features, and other possible features under discussion: https://cwiki.apache.org/confluence/display/HCATALOG/HCatalog+Roadmap So we are now approaching you to see if there is agreement in the Hive community that HCatalog graduating into Hive would make sense. Alan.
