On Mon, Jun 25, 2012 at 5:25 PM, Alan Gates <[email protected]> wrote: > My concern with these proposals are that they all involve experimentation. I > don't have strong opinions on maven vs ant. I don't know anything about > Gradle and need to learn more about it before I vote for or against it. > > Right now HCat needs to move quickly to fix some of the issues we're finding > in 0.4.0. Also, we need to get the Templeton work checked in. Locking the > trunk for a few days (which I imagine will extend to a week or two as we > won't get this right in the first pass) concerns me greatly. > > Could we instead do this work in a branch? I know that will put more onus on > those doing the conversion, but it does allow others to continue to make > progress on the trunk. Merges can be done from trunk to branch so that by > the time the branch is ready we can painlessly switch over. Also, this will > give others a chance to play with the proposed changes before committing to > them. >
With this in mind - what about the initial approach from the RB, where we do incremental improvements and keep the build functional. I'm hesitant to do this in a branch because of the overhead. Thoughts on the incremental approach? --travis > Alan. > > On Jun 25, 2012, at 8:04 AM, Travis Crawford wrote: > >> Trying out gradle would be pretty cool. I've never used it before but >> it looks promising. >> >> As long as we reorganize the project layout to match maven conventions >> we could try out gradle. If it doesn't work out for some reason we >> could easily add maven configs & trash the gradle ones. >> >> Thoughts? Regardless of gradle/maven, I like Rohini's idea to tolerate >> some build breakage for a couple days & crank this out. We'd need to >> give everyone enough advance warning to get outstanding patches >> committed because reworking them would be annoying after changing the >> layout. >> >> --travis >> >> >> On Sat, Jun 23, 2012 at 11:42 AM, Rohini Palaniswamy >> <[email protected]> wrote: >>> Adding to Travis list these would be hcatalog's requirements for a build >>> system: >>> * ability to easily specify dependencies and dependency repositories (I >>> hate ivy :) ). >>> * good support for easily writing multi-module projects with minimal >>> configuration (ties in with Travis thought on separating modules) >>> * plugins for junit, clover, checkstyle, findbugs and forrest >>> * easy way to run e2e tests. >>> >>> All of these are easily satisfied by maven (e2e will require ant >>> plugin). Gradle also satisfies them though some things are more difficult >>> considering it is still in its infancy compared to maven. For eg: clover >>> requires a lot of code (http://docs.codehaus.org/display/GRADLE/Cookbook), >>> but there are examples out there already and we can use them. The positive >>> thing with Gradle is it provides lot of flexibility unlike maven and it can >>> make doing a lot of things easier. For eg: If you have to copy file to some >>> other location in maven during your build you will have to use the ant >>> plugin. It also has good integration with ant, and you can easily write >>> ant tasks natively in gradle if a plugin is not available. The only con I >>> see with gradle is that if we don't write the build script well >>> modularized and cleanly, it might turn out to be unwieldy and difficult >>> to maneuver like a build.xml file. With maven the chances of that are less >>> because of the conventions and xml configuration. But if we have someone >>> (Jakob) who has good experience with it then it might not be a problem. I >>> am fine with going with either. >>> >>> In fact Gradle sounds more interesting just for the reason it is >>> something new and is gaining popularity and we would be an early adopter in >>> the hadoop family. But that's just the eager beaver in me talking :). We >>> don't have much complexity in our builds yet. So this would be the time to >>> decide maven or gradle and go with one of it. >>> >>> Regards, >>> Rohini >>> >>> On Fri, Jun 22, 2012 at 5:48 PM, Travis Crawford >>> <[email protected]>wrote: >>> >>>> On Fri, Jun 22, 2012 at 5:28 PM, Jakob Homan <[email protected]> wrote: >>>>> I've been wanting to take a crack at this with Gradle, which we've >>>>> used extensively with great results. I was hoping to show up with >>>>> something in hand rather than suggest something new, igniting a whole >>>>> big war. Any resistance to such an effort? >>>> >>>> Things I care about: >>>> >>>> * artifacts in the central maven repo >>>> * per-framework adapters packaged separate from the core functionality >>>> so dependencies are sane >>>> >>>> I don't have a strong preference for how that happens. Before having a >>>> build system throwdown - do those sound like the goals we're >>>> interested in working towards? >>>> >>>> --travis >>>> >>>> >>>>> On Fri, Jun 22, 2012 at 5:10 PM, Travis Crawford >>>>> <[email protected]> wrote: >>>>>> On Fri, Jun 22, 2012 at 11:48 AM, Rohini Palaniswamy < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Replying to the dev list instead of the jira as this requires >>>> everybody's >>>>>>> attention/thoughts. >>>>>>> >>>>>>> I would prefer all at once for mavenization. All the ivy stuff in ant >>>>>>> will be throwaway in maven. The directory structure would require good >>>>>>> amount of reorganizing (properly modularizing, moving code to standard >>>>>>> directory structure src/main/java, src/test/java, src/test/resources). >>>>>>> Trying to do it incrementally, moving out code and creating >>>> sub-projects is >>>>>>> ok, but the effort spent on changing ant scripts for that might be a >>>> waste >>>>>>> of effort as it is going to be thrown away. It would be better to have >>>> the >>>>>>> build in broken state for few days and complete mavenization faster >>>> instead >>>>>>> of trying to keep ant working when we move to maven. Since hcatalog is >>>> not >>>>>>> that complex, mavenizing should be easy and quick except for the e2e >>>> tests. >>>>>>> Even that can be quickly done with maven ant plugin before completely >>>>>>> moving to maven. >>>>>>> >>>>>> >>>>>> I'm down with this plan. Maybe have a hack day and crank this out? >>>> Anyone >>>>>> around SF who wants to work on this? I could host at the new Twitter >>>> diggs. >>>>>> >>>>>> If there's interest I can see how hosting people works and we can figure >>>>>> out a date. >>>>>> >>>>>> --travis >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Rohini >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 22, 2012 at 11:14 AM, Travis Crawford < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> This is an automatically generated e-mail. To reply, visit: >>>>>>>> https://reviews.apache.org/r/5496/ >>>>>>>> >>>>>>>> On June 22nd, 2012, 5:21 p.m., *Rohini Palaniswamy* wrote: >>>>>>>> >>>>>>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build.xml< >>>> https://reviews.apache.org/r/5496/diff/2/?file=115880#file115880line329> >>>> (Diff >>>>>>>> revision 2) >>>>>>>> >>>>>>>> 329 >>>>>>>> >>>>>>>> <antcall target="ivy-publish"/> >>>>>>>> >>>>>>>> Do we want to do ivy-publish inside the jar target. Would prefer it >>>> to be called separately in only apache hudson build. ivy-publish can >>>> probably depend on jar. All of us run jar in our hosts. Putting ivy-publish >>>> here might lead to issues. >>>>>>>> >>>>>>>> On June 22nd, 2012, 5:27 p.m., *Travis Crawford* wrote: >>>>>>>> >>>>>>>> So this publishes to the local ivy cache, which is needed because >>>> hcatalog-pig-adapter depends on hcatalog.jar. Otherwise the subproject >>>> dependency does not work. >>>>>>>> >>>>>>>> On June 22nd, 2012, 6 p.m., *Rohini Palaniswamy* wrote: >>>>>>>> >>>>>>>> Good then. Thought that it publishes a snapshot jar to the maven >>>> repo. Should have paid more attention to the "local" keyword there. >>>>>>>> >>>>>>>> What are your thoughts on this general approach of reorganizing the >>>> repo to match the maven layout, then we can mavenize? Are you okay with >>>> this incremental approach, or do you think it would be better all at once? >>>>>>>> >>>>>>>> Looking at the current build, I think we also want subprojects for: >>>>>>>> >>>>>>>> * hcatalog-server-extensions.jar >>>>>>>> * hcatalog-core.jar (today just hcatalog.jar; we could keep this just >>>> hcatalog.jar if people like it) >>>>>>>> >>>>>>>> I don't fully understand the storage-handler stuff, since doesn't >>>> that stuff belong in Hive? For now I was planning to leave as-is. >>>>>>>> >>>>>>>> Let me know if you want to do this all at once, or incremental. >>>>>>>> >>>>>>>> >>>>>>>> - Travis >>>>>>>> >>>>>>>> On June 22nd, 2012, 3:45 p.m., Travis Crawford wrote: >>>>>>>> Review request for hcatalog. >>>>>>>> By Travis Crawford. >>>>>>>> >>>>>>>> *Updated June 22, 2012, 3:45 p.m.* >>>>>>>> Description >>>>>>>> >>>>>>>> Update HCatalog build to package pig classes as a separate jar. I did >>>> not update ivy yet, but if the general approach looks good I will update. >>>> This will let the core hcatalog.jar depends only on stuff needed by all >>>> processing frameworks; then people that want to use pig can use the pig >>>> adapter which has the pig dependency. >>>>>>>> >>>>>>>> I believe we'll have more adapters in the future, so I'm trying to >>>> make this reusable. >>>>>>>> >>>>>>>> For example: >>>>>>>> >>>>>>>> Traviss-iMac:hcatalog travis$ jar -tvf >>>> hcatalog-pig-adapter/build/hcatalog-pig-adapter-0.5.0-dev.jar >>>>>>>> 0 Thu Jun 21 10:34:18 PDT 2012 META-INF/ >>>>>>>> 107 Thu Jun 21 10:34:16 PDT 2012 META-INF/MANIFEST.MF >>>>>>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/ >>>>>>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/ >>>>>>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/ >>>>>>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/pig/ >>>>>>>> 0 Thu Jun 21 10:34:16 PDT 2012 org/apache/hcatalog/pig/drivers/ >>>>>>>> 4352 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatBaseLoader.class >>>>>>>> 1261 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatBaseStorer$1.class >>>>>>>> 12413 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatBaseStorer.class >>>>>>>> 632 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatLoader$1.class >>>>>>>> 8518 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatLoader.class >>>>>>>> 6801 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/HCatStorer.class >>>>>>>> 1019 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/PigHCatUtil$1.class >>>>>>>> 13117 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/PigHCatUtil.class >>>>>>>> 3711 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat$LoadFuncBasedRecordReader.class >>>>>>>> 2383 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/LoadFuncBasedInputFormat.class >>>>>>>> 2189 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat$StoreFuncBasedOutputCommitter.class >>>>>>>> 1775 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat$StoreFuncBasedRecordWriter.class >>>>>>>> 2647 Thu Jun 21 10:34:16 PDT 2012 >>>> org/apache/hcatalog/pig/drivers/StoreFuncBasedOutputFormat.class >>>>>>>> Traviss-iMac:hcatalog travis$ >>>>>>>> >>>>>>>> Diffs >>>>>>>> >>>>>>>> - >>>>>>>> >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build-common-new.xml >>>>>>>> (PRE-CREATION) >>>>>>>> - >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/build.xml >>>>>>>> (1352540) >>>>>>>> - >>>>>>>> >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/hcatalog-pig-adapter/build.xml >>>>>>>> (PRE-CREATION) >>>>>>>> - >>>>>>>> >>>> http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/hcatalog-pig-adapter/ivy.xml >>>>>>>> (PRE-CREATION) >>>>>>>> - http://svn.apache.org/repos/asf/incubator/hcatalog/trunk/ivy.xml >>>>>>>> (1352540) >>>>>>>> >>>>>>>> View Diff <https://reviews.apache.org/r/5496/diff/> >>>>>>>> >>>>>>> >>>>>>> >>>> >
