On Sat, Feb 13, 2010 at 1:51 AM, Dhruba Borthakur <dhr...@gmail.com> wrote: > My personal experience is that the ivy-maven-stuff introduced into the > Hadoop build system has tremendously slowed down the Hadoop build process. I > am sure that this disadvantage is offet-ed by some advantages that I am not > aware of. If you could educate me on the top two advantages that accrued to > Hadoop after moving to the new build process, that would be awesome. > I'll try and answer your question Dhruba.
The hadoop build system change was done over in HADOOP-3305. The justification for the change was, to quote Steve, "To let people downstream build/test with hadoop, using Apache Ivy or Apache Maven2 to pull it down, hadoop-core needs to be published to the apache repository with a .pom file that lists its mandatory dependencies." Ivy was introduced to facilitate the above. I don't see too much discussion of it up on hadoop lists (My search may be wonky). I see Ashish asking back in December of 2008 about how to manage dependencies between subprojects and hadoop core and the ivy being suggested because it doesn't replace ant, whereas maven does. So, it would seem there was only one advantage/improvement brought by the change in the hadoop build system: hadoop artifacts are now available in maven repositories -- I don't think they are published other than to the apache maven repository (ivy knows how to deal with maven repos) -- with a pom that describes their dependencies so that a maven/ivy build downstream can pull and integrate hadoop artifacts. St.Ack > thanks a bunch, > dhruba > > > On Sat, Feb 13, 2010 at 1:44 AM, Kay Kay <kaykay.uni...@gmail.com> wrote: > >> On 02/13/2010 01:29 AM, Dhruba Borthakur wrote: >> >>> From what I understand, the slowness of 'ivy' can be reduced if you can >>> fetch dependent jars from local ivy server, isn't it? >>> >>> >> The problem discussed is an artifact of hbase , trying to keep up with the >> most recent snapshots of hadoop-core/ hdfs / mapred and hence the ivy >> resolution is expensive that every time it hits the mvn repository to check >> for the latest snapshot , if any. So the slowness is due to the necessity >> to keep up with the dependencies to identify issues early in the cycle. >> Specifically this can be attributed to the - changing="true" in all the >> ivy.xml-s in hbase, for hadoop artifacts . I am looking to making it a >> configurable option to avoid expensive build time. >> >> >> This will not be an issue if this were a hbase release, depending on other >> releases of hadoop-core / mapred / common etc. >> Both ivy and maven does cache the artifacts locally making the roundtrip >> redundant (except for the first time, of course), so this should not be an >> issue for people trying to build the release from sources, since it should >> be moot by then. >> >> >> >> >> >> thanks, >>> dhruba >>> >>> On Sat, Feb 13, 2010 at 12:25 AM, Kay Kay<kaykay.uni...@gmail.com> >>> wrote: >>> >>> >>> >>>> Mathias - >>>> I have been using Ivy / Maven , interchangeably in different projects >>>> for >>>> the build management. Both of them clearly have their strong points and >>>> drawbacks. Ivy fits great for thrift because of the nature of tasks , >>>> involved using some external command-line (thrift generators) etc. As I >>>> mentioned before - HBase does not have such cross maven goals / between >>>> the >>>> hairs as the build lifecycle is pretty straight-forward. >>>> In any case - the intention is to get to publish HBase artifacts and >>>> maintain a smaller core and encouraging contribs. from the artifacts as >>>> opposed to getting into the codebase. >>>> Once there are HBase artifacts published , the contrib / plugins for the >>>> same would be free to use ivy (with m2compatible="true") / maven as >>>> appropriate. >>>> >>>> Ryan - >>>> The slowness is attributed to the 'changing="true" ' in ivy.xml-s for >>>> all >>>> the hadoop-common / -hdfs / -mapreduce snapshots that we are using. I am >>>> facing similar 'slowness' with other mvn hadoop (snapshot) dependencies >>>> as >>>> well. In retrospective, that should have been made a configurable flag in >>>> libraries.properties , to ease things. Hopefully that is sorted out soon. >>>> >>>> >>>> >>>> >>>> On 02/13/2010 12:10 AM, Ryan Rawson wrote: >>>> >>>> >>>> >>>>> Would you mind elaborating more? At the moment, most people do not >>>>> build hbase, and the POM/jar/publishing thing is orthogonal - those >>>>> who wish to build their own projects with ivy and/or ant are free to >>>>> do so and not be impacted by our use of maven. >>>>> >>>>> We have ivy, but it doesnt integrate with our IDEs and is rather slow >>>>> to build and rebuild. >>>>> >>>>> On Sat, Feb 13, 2010 at 12:03 AM, Mathias Herberts >>>>> <mathias.herbe...@gmail.com> wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> -1 >>>>>> >>>>>> I think Maven is too complex and will lower the adoption of HBase by >>>>>> people today willing to build it. >>>>>> >>>>>> I would suggest using Ivy for dependency management as was done in >>>>>> Thrift. >>>>>> >>>>>> Mathias. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >>> >> >> > > > -- > Connect to me at http://www.facebook.com/dhruba >
