Ya I amended this for pushing the releases to maven. I actually changed that later to a different URI because it was incorrect (should have been https://)
On Wed, Sep 11, 2013 at 11:41 PM, Gary Struthers <[email protected]> wrote: > It looks like scm in the pom is only used to upload artifacts to Maven > Central and isn't needed for building. Explained here: > http://books.sonatype.com/nexus-book/reference/_introduction_7.html > > If you run into a transitive dependency problem, there is a nice way to > exclude them. Explained here: > http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-project-dependencies.html#pom-relationships-sect-conflict > > By far the best Maven documentation is in Sonatype's free books. They're > accurate and up to date, unlike the Apache Maven docs or StackOverflow. > http://www.sonatype.com/resources/books > > Gary Struthers > > On Sep 11, 2013, at 11:21 PM, Mark Hamstra <[email protected]> wrote: > >> Ah hah! Patrick figured it out: >> https://github.com/apache/incubator-spark/commit/905edf59db662868f55525118131cf102d366587. >> The buildnumber plugin and Debian packaging works correctly again. >> Thanks! I'll go through the poms tomorrow and see if there are any other >> such incompletely overriden elements that could be causing us troubles. >> >> Right now I am seeing two problems with the maven build: 1) I just had a >> build fail from exhausted PermGen, so we probably need to increase >> MaxPermGen in spark/pom.xml; 2) Not all of the modules got picked up in the >> recent invocations of the maven-release-plugin -- did your prepare & >> release use -Phadoop2-yar,repl-bin , Patrick? >> >> >> >> On Wed, Sep 11, 2013 at 10:31 PM, Mark Hamstra >> <[email protected]>wrote: >> >>> Yes, exactly. If I comment out the reference to the parent Apache pom, >>> then the buildnumber plugin works correctly. Similarly if I leave the >>> parent Apache pom reference in place but go into my local .m2 cache and >>> comment out just the <scm> in the org.apache:apache pom. In other words, >>> if both <scm>s are present, the buildnumber plugin only sees the SVN one >>> from the Apache pom, which makes things go wrong. >>> >>> >>> On Wed, Sep 11, 2013 at 9:18 PM, Henry Saputra >>> <[email protected]>wrote: >>> >>>> Hi Mark, >>>> >>>> But the spark main pom.xml contains <scm> element that should override >>>> <scm> from the parent pom.xml. >>>> >>>> Are you saying the pom.xml from parent Apache pom.xml overrides the >>>> Spark <scm> definition instead? >>>> >>>> >>>> >>>> >>>> On Wed, Sep 11, 2013 at 3:04 PM, Mark Hamstra <[email protected]> >>>> wrote: >>>>> I'm in the process of trying to build Apache-fied Spark into our stack >>>> at >>>>> ClearStory. We've been using Maven to do Debian packaging as found in >>>> the >>>>> repl-bin module for quite a while, but that doesn't work now. To see >>>> the >>>>> failure, you need to build repl-bin with the `deb` profile -- e.g. "mvn >>>>> -Prepl-bin,deb install" from the project directory. The problem is with >>>>> the buildnumber-maven-plugin, which is used to attach the 8-digit short >>>> SHA >>>>> of the current git commit to the Debian package -- so, e.g., >>>>> spark_0.8.0-SNAPSHOT-b993b2a3_all.deb. The buildnumber plugin relies >>>> upon >>>>> the <scm> tag to determine which source code management system should be >>>>> used to lookup the version number. >>>>> >>>>> That all worked great until the following was added to spark/pom.xml: >>>>> >>>>> <parent> >>>>> <groupId>org.apache</groupId> >>>>> <artifactId>apache</artifactId> >>>>> <version>11</version> >>>>> </parent> >>>>> >>>>> The problem is that the parent pom that the above includes has this >>>> within >>>>> it: >>>>> >>>>> <scm> >>>>> <connection>scm:svn: >>>>> http://svn.apache.org/repos/asf/maven/pom/tags/apache-11</connection >>>>> <developerConnection>scm:svn: >>>>> https://svn.apache.org/repos/asf/maven/pom/tags/apache-11 >>>>> </developerConnection> >>>>> <url>http://svn.apache.org/viewvc/maven/pom/tags/apache-11</url> >>>>> </scm> >>>>> >>>>> And that essentially overrides the <scm> section of spark/pom.xml, which >>>>> means that the buildnumber plugin thinks that we are using SVN instead >>>> of >>>>> git, and the SHA lookup fails. >>>>> >>>>> So, that is one known problem in the Maven build. I'm not at all >>>> certain >>>>> that there aren't others or lurking problems because of different >>>> versions >>>>> of dependencies, plugins, etc. being included by that org.apache:apache >>>>> artifact and also specified within the Spark pom files. >>>>> >>>>> I'm pretty sure that we can't not include the equivalent of >>>>> org.apache:apache:11, since there is a lot of important stuff in there >>>> to >>>>> do Apache release management, etc. So, is there another, equivalent >>>>> artifact that we can use that doesn't confuse the <scm> issue, or some >>>>> other way around this problem? It looks like there are some other >>>>> git-specific plugins out there that we may be able to use instead of >>>>> buildnumber-maven-plugin, so that may be a valid and necessary solution >>>> to >>>>> the Debian packaging problem. >>>>> >>>>> Can someone who knows Maven better than I take a look at the overlap >>>>> between org.apache:apache and the Spark pom files to see if there are >>>>> problems lurking there? Even if there aren't problems other than with >>>> the >>>>> buildnumber plugin, we may be able to trim a lot out of the Spark poms >>>> that >>>>> is already present in the apache parent pom. >>>> >>> >>> >
