It looks like there might be many other maven-jar-plugin executions without an <id> specified and without a classifier. In 2.4.1:
**/hadoop-mapreduce-client-app/pom.xml **/hadoop-mapreduce-client-jobclient/pom.xml **/hadoop-yarn-applications-distributedshell/pom.xml **/hadoop-yarn-common/pom.xml **/hadoop-yarn-server-resourcemanager/pom.xml **/hadoop-yarn-server-tests/pom.xml **/hadoop-yarn-server-web-proxy/pom.xml And some with an extra execution without a separate classifier at all: hadoop-project-dist/pom.xml So, it looks like it affects more than just the yarn components (should I file against YARN still, or a more general component?). I haven't been able to compare with other versions to see how far this problem persists. On Mon, Jun 20, 2016 at 3:58 PM Chris Nauroth <[email protected]> wrote: > I believe the intent there is for the Distributed Shell jar to be fully > built and available for use during a test run, which requires submitting > the jar through the YARN machinery as part of the application submission. > Potential solutions would be the configuration you suggested to skip the > default execution, or possibly move this test into the integration test > phase, which is not something we're currently taking advantage of in the > Hadoop build. > > Also, I have a small correction on my last email. If this problem is > limited to the YARN portion of the build (e.g. just > hadoop-yarn-applications-distributedshell), then please file the JIRA > against the YARN project. > > --Chris Nauroth > > From: Christopher <[email protected]> > Date: Monday, June 20, 2016 at 12:48 PM > > To: Chris Nauroth <[email protected]>, "[email protected]" < > [email protected]> > Subject: Re: i686 support > > I'm still in information-gathering mode right now... hoping somebody can > tell me what the *expected* behavior of this module's build is supposed to > be, so I can see exactly where there's a discrepancy and what the right > direction is. I can't file a JIRA saying it's not doing what it's supposed > to do, if I don't know what it's supposed to do. All I know right now is > that some portion of the build breaks if you upgrade the version of the > maven-jar-plugin. > > I *think* the intent is simply to build the main jar earlier in the > lifecycle (vs. creating a second jar). If somebody can confirm that's true, > then I'll have more information to file the JIRA with. > > On Mon, Jun 20, 2016 at 3:33 PM Chris Nauroth <[email protected]> > wrote: > >> Christopher, thank you for the follow-up information. Please feel free >> to file an Apache JIRA in the HADOOP project with your findings. >> >> --Chris Nauroth >> >> From: Christopher <[email protected]> >> Date: Monday, June 20, 2016 at 12:26 PM >> >> To: Chris Nauroth <[email protected]>, "[email protected]" < >> [email protected]> >> Subject: Re: i686 support >> >> As it turns out, it looks like it's because the version of >> maven-jar-plugin in Fedora was upgraded, and hadoop's build takes advantage >> of the fact that older maven-jar-plugin versions don't notice clobbering >> artifacts with altered ones. This is probably going to bite Hadoop (if not >> already) when upgrading to ASF parent pom 18, which specifies the newest >> maven-jar-plugin. >> >> The solution, which was suggested to me on the Fedora lists, is to skip >> the default-jar execution of the maven-jar-plugin, so that only the >> customized execution would run: >> >> <execution> >> <id>default-jar</id> >> <phase>skip</phase> >> </execution> >> >> Unfortunately, maven-jar-plugin doesn't seem to provide a proper "skip" >> flag (all plugins should!), so one has got to hack the execution by >> instructing it to run on a non-existing phase. >> >> Alternatively, Hadoop should really override the "default-jar" execution >> with its own configuration (which as far as I can tell, it just makes the >> jar build earlier in the lifecycle; side note: this is the wrong way to >> go... the "unit test" described in the comment, which requires the main >> jar, should probably be an "integration test"). >> >> Thoughts? >> >> (I'm not sure why the build worked for me fine in x86_64, vs. i686, but >> it appears to be a fluke... maybe I had an older version of >> maven-jar-plugin installed already?) >> >> On Fri, Jun 17, 2016 at 5:45 PM Christopher <[email protected]> wrote: >> >>> It's possible it's a problem with the Fedora-packaged Maven or the jar >>> plugin. However, I was able to do a mock build (mock in Fedora uses >>> containers) to build it fine on x86_64. I've been given a hint as to how to >>> tell mock to use a different arch, but haven't yet had a chance to try it. >>> >>> Mainly, I just want to make sure it's supported upstream. If it's not, >>> and never will be, then I can use ExcludeArch in the RPM spec. Otherwise, >>> I've gotta try to figure out why it's broken. >>> >>> On Thu, Jun 16, 2016 at 4:31 PM Chris Nauroth <[email protected]> >>> wrote: >>> >>>> Thank you for sharing that. >>>> >>>> I was expecting to see some kind of failure in building our C codebase, >>>> where a 32-bit vs. 64-bit portability problem is likely to happen. >>>> Instead, that's a failure in Maven's handling of a jar dependency. I tried >>>> building from the 2.4.0 source, and I couldn't reproduce the problem. >>>> >>>> Is it possible that this is a problem specific to the Maven version >>>> running your build? The BUILDING.txt file in the root of the source tree >>>> has details on what tools and what versions are required for the build. >>>> >>>> --Chris Nauroth >>>> >>>> From: Christopher <[email protected]> >>>> Date: Thursday, June 16, 2016 at 12:48 PM >>>> To: Chris Nauroth <[email protected]>, "[email protected]" >>>> <[email protected]> >>>> Subject: Re: i686 support >>>> >>>> Well, the error I got was: >>>> >>>> [ERROR] Failed to execute goal >>>> org.apache.maven.plugins:maven-jar-plugin:3.0.1:jar (default-jar) on >>>> project hadoop-yarn-applications-distributedshell: You have to use a >>>> classifier to attach supplemental artifacts to the project instead of >>>> replacing them. -> [Help 1] >>>> >>>> >>>> https://kojipkgs.fedoraproject.org//work/tasks/1712/14521712/build.log >>>> >>>> I'm not sure if this is a Fedora-specific problem (due to its >>>> dependency convergence and the specific version of the maven-jar-plugin, >>>> for example), or if it's an upstream problem. This was version 2.4.1. Right >>>> now, I'm just trying to stabilize the currently packaged 2.4 version, >>>> before I begin moving towards an upgrade path for users to migrate to a >>>> newer version (2.6, probably). It's just that I'm a bit overwhelmed with >>>> Hadoop's build and quantity of components. >>>> >>>> Not sure where to begin looking. >>>> >>>> On Thu, Jun 16, 2016 at 3:42 PM Chris Nauroth <[email protected]> >>>> wrote: >>>> >>>>> Hello Christopher, >>>>> >>>>> While 64-bit is certainly the common case at this point, we also have >>>>> produced 32-bit builds in the past. I'm not aware of any conscious choice >>>>> by the community to halt 32-bit support, so if that's not working, then it >>>>> might be a bug. Do you have more details on the build failure? That >>>>> might >>>>> help us confirm if it's a bug, and if so, move to tracking it in an Apache >>>>> JIRA. >>>>> >>>>> --Chris Nauroth >>>>> >>>>> From: Christopher <[email protected]> >>>>> Date: Thursday, June 16, 2016 at 12:35 PM >>>>> To: "[email protected]" <[email protected]> >>>>> Subject: i686 support >>>>> >>>>> Hi, >>>>> >>>>> Is i686 a supported arch by the upstream Hadoop developers? I'm trying >>>>> to package hadoop RPMs for Fedora (reviving somebody else's previous >>>>> efforts), and can't get things to build on i686. Is this normal? Does >>>>> upstream care about 32-bit users? What about release testing on i686? >>>>> >>>>> I normally wouldn't care about 32-bit, but Fedora packaging standards >>>>> encourage me to get it to work for all core architectures supported by >>>>> Fedora. >>>>> >>>>> Has anybody else done any 32-bit packaging that could help me >>>>> troubleshoot? I don't even own 32-bit hardware. (Even better if you want >>>>> to >>>>> help co-maintain Hadoop in Fedora.. that would be awesome). >>>>> >>>>>
