[
https://issues.apache.org/jira/browse/GIRAPH-631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631362#comment-13631362
]
Eli Reisman commented on GIRAPH-631:
------------------------------------
I'm not going to have time to attack this right now. Here's what I know:
In the Maven profiles for various Hadoops, we hardcode a "hadoop.version" Maven
property for each. So for instance "hadoop_2.0.3" profile hardcodes
hadoop.version to be "2.0.3-alpha" so that the Maven repo deps can be properly
downloaded and resolved.
Of course this can all be overriden at the command line with -D options. But
something about the way our profiles interact (and/or the way the subprojects
like giraph-examples and the IO subprojects for Hive etc.) breaks still and I
didn't have time to investigate why.
If you decide to dive in and try this, I'm happy to help, ping me and I will
attempt to guide you or advise. But in the end there will be some changes to
our Maven set up.
There is always the chance that there is a more radical approach such as making
giraph-yarn a subproject but for various reasons I rejected this as a less
natural fit, especially when I saw that we could stitch the YARN code into our
real Giraph code with so little munging, and that this munging was required
anyway to get the profile to integrate with our MapReduce-based profiles at
all. If you want to pursue a more involved solution like this, I'm not against
it and am also happy to help where I can.
Finally, if no one cares about this and you leave it to collect dust, I'll come
back at some point and do it myself ;)
> Remove Hardcoded Dependency on Hadoop-2.0.3-alpha from YARN and replace with
> a more flexible Maven config
> ---------------------------------------------------------------------------------------------------------
>
> Key: GIRAPH-631
> URL: https://issues.apache.org/jira/browse/GIRAPH-631
> Project: Giraph
> Issue Type: Improvement
> Components: conf and scripts
> Affects Versions: 1.0, 1.1
> Reporter: Eli Reisman
> Fix For: 1.0, 1.1
>
>
> Currently, Giraph's YARN profile is hardcoded to Version 2.0.3-alpha of
> Hadoop. This is because of two problems:
> 1. Simply creating profiles that can "coexist" such as Hadoop's own
> -Pdist,native type mvn calls is not possible for us since we use munging and
> excludes in Maven to prevent compilation of the YARN code where the deps are
> not included (many profiles) and these excludes don't seem overridable. This
> has been documented online as a Maven "feature" already.
> 2. Simply resetting hadoop.version for the Maven build using a -D option
> should work and should probably be the right fix for us but in the brief time
> I played with it (and with our versioning story that affects backporting not
> decided yet) I did not get it to work myself for Giraph-13 (this is all
> documented there)
> Option 2 will look like:
> {code}
> mvn -Phadoop_yarn -Dhadoop.version=YOUR_FAVORITE_YARNY_HADOOP clean install
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
