Thanks Hyunsik and Avery for your comments!
I've added a JIRA here:
https://issues.apache.org/jira/browse/GIRAPH-168 to present a patch
that adds two new profiles:
hadoop_0.20.203 is the default (i.e. same as if you do not give a -P
I tried "mvn clean verify && mvn -Phadoop_0.20.203 clean verify && mvn
-Phadoop_trunk clean verify" and this worked with the GIRAPH-168 patch
As I said on GIRAPH-168, I could not get hadoop_non_secure or
hadoop_facebook to compile.
I agree that munge is hacky - in my patch I tried to modify our use of
munge a bit to make it simpler: I removed the usages of HADOOP (e.g.
"/*if_not[HADOOP]"). I felt like this usage is misleading, because it
might make you think that Giraph can be compiled without Hadoop support
at all, which is not what is meant by "/*if_not[HADOOP]": in fact, what
it actually means is that the Hadoop to be used is the default version
On 3/21/12 3:46 AM, Hyunsik Choi wrote:
This approach looks good.
On Wed, Mar 21, 2012 at 3:53 PM, Avery Ching<ach...@apache.org> wrote:
I agree with this approach, although munge is kinda hacky. It is easy
On 3/20/12 5:52 PM, Eugene Koontz wrote:
I think it might be good to look at how we can add support for new
hadoop versions. Currently we have hadoop_facebook (
I am considering adding new ones such as hadoop_0.24. Looking at the code,
it seems that the main hadoop variation between the stock hadoop used
(0.203.0) versus facebook has to do with the new security-related APIs in
the latter that is, fortunately, also available in hadoop 0.23 and 0.24.
So, hopefully we can make use of the existing work that Avery has done for
hadoop_facebook and apply it to other hadoop versions. Therefore I would
1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java and
a few other places, where we are currently checking for HADOOP_FACEBOOK and
2. we make a new profile called hadoop_secure, which, as with
hadoop_facebook, will use the above munge flag.
3. we make a new profile "hadoop_0.20.203" for the existing default
hadoop and make it the default profile (activeByDefault=true). This will
makes it easier to handle the differences in the hadoop library dependency
set that have happened between 0.20.203 and hadoop trunk.
my branch that implements the above.