I think it might be good to look at how we can add support for new
hadoop versions. Currently we have hadoop_facebook
(https://issues.apache.org/jira/browse/GIRAPH-14). I am considering
adding new ones such as hadoop_0.24. Looking at the code, it seems that
the main hadoop variation between the stock hadoop used (0.203.0) versus
facebook has to do with the new security-related APIs in the latter that
is, fortunately, also available in hadoop 0.23 and 0.24. So, hopefully
we can make use of the existing work that Avery has done for
hadoop_facebook and apply it to other hadoop versions. Therefore I would
1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java
and a few other places, where we are currently checking for
HADOOP_FACEBOOK and HADOOP.
2. we make a new profile called hadoop_secure, which, as with
hadoop_facebook, will use the above munge flag.
3. we make a new profile "hadoop_0.20.203" for the existing default
hadoop and make it the default profile (activeByDefault=true). This will
makes it easier to handle the differences in the hadoop library
dependency set that have happened between 0.20.203 and hadoop trunk.
Please see https://github.com/ekoontz/giraph/tree/security-profile for
my branch that implements the above.
- hadoop version profiles Eugene Koontz