Hi Giraphers,
I think it might be good to look at how we can add support for new hadoop versions. Currently we have hadoop_facebook (https://issues.apache.org/jira/browse/GIRAPH-14). I am considering adding new ones such as hadoop_0.24. Looking at the code, it seems that the main hadoop variation between the stock hadoop used (0.203.0) versus facebook has to do with the new security-related APIs in the latter that is, fortunately, also available in hadoop 0.23 and 0.24. So, hopefully we can make use of the existing work that Avery has done for hadoop_facebook and apply it to other hadoop versions. Therefore I would propose that:


1. a new munge flag HADOOP_SECURE to be used in RPCCommunication.java and a few other places, where we are currently checking for HADOOP_FACEBOOK and HADOOP. 2. we make a new profile called hadoop_secure, which, as with hadoop_facebook, will use the above munge flag. 3. we make a new profile "hadoop_0.20.203" for the existing default hadoop and make it the default profile (activeByDefault=true). This will makes it easier to handle the differences in the hadoop library dependency set that have happened between 0.20.203 and hadoop trunk.

Please see https://github.com/ekoontz/giraph/tree/security-profile for my branch that implements the above.

Thanks,

-Eugene

Reply via email to