Author: aching
Date: Tue Dec 20 19:53:47 2011
New Revision: 1221461
URL: http://svn.apache.org/viewvc?rev=1221461&view=rev
Log:
GIRAPH-110: Add guide to setup the enviroment for running the
unittests in a pseudo-distributed hadoop instance. (ssc via aching)
Modified:
incubator/giraph/trunk/CHANGELOG
incubator/giraph/trunk/README
Modified: incubator/giraph/trunk/CHANGELOG
URL:
http://svn.apache.org/viewvc/incubator/giraph/trunk/CHANGELOG?rev=1221461&r1=1221460&r2=1221461&view=diff
==============================================================================
--- incubator/giraph/trunk/CHANGELOG (original)
+++ incubator/giraph/trunk/CHANGELOG Tue Dec 20 19:53:47 2011
@@ -2,6 +2,9 @@ Giraph Change Log
Release 0.70.0 - unreleased
+ GIRAPH-110: Add guide to setup the enviroment for running the
+ unittests in a pseudo-distributed hadoop instance. (ssc via aching)
+
GIRAPH-73: A little refactoring. (ssc via aching)
GIRAPH-106: Change prepareSuperstep() to make
Modified: incubator/giraph/trunk/README
URL:
http://svn.apache.org/viewvc/incubator/giraph/trunk/README?rev=1221461&r1=1221460&r2=1221461&view=diff
==============================================================================
--- incubator/giraph/trunk/README (original)
+++ incubator/giraph/trunk/README Tue Dec 20 19:53:47 2011
@@ -56,7 +56,8 @@ Building and testing:
You will need the following:
- Java 1.6
-- Maven 3 or higher. Giraph uses the munge plugin
(http://sonatype.github.com/munge-maven-plugin/),
+- Maven 3 or higher. Giraph uses the munge plugin
+ (http://sonatype.github.com/munge-maven-plugin/),
which requires Maven 3, to support multiple versions of Hadoop. Also, the
web site plugin requires Maven 3.
@@ -64,8 +65,6 @@ Use the maven commands with secure Hadoo
- compile (i.e mvn compile)
- package (i.e. mvn package)
- test (i.e. mvn test)
--- For testing, one can submit the test to a running Hadoop instance
- (i.e. mvn test -Dprop.mapred.job.tracker=localhost:50300)
For the non-secure versions of Hadoop, run the maven commands with the
additional argument '-Dhadoop=non_secure' to enable the maven profile
@@ -79,6 +78,51 @@ example compilation command is 'mvn -Dha
-Dhadoop.jar.path=/tmp/hadoop-0.20.1-core.jar compile'.
+How to run the unittests on a local pseudo-distributed Hadoop instance:
+
+As mentioned earlier, Giraph supports several versions of Hadoop. In
+this section, we describe how to run the Giraph unittests against a single
+node instance of Apache Hadoop 0.20.203.
+
+Download Apache Hadoop 0.20.203 (hadoop-0.20.203.0/hadoop-0.20.203.0rc1.tar.gz)
+from a mirror picked at http://www.apache.org/dyn/closer.cgi/hadoop/common/
+and unpack it into a local directory
+
+Follow the guide at
+http://hadoop.apache.org/common/docs/r0.20.2/quickstart.html#PseudoDistributed
+to setup a pseudo-distributed single node Hadoop cluster.
+
+Giraphâs code assumes that you can run at least 4 mappers at once,
+unfortunately the default configuration allows only 2. Therefore you need
+to update conf/mapred-site.xml:
+
+<property>
+ <name>mapred.tasktracker.map.tasks.maximum</name>
+ <value>4</value>
+</property>
+
+<property>
+ <name>mapred.map.tasks</name>
+ <value>4</value>
+</property>
+
+After preparing the local filesystem with:
+
+rm -rf /tmp/hadoop-<username>
+/path/to/hadoop/bin/hadoop namenode -format
+
+you can start the local hadoop instance:
+
+/path/to/hadoop/bin/start-all.sh
+
+and finally run Giraphâs unittests:
+
+mvn clean test -Dprop.mapred.job.tracker=localhost:9001
+
+Now you can open a browser, point it to http://localhost:50030 and watch the
+Giraph jobs from the unittests running on your local Hadoop instance!
+
+
Notes:
Counter limit: In Hadoop 0.20.203.0 onwards, there is a limit on the number of
counters one can use, which is set to 120 by default. This limit restricts the
