Author: vinodkv
Date: Thu Jul 7 12:54:41 2011
New Revision: 1143803
URL: http://svn.apache.org/viewvc?rev=1143803&view=rev
Log:
MAPREDUCE-2645. Updates to MRv2 INSTALL and README documentation. Contributed
by Josh Wills.
Modified:
hadoop/common/branches/MR-279/mapreduce/CHANGES.txt
hadoop/common/branches/MR-279/mapreduce/INSTALL
hadoop/common/branches/MR-279/mapreduce/yarn/README
Modified: hadoop/common/branches/MR-279/mapreduce/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/CHANGES.txt?rev=1143803&r1=1143802&r2=1143803&view=diff
==============================================================================
--- hadoop/common/branches/MR-279/mapreduce/CHANGES.txt (original)
+++ hadoop/common/branches/MR-279/mapreduce/CHANGES.txt Thu Jul 7 12:54:41 2011
@@ -5,6 +5,9 @@ Trunk (unreleased changes)
MAPREDUCE-279
+ MAPREDUCE-2645. Updates to MRv2 INSTALL and README documentation.
+ (Josh Wills via vinodkv)
+
MAPREDUCE-2633. Add a getCounter(Enum) method to the Counters record.
(Josh Wills via sharad)
Modified: hadoop/common/branches/MR-279/mapreduce/INSTALL
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/INSTALL?rev=1143803&r1=1143802&r2=1143803&view=diff
==============================================================================
--- hadoop/common/branches/MR-279/mapreduce/INSTALL (original)
+++ hadoop/common/branches/MR-279/mapreduce/INSTALL Thu Jul 7 12:54:41 2011
@@ -29,7 +29,7 @@ ant veryclean jar jar-test -Dresolvers=
In case you want to skip the tests run:
-mvn clean install assembly:assembly -Dmaven.test.skip.exec=true
+mvn clean install assembly:assembly -DskipTests
ant veryclean jar jar-test -Dresolvers=internal
You will see a tarball in
@@ -55,7 +55,9 @@ Step 9) bin/yarn-daemon.sh start resourc
Step 10) bin/yarn-daemon.sh start nodemanager
-Step 11) Create the following symlinks in hadoop-common/lib
+Step 11) bin/yarn-daemon.sh start historyserver
+
+Step 12) Create the following symlinks in hadoop-common/lib
ln -s
$HADOOP_YARN_INSTALL/modules/hadoop-mapreduce-client-app-1.0-SNAPSHOT.jar .
ln -s $HADOOP_YARN_INSTALL/modules/yarn-api-1.0-SNAPSHOT.jar .
@@ -66,8 +68,22 @@ ln -s $HADOOP_YARN_INSTALL/modules/yarn-
ln -s
$HADOOP_YARN_INSTALL/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
.
ln -s $HADOOP_YARN_INSTALL/lib/protobuf-java-2.4.0a.jar .
-Step 12) You are all set, an example on how to run a job is:
-
-$HADOOP_COMMON_HOME/bin/hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$HADOOP_YARN_INSTALL/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output
+Step 13) Yarn daemons are up! But for running mapreduce applications, which
now are in user land, you need to setup nodemanager with the following
configuration in your yarn-site.xml before you start the nodemanager.
+ <property>
+ <name>nodemanager.auxiluary.services</name>
+ <value>mapreduce.shuffle</value>
+ </property>
+
+ <property>
+ <name>nodemanager.aux.service.mapreduce.shuffle.class</name>
+ <value>org.apache.hadoop.mapred.ShuffleHandler</value>
+ </property>
+
+Step 14) You are all set, an example on how to run a mapreduce job is:
+
+cd $HADOOP_MAPRED_HOME
+ant examples -Dresolvers=internal
+$HADOOP_COMMON_HOME/bin/hadoop jar
$HADOOP_MAPRED_HOME/build/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar
randomwriter -Dmapreduce.job.user.name=$USER
-Dmapreduce.clientfactory.class.name=org.apache.hadoop.mapred.YarnClientFactory
-Dmapreduce.randomwriter.bytespermap=10000 -Ddfs.blocksize=536870912
-Ddfs.block.size=536870912 -libjars
$HADOOP_YARN_INSTALL/hadoop-mapreduce-1.0-SNAPSHOT/modules/hadoop-mapreduce-client-jobclient-1.0-SNAPSHOT.jar
output
+The output on the command line should be almost similar to what you see in the
JT/TT setup (Hadoop 0.20/0.21)
Modified: hadoop/common/branches/MR-279/mapreduce/yarn/README
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/yarn/README?rev=1143803&r1=1143802&r2=1143803&view=diff
==============================================================================
--- hadoop/common/branches/MR-279/mapreduce/yarn/README (original)
+++ hadoop/common/branches/MR-279/mapreduce/yarn/README Thu Jul 7 12:54:41 2011
@@ -8,9 +8,6 @@ Maven: Maven 2
Setup
-----
-Install the Avro maven plugin in local maven repository:
-mvn install:install-file
-Dfile=./avro-maven-plugin/avro-maven-plugin-1.4.0-SNAPSHOT.jar
-DpomFile=./avro-maven-plugin/avro-maven-plugin-1.4.0-SNAPSHOT.pom
-
Install protobuf 2.4.0a or higher (Download from
http://code.google.com/p/protobuf/downloads/list)
- install the protoc executable (configure, make, make install)
- install the maven artifact (cd java; mvn install)
@@ -72,32 +69,5 @@ yarn-common - Utilities which can be use
yarn-server - Implementation of the yarn-api
yarn-server-common - APIs shared between resourcemanager and nodemanager
- yarn-server-nodemanager
- yarn-server-resourcemanager
-
-yarn-prototype - AppMaster (Prototype) to launch TTs via nodemanager
-
-yarn-mapreduce - Mapreduce specific parts
- yarn-mapreduce-common - Mapreduce utilities and interfae shared between
client and mapreduce AppMaster
- yarn-mapreduce-client - Mapreduce client (uses JobClient)
- yarn-mapreduce-app - Mapreduce AppMaster
-
-yarn-master-worker - master-worker AppMaster
-
-General instructions for starting Yarn and MapReduce
-----------------------------------------------------
-At yarn project root,
-
- mvn clean install
-
-ResourceManager:
- cd yarn-server/yarn-server-resourcemanager
- mvn exec:java
-Dexec.mainClass=org.apache.hadoop.yarn.server.resourcemanager.ResourceManager
-
-NodeManager:
- cd yarn-server/yarn-server-nodemanager
- mvn exec:java
-Dexec.mainClass=org.apache.hadoop.yarn.server.nodemanager.NodeManager
-
-MRAppMaster
- cd yarn-mapreduce/yarn-mapreduce-app
- mvn exec:java
-Dexec.mainClass=org.apache.hadoop.yarn.mapreduce.app.MRAppMaster
-Dexec.args="<clusterts> <appid>"
+ yarn-server-nodemanager (TaskTracker replacement)
+ yarn-server-resourcemanager (JobTracker replacement)
