P Mohan created BIGTOP-764:
------------------------------
Summary: Bigtop mapreduce job fails in ubuntu precise - Invalid
shuffle port number
Key: BIGTOP-764
URL: https://issues.apache.org/jira/browse/BIGTOP-764
Project: Bigtop
Issue Type: Bug
Components: Deployment, VM
Affects Versions: 0.4.0
Environment: Ubuntu Precise 12.04 LTS 64 bit running on VMWare Fusion
Reporter: P Mohan
Priority: Minor
i installed bigtop 0.4.0 following the instructions in the wiki. Installation
was fine. i ran into "Invalid shuffle port number" when i tried to run the pi
calculation example. I did a initial check on the internet and found -
https://issues.apache.org/jira/browse/MAPREDUCE-2983 . but this did not help.
did anyone face this error before ?
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples*.jar pi 10 1000
error message:
12/10/21 19:26:03 INFO mapreduce.Job: map 0% reduce 0%
12/10/21 19:26:04 INFO mapreduce.Job: Task Id :
attempt_1350872346733_0001_m_000004_0, Status : FAILED
Container launch failed for container_1350872346733_0001_01_000006 :
java.lang.IllegalStateException: Invalid shuffle port number -1 returned for
attempt_1350872346733_0001_m_000004_0
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:162)
at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:373)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
12/10/21 19:26:04 WARN mapreduce.Job: Error reading task output Connection
refused
I am attaching the installation instructions i followed below. Any help would
be highly appreciated.
installation instructions
#Environment : Ubuntu 12.04 LTS 64 bit running on VMWare fusion
#Install JAVA
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java7-installer
#Set Java Home
JAVA_HOME=/usr/lib/jvm/java-7-oracle/
export JAVA_HOME
# add big top repos
wget -O-
http://www.apache.org/dist/incubator/bigtop/bigtop-0.4.0-incubating/repos/GPG-KEY-bigtop
| sudo apt-key add -
sudo wget -O /etc/apt/sources.list.d/bigtop.list
http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-0.4.0/label=precise/lastSuccessfulBuild/artifact/output/bigtop.list
sudo apt-get update
apt-cache search hadoop
#install hadoop
sudo apt-get install hadoop\* flume-* mahout\* oozie\* whirr-*
#initialize namenode - format
sudo /etc/init.d/hadoop-hdfs-namenode init
for i in hadoop-hdfs-namenode hadoop-hdfs-datanode ; do sudo service $i start ;
done
#setup folders necessary for hadoop operarions
sudo -u hdfs hadoop fs -mkdir /user/$USER
sudo -u hdfs hadoop fs -chown $USER:$USER /user/$USER
sudo -u hdfs hadoop fs -mkdir /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
sudo -u hdfs hadoop fs -mkdir /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn
sudo -u hdfs hadoop fs -mkdir /user/history
sudo -u hdfs hadoop fs -chown mapred:mapred /user/history
sudo -u hdfs hadoop fs -chmod 770 /user/history
sudo -u hdfs hadoop fs -mkdir /tmp/hadoop-yarn/staging
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp/hadoop-yarn/staging
sudo -u hdfs hadoop fs -mkdir /tmp/hadoop-yarn/staging/history/done_intermediate
sudo -u hdfs hadoop fs -chmod -R 1777
/tmp/hadoop-yarn/staging/history/done_intermediate
sudo -u hdfs hadoop fs -chown -R mapred:mapred /tmp/hadoop-yarn/staging
sudo service hadoop-yarn-resourcemanager start
sudo service hadoop-yarn-nodemanager start
#test
hadoop fs -lsr /
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
