Hi,
I am working on the types branch and am trying to use the
MiniCluster class for testing Job Submission and Monitoring in the
redesigned pipeline. I have hit into an issue.
The MiniCluster's Map Reduce part does not seem to work. I tried to
submit a wordCount MR job that exists in the hadoop examples jar. It
does not work. Here are the steps I followed:
(i) Started the MiniCluster using this:
package org.apache.pig.test;
public class TestMiniCluster {
public static void main(String[] args) throws
InterruptedException {
MiniCluster cluster = MiniCluster.buildCluster();
while(true){
Thread.sleep(10000);
}
}
}
(ii) hadoop --config ~/pigtest/conf dfs -put /etc/passwd passwd
(iii) hadoop --config ~/pigtest/conf dfs -ls
Found 1 items
/user/shravanm/passwd <r 3> 2768 2008-04-30 04:37
Verified that the file had been created
(iv) hadoop --config ~/pigtest/conf jar hadoop-0.15.3-examples.jar
wordcount passwd passwdCnt
Got the following exceptions:
08/04/30 04:37:41 INFO mapred.FileInputFormat: Total input paths to
process : 1
08/04/30 04:37:41 INFO mapred.JobClient: Running job:
job_200804300435_0001
08/04/30 04:37:42 INFO mapred.JobClient: map 0% reduce 0%
08/04/30 04:37:54 INFO mapred.JobClient: Task Id :
task_200804300435_0001_m_000000_0, Status : FAILED
08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_0&filter=stdout
08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_0&filter=stderr
08/04/30 04:37:55 INFO mapred.JobClient: Task Id :
task_200804300435_0001_m_000001_0, Status : FAILED
08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_0&filter=stdout
08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_0&filter=stderr
08/04/30 04:37:55 INFO mapred.JobClient: Task Id :
task_200804300435_0001_r_000000_0, Status : FAILED
08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_0&filter=stdout
08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_0&filter=stderr
08/04/30 04:37:58 INFO mapred.JobClient: Task Id :
task_200804300435_0001_m_000001_1, Status : FAILED
08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_1&filter=stdout
08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_1&filter=stderr
08/04/30 04:37:58 INFO mapred.JobClient: Task Id :
task_200804300435_0001_m_000000_1, Status : FAILED
08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_1&filter=stdout
08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_1&filter=stderr
08/04/30 04:38:01 INFO mapred.JobClient: Task Id :
task_200804300435_0001_r_000000_1, Status : FAILED
08/04/30 04:38:01 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_1&filter=stdout
08/04/30 04:38:01 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_1&filter=stderr
08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
task_200804300435_0001_m_000000_2, Status : FAILED
08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_2&filter=stdout
08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_2&filter=stderr
08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
task_200804300435_0001_r_000000_2, Status : FAILED
08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_2&filter=stdout
08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_2&filter=stderr
08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
task_200804300435_0001_m_000001_2, Status : FAILED
08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_2&filter=stdout
08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_2&filter=stderr
08/04/30 04:38:08 INFO mapred.JobClient: map 100% reduce 100%
08/04/30 04:38:08 INFO mapred.JobClient: Task Id :
task_200804300435_0001_r_000000_3, Status : TIPFAILED
08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_3&filter=stdout
08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_3&filter=stderr
08/04/30 04:38:08 INFO mapred.JobClient: Task Id :
task_200804300435_0001_m_000000_3, Status : TIPFAILED
08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_3&filter=stdout
08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_3&filter=stderr
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
at org.apache.hadoop.examples.WordCount.run(WordCount.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.examples.WordCount.main(WordCount.java:153)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at
org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:49)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
The same sequence when tried with an actual cluster setup on my local
machine works perfectly.
Am I doing something wrong? Should I use the MiniCluster in a different way?
I am using revision 631443, a bit old but this is the one that exists in
the type branch..
Thanks,
--Shravan
