Shravan,
Have you got the solution on this yet? I'm also expecting the answer from
someone else but seems like nobody knows. I guess what you have to do is to
access MiniCluster via PigServer API instead (=just a tiny : )pointer).
Please tell me how if you've already got this working.
Pi
On 4/30/08, Shravan M Narayanamurthy <[EMAIL PROTECTED]> wrote:
>
> Hi,
> I am working on the types branch and am trying to use the MiniCluster
> class for testing Job Submission and Monitoring in the redesigned pipeline.
> I have hit into an issue.
>
> The MiniCluster's Map Reduce part does not seem to work. I tried to submit
> a wordCount MR job that exists in the hadoop examples jar. It does not work.
> Here are the steps I followed:
>
> (i) Started the MiniCluster using this:
>
> package org.apache.pig.test;
>
> public class TestMiniCluster {
> public static void main(String[] args) throws
> InterruptedException {
> MiniCluster cluster = MiniCluster.buildCluster();
> while(true){
> Thread.sleep(10000);
> }
> }
> }
>
> (ii) hadoop --config ~/pigtest/conf dfs -put /etc/passwd passwd
>
> (iii) hadoop --config ~/pigtest/conf dfs -ls
> Found 1 items
> /user/shravanm/passwd <r 3> 2768 2008-04-30 04:37
>
> Verified that the file had been created
>
> (iv) hadoop --config ~/pigtest/conf jar hadoop-0.15.3-examples.jar
> wordcount passwd passwdCnt
> Got the following exceptions:
> 08/04/30 04:37:41 INFO mapred.FileInputFormat: Total input paths to
> process : 1
> 08/04/30 04:37:41 INFO mapred.JobClient: Running job:
> job_200804300435_0001
> 08/04/30 04:37:42 INFO mapred.JobClient: map 0% reduce 0%
> 08/04/30 04:37:54 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_m_000000_0, Status : FAILED
> 08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_0&filter=stdout
> 08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_0&filter=stderr
> 08/04/30 04:37:55 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_m_000001_0, Status : FAILED
> 08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_0&filter=stdout
> 08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_0&filter=stderr
> 08/04/30 04:37:55 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_r_000000_0, Status : FAILED
> 08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_0&filter=stdout
> 08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_0&filter=stderr
> 08/04/30 04:37:58 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_m_000001_1, Status : FAILED
> 08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_1&filter=stdout
> 08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_1&filter=stderr
> 08/04/30 04:37:58 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_m_000000_1, Status : FAILED
> 08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_1&filter=stdout
> 08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_1&filter=stderr
> 08/04/30 04:38:01 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_r_000000_1, Status : FAILED
> 08/04/30 04:38:01 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_1&filter=stdout
> 08/04/30 04:38:01 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_1&filter=stderr
> 08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_m_000000_2, Status : FAILED
> 08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_2&filter=stdout
> 08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_2&filter=stderr
> 08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_r_000000_2, Status : FAILED
> 08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_2&filter=stdout
> 08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_2&filter=stderr
> 08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_m_000001_2, Status : FAILED
> 08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_2&filter=stdout
> 08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_2&filter=stderr
> 08/04/30 04:38:08 INFO mapred.JobClient: map 100% reduce 100%
> 08/04/30 04:38:08 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_r_000000_3, Status : TIPFAILED
> 08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_3&filter=stdout
> 08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_3&filter=stderr
> 08/04/30 04:38:08 INFO mapred.JobClient: Task Id :
> task_200804300435_0001_m_000000_3, Status : TIPFAILED
> 08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_3&filter=stdout
> 08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
> outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_3&filter=stderr
> java.io.IOException: Job failed!
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
> at org.apache.hadoop.examples.WordCount.run(WordCount.java:147)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:153)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
>
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:49)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>
> The same sequence when tried with an actual cluster setup on my local
> machine works perfectly.
>
> Am I doing something wrong? Should I use the MiniCluster in a different
> way?
> I am using revision 631443, a bit old but this is the one that exists in
> the type branch..
>
> Thanks,
> --Shravan
>
>
>
