Try changing your command to read hadoop-streaming \ -mapper "/usr/bin/python mapper.py" \ -file "/home/amcnabb/svn/mrpso/python/mapper.py" \ -reducer "/usr/bin/python reducer.py" \ -file "/home/amcnabb/svn/mrpso/python/reducer.py" \ -input kjv \ -output kjvout
I assume kjv is a file and kjvout is a directory - they should be. I also assume /usr/bin/python is the path to python *on the cluster machines*. Otherwise, you can do -mapper "python mapper.py" -file /usr/bin/python -file /home/amcnabb/svn/mrpso/python/mapper.py I recommend adding -jobconf mapred.job.name="kjv", to make the jobtracker history more readable. Yoram > -----Original Message----- > From: Andrew McNabb [mailto:[EMAIL PROTECTED] > Sent: Tuesday, October 17, 2006 1:12 PM > To: [email protected] > Subject: HadoopStreaming > > HadoopStreaming looks really cool, and I'm trying it for the > first time. > I'm obviously doing something wrong, but I have no clue what. > > I made a goofy little wordcount mapper and reducer in Python, and I'm > running HadoopStreaming with the following alias: > > alias hadoop-streaming='/home/amcnabb/hadoop/bin/hadoop jar > /home/amcnabb/hadoop/build/hadoop-streaming.jar' > > Here is the job I ran, after doing "hadoop dfs -put kjv kjv". > From the output, > it is clear that: "Job not Successful!" However, I have no idea of > what's causing the problem. Am I messing up something obvious? Where > should I look to see what's really happening? Thanks. > > [EMAIL PROTECTED]:~/svn/mrpso/python% hadoop-streaming -mapper > "/usr/bin/python /home/amcnabb/svn/mrpso/python/mapper.py" > -reducer "/usr/bin/python > /home/amcnabb/svn/mrpso/python/reducer.py" -input kjv -output kjvout > 06/10/17 14:09:11 INFO conf.Configuration: parsing > file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml > 06/10/17 14:09:11 INFO conf.Configuration: parsing > file:/home/amcnabb/hadoop-0.6.2/conf/mapred-default.xml > 06/10/17 14:09:11 INFO conf.Configuration: parsing > file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml > 06/10/17 14:09:11 INFO ipc.Client: > org.apache.hadoop.io.ObjectWritable ConnectionCuller > maxidletime=1000ms: starting > packageJobJar: [/tmp/hadoop-unjar29550] [] /tmp/streamjob29551.jar > 06/10/17 14:09:11 INFO conf.Configuration: parsing > file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml > 06/10/17 14:09:11 INFO conf.Configuration: parsing > file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml > 06/10/17 14:09:11 INFO streaming.StreamJob: getLocalDirs(): > [/tmp/hadoop-amcnabb/mapred/local] > 06/10/17 14:09:11 INFO streaming.StreamJob: Running job: job_0013 > 06/10/17 14:09:11 INFO streaming.StreamJob: To kill this job, run: > 06/10/17 14:09:11 INFO streaming.StreamJob: > /home/amcnabb/hadoop/bin/../bin/hadoop job > -Dmapred.job.tracker=prodigy:50006 -kill job_0013 > 06/10/17 14:09:11 INFO streaming.StreamJob: Tracking URL: > http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013 > 06/10/17 14:09:12 INFO streaming.StreamJob: map 100% reduce 100% > 06/10/17 14:09:12 INFO streaming.StreamJob: To kill this job, run: > 06/10/17 14:09:12 INFO streaming.StreamJob: > /home/amcnabb/hadoop/bin/../bin/hadoop job > -Dmapred.job.tracker=prodigy:50006 -kill job_0013 > 06/10/17 14:09:12 INFO streaming.StreamJob: Tracking URL: > http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013 > 06/10/17 14:09:12 INFO streaming.StreamJob: killJob... > Exception in thread "main" java.io.IOException: Job not Successful! > at > org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(Stre > amJob.java:558) > at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:63) > at > org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreami > ng.java:29) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess > orImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth > odAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:585) > at org.apache.hadoop.util.RunJar.main(RunJar.java:137) > [EMAIL PROTECTED]:~/svn/mrpso/python% > > -- > Andrew McNabb > http://www.mcnabbs.org/andrew/ > PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868 >
