HadoopStreaming looks really cool, and I'm trying it for the first time. I'm obviously doing something wrong, but I have no clue what.
I made a goofy little wordcount mapper and reducer in Python, and I'm running HadoopStreaming with the following alias: alias hadoop-streaming='/home/amcnabb/hadoop/bin/hadoop jar /home/amcnabb/hadoop/build/hadoop-streaming.jar' Here is the job I ran, after doing "hadoop dfs -put kjv kjv". From the output, it is clear that: "Job not Successful!" However, I have no idea of what's causing the problem. Am I messing up something obvious? Where should I look to see what's really happening? Thanks. [EMAIL PROTECTED]:~/svn/mrpso/python% hadoop-streaming -mapper "/usr/bin/python /home/amcnabb/svn/mrpso/python/mapper.py" -reducer "/usr/bin/python /home/amcnabb/svn/mrpso/python/reducer.py" -input kjv -output kjvout 06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml 06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/mapred-default.xml 06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml 06/10/17 14:09:11 INFO ipc.Client: org.apache.hadoop.io.ObjectWritable ConnectionCuller maxidletime=1000ms: starting packageJobJar: [/tmp/hadoop-unjar29550] [] /tmp/streamjob29551.jar 06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-default.xml 06/10/17 14:09:11 INFO conf.Configuration: parsing file:/home/amcnabb/hadoop-0.6.2/conf/hadoop-site.xml 06/10/17 14:09:11 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-amcnabb/mapred/local] 06/10/17 14:09:11 INFO streaming.StreamJob: Running job: job_0013 06/10/17 14:09:11 INFO streaming.StreamJob: To kill this job, run: 06/10/17 14:09:11 INFO streaming.StreamJob: /home/amcnabb/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=prodigy:50006 -kill job_0013 06/10/17 14:09:11 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013 06/10/17 14:09:12 INFO streaming.StreamJob: map 100% reduce 100% 06/10/17 14:09:12 INFO streaming.StreamJob: To kill this job, run: 06/10/17 14:09:12 INFO streaming.StreamJob: /home/amcnabb/hadoop/bin/../bin/hadoop job -Dmapred.job.tracker=prodigy:50006 -kill job_0013 06/10/17 14:09:12 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_0013 06/10/17 14:09:12 INFO streaming.StreamJob: killJob... Exception in thread "main" java.io.IOException: Job not Successful! at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:558) at org.apache.hadoop.streaming.StreamJob.go(StreamJob.java:63) at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:29) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.hadoop.util.RunJar.main(RunJar.java:137) [EMAIL PROTECTED]:~/svn/mrpso/python% -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
signature.asc
Description: Digital signature
