Maybe you stepped out of your working directory. "$ ls -l" Do you see your .jar?
From: Sandeep Jangra [mailto:[email protected]] Sent: Thursday, November 29, 2012 3:46 PM To: [email protected] Subject: Re: Trouble with Word Count example Hi Harsh, I tried putting the generic option first, but it throws exception file not found. The jar is in current directory. Then I tried giving absolute path of this jar, but that also brought no luck. sudo -u hdfs hadoop jar word_cnt.jar WordCount2 -libjars=word_cnt.jar /tmp/root/input /tmp/root/output17 Exception in thread "main" java.io.FileNotFoundException: File word_cnt.jar does not exist. at org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:384) at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:280) at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:418) at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:168) at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:151) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64) Also, I have been deleting my jars and the class directory before each new try. So even I am suspicious why do I see this: "12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String)." Could it be that my hadoop is running on old jar files (the one with package name "mapred" (not mapreduce)) But my program is using new jars as well. I can try going back to old word count example on the apache site and using old jars. Any other pointers would be highly appreciated. Thanks On Thu, Nov 29, 2012 at 2:42 PM, Harsh J <[email protected]<mailto:[email protected]>> wrote: I think you may have not recompiled your application properly. Your runtime shows this: 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). Which should not appear, cause your code has this (which I suspect you may have added later, accidentally?): job.setJarByClass(WordCount2.class); So if you can try deleting the older jar and recompiling it, the problem would go away. Also, when passing generic options such as -libjars, etc., they need to go first in order. I mean, it should always be [Classname] [Generic Options] [Application Options]. Otherwise, they may not get utilized properly. On Fri, Nov 30, 2012 at 12:51 AM, Sandeep Jangra <[email protected]<mailto:[email protected]>> wrote: > Yups I can see my class files there. > > > On Thu, Nov 29, 2012 at 2:13 PM, Kartashov, Andy > <[email protected]<mailto:[email protected]>> > wrote: >> >> Can you try running jar -tvf word_cnt.jar and see if your static nested >> classes WordCount2$Map.class and WordCount2$Reduce.class have actually been >> added to the jar. >> >> >> >> Rgds, >> >> AK47 >> >> >> >> >> >> From: Sandeep Jangra >> [mailto:[email protected]<mailto:[email protected]>] >> Sent: Thursday, November 29, 2012 1:36 PM >> To: [email protected]<mailto:[email protected]> >> Subject: Re: Trouble with Word Count example >> >> >> >> Also, I did set the HADOOP_CLASSPATH variable to point to the word_cnt.jar >> only. >> >> >> >> On Thu, Nov 29, 2012 at 10:54 AM, Sandeep Jangra >> <[email protected]<mailto:[email protected]>> >> wrote: >> >> Thanks for the quick response Mahesh. >> >> >> >> I am using the following command: >> >> >> >> sudo -u hdfs hadoop jar word_cnt.jar WordCount2 /tmp/root/input >> /tmp/root/output15 -libjars=word_cnt.jar >> >> (The input directory exists on the hdfs) >> >> >> >> This is how I compiled and packaged it: >> >> >> >> javac -classpath >> /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar:/usr/lib/hadoop/* -d >> word_cnt WordCount2.java >> >> jar -cvf word_cnt.jar -C word_cnt/ . >> >> >> >> >> >> >> >> On Thu, Nov 29, 2012 at 10:46 AM, Mahesh Balija >> <[email protected]<mailto:[email protected]>> wrote: >> >> Hi Sandeep, >> >> >> >> For me everything seems to be alright. >> >> Can you tell us how are you running this job? >> >> >> >> Best, >> >> Mahesh.B. >> >> Calsoft Labs. >> >> On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra >> <[email protected]<mailto:[email protected]>> >> wrote: >> >> Hello everyone, >> >> >> >> Like most others I am also running into some problems while running my >> word count example. >> >> I tried the various suggestion available on internet, but I guess it;s >> time to go on email :) >> >> >> >> Here is the error that I am getting: >> >> 12/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParser for >> parsing the arguments. Applications should implement Tool for the same. >> >> 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set. User >> classes may not be found. See JobConf(Class) or JobConf#setJar(String). >> >> 12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to process >> : 1 >> >> 12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop >> library >> >> 12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is >> available >> >> 12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded >> >> 12/11/29 10:21:00 INFO mapred.JobClient: Running job: >> job_201210310210_0040 >> >> 12/11/29 10:21:01 INFO mapred.JobClient: map 0% reduce 0% >> >> 12/11/29 10:21:07 INFO mapred.JobClient: Task Id : >> attempt_201210310210_0040_m_000000_0, Status : FAILED >> >> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class >> WordCount2$Map not found >> >> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1439) >> >> at >> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191) >> >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605) >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) >> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:270) >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> at javax.security.auth.Subject.doAs(Subject.java:416) >> >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) >> >> at org.apache.hadoop.mapred.Child.main(Child.java:264) >> >> Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not >> found >> >> at >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1350) >> >> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1437) >> >> ... 8 more >> >> >> >> And here is the source code: >> >> >> >> >> >> import org.apache.hadoop.conf.Configuration; >> >> import org.apache.hadoop.conf.Configured; >> >> import org.apache.hadoop.fs.Path; >> >> import org.apache.hadoop.io.IntWritable; >> >> import org.apache.hadoop.io.LongWritable; >> >> import org.apache.hadoop.io.Text; >> >> import org.apache.hadoop.mapreduce.Job; >> >> import org.apache.hadoop.mapreduce.Mapper; >> >> import org.apache.hadoop.mapreduce.Reducer; >> >> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; >> >> import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; >> >> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; >> >> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; >> >> import org.apache.hadoop.util.Tool; >> >> import org.apache.hadoop.util.ToolRunner; >> >> >> >> import java.io.IOException; >> >> import java.util.StringTokenizer; >> >> >> >> public class WordCount2 extends Configured implements Tool { >> >> >> >> >> >> public static class Map extends Mapper<LongWritable, Text, Text, >> IntWritable> { >> >> private final static IntWritable one = new IntWritable(1); >> >> private Text word = new Text(); >> >> >> >> @Override >> >> protected void map(LongWritable key, Text value, Context context) >> throws IOException, InterruptedException { >> >> String line = value.toString(); >> >> StringTokenizer tokenizer = new StringTokenizer(line); >> >> while (tokenizer.hasMoreTokens()) { >> >> word.set(tokenizer.nextToken()); >> >> context.write(word, one); >> >> } >> >> } >> >> >> >> } >> >> >> >> >> >> public static class Reduce extends Reducer<Text, IntWritable, Text, >> IntWritable> { >> >> >> >> @Override >> >> protected void reduce(Text key, Iterable<IntWritable> values, >> Context context) throws IOException, InterruptedException { >> >> >> >> int sum = 0; >> >> >> >> for(IntWritable value : values) { >> >> sum += value.get(); >> >> } >> >> // while (values.hasNext()) { >> >> // sum += values.next().get(); >> >> // } >> >> context.write(key, new IntWritable(sum)); >> >> } >> >> >> >> } >> >> >> >> @Override >> >> public int run(String[] args) throws Exception { >> >> Configuration conf = getConf(); >> >> for (java.util.Map.Entry<String, String> entry: conf) { >> >> System.out.printf("%s=%s\n", entry.getKey(), >> entry.getValue()); >> >> } >> >> >> >> System.out.println("arg[0]= "+args[0] + " args[1]= "+ args[1]); >> >> >> >> Job job = new Job(conf, WordCount2.class.getSimpleName()); >> >> job.setJobName("wordcount2"); >> >> job.setJarByClass(WordCount2.class); >> >> >> >> job.setMapOutputKeyClass(Text.class); >> >> job.setMapOutputValueClass(IntWritable.class); >> >> >> >> job.setOutputKeyClass(Text.class); >> >> job.setOutputValueClass(IntWritable.class); >> >> >> >> job.setMapperClass(Map.class); >> >> job.setCombinerClass(Reduce.class); >> >> job.setReducerClass(Reduce.class); >> >> >> >> job.setInputFormatClass(TextInputFormat.class); >> >> job.setOutputFormatClass(TextOutputFormat.class); >> >> >> >> FileInputFormat.setInputPaths(job, new Path(args[0])); >> >> FileOutputFormat.setOutputPath(job, new Path(args[1])); >> >> >> >> System.exit(job.waitForCompletion(true) ? 0 : 1); >> >> >> >> return 0; >> >> } >> >> >> >> >> >> >> >> public static void main(String[] args) throws Exception { >> >> int exitCode = ToolRunner.run(new WordCount2(), args); >> >> System.exit(exitCode); >> >> } >> >> } >> >> >> >> >> >> >> >> >> >> >> >> NOTICE: This e-mail message and any attachments are confidential, subject >> to copyright and may be privileged. Any unauthorized use, copying or >> disclosure is prohibited. If you are not the intended recipient, please >> delete and contact the sender immediately. Please consider the environment >> before printing this e-mail. AVIS : le présent courriel et toute pièce >> jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur >> et peuvent être couverts par le secret professionnel. Toute utilisation, >> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le >> destinataire prévu de ce courriel, supprimez-le et contactez immédiatement >> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent >> courriel > > -- Harsh J NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
