Hi Sandeep,
One important thing here is, if you are passing your jar with
hadoop jar your.jar then it is absolutely NOT required to pass it once
again with -libjars option.
As Harsh said do recompile once again and see running your
command without using -libjars option.
Also you can try your luck by running the JOB in old and new
versions.
Best,
Mahesh Balija,
Calsoft Labs.
On Fri, Nov 30, 2012 at 2:16 AM, Sandeep Jangra <[email protected]>wrote:
> Hi Harsh,
>
> I tried putting the generic option first, but it throws exception file
> not found.
> The jar is in current directory. Then I tried giving absolute path of
> this jar, but that also brought no luck.
>
> sudo -u hdfs hadoop jar word_cnt.jar WordCount2 -libjars=word_cnt.jar
> /tmp/root/input /tmp/root/output17
> Exception in thread "main" java.io.FileNotFoundException: File
> word_cnt.jar does not exist.
> at
> org.apache.hadoop.util.GenericOptionsParser.validateFiles(GenericOptionsParser.java:384)
> at
> org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:280)
> at
> org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:418)
> at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:168)
> at
> org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:151)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
>
> Also, I have been deleting my jars and the class directory before each
> new try. So even I am suspicious why do I see this:
> "12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set.
> User classes may not be found. See JobConf(Class)
> or JobConf#setJar(String)."
>
> Could it be that my hadoop is running on old jar files (the one with
> package name "mapred" (not mapreduce))
> But my program is using new jars as well.
>
> I can try going back to old word count example on the apache site and
> using old jars.
>
> Any other pointers would be highly appreciated. Thanks
>
>
>
> On Thu, Nov 29, 2012 at 2:42 PM, Harsh J <[email protected]> wrote:
>
>> I think you may have not recompiled your application properly.
>>
>> Your runtime shows this:
>>
>> 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set. User
>> classes may not be found. See JobConf(Class) or
>> JobConf#setJar(String).
>>
>> Which should not appear, cause your code has this (which I suspect you
>> may have added later, accidentally?):
>>
>> job.setJarByClass(WordCount2.class);
>>
>> So if you can try deleting the older jar and recompiling it, the
>> problem would go away.
>>
>> Also, when passing generic options such as -libjars, etc., they need
>> to go first in order. I mean, it should always be [Classname] [Generic
>> Options] [Application Options]. Otherwise, they may not get utilized
>> properly.
>>
>> On Fri, Nov 30, 2012 at 12:51 AM, Sandeep Jangra
>> <[email protected]> wrote:
>> > Yups I can see my class files there.
>> >
>> >
>> > On Thu, Nov 29, 2012 at 2:13 PM, Kartashov, Andy <
>> [email protected]>
>> > wrote:
>> >>
>> >> Can you try running jar –tvf word_cnt.jar and see if your static nested
>> >> classes WordCount2$Map.class and WordCount2$Reduce.class have actually
>> been
>> >> added to the jar.
>> >>
>> >>
>> >>
>> >> Rgds,
>> >>
>> >> AK47
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> From: Sandeep Jangra [mailto:[email protected]]
>> >> Sent: Thursday, November 29, 2012 1:36 PM
>> >> To: [email protected]
>> >> Subject: Re: Trouble with Word Count example
>> >>
>> >>
>> >>
>> >> Also, I did set the HADOOP_CLASSPATH variable to point to the
>> word_cnt.jar
>> >> only.
>> >>
>> >>
>> >>
>> >> On Thu, Nov 29, 2012 at 10:54 AM, Sandeep Jangra <
>> [email protected]>
>> >> wrote:
>> >>
>> >> Thanks for the quick response Mahesh.
>> >>
>> >>
>> >>
>> >> I am using the following command:
>> >>
>> >>
>> >>
>> >> sudo -u hdfs hadoop jar word_cnt.jar WordCount2 /tmp/root/input
>> >> /tmp/root/output15 -libjars=word_cnt.jar
>> >>
>> >> (The input directory exists on the hdfs)
>> >>
>> >>
>> >>
>> >> This is how I compiled and packaged it:
>> >>
>> >>
>> >>
>> >> javac -classpath
>> >> /usr/lib/hadoop-0.20-mapreduce/hadoop-core.jar:/usr/lib/hadoop/* -d
>> >> word_cnt WordCount2.java
>> >>
>> >> jar -cvf word_cnt.jar -C word_cnt/ .
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, Nov 29, 2012 at 10:46 AM, Mahesh Balija
>> >> <[email protected]> wrote:
>> >>
>> >> Hi Sandeep,
>> >>
>> >>
>> >>
>> >> For me everything seems to be alright.
>> >>
>> >> Can you tell us how are you running this job?
>> >>
>> >>
>> >>
>> >> Best,
>> >>
>> >> Mahesh.B.
>> >>
>> >> Calsoft Labs.
>> >>
>> >> On Thu, Nov 29, 2012 at 9:01 PM, Sandeep Jangra <
>> [email protected]>
>> >> wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >>
>> >>
>> >> Like most others I am also running into some problems while running
>> my
>> >> word count example.
>> >>
>> >> I tried the various suggestion available on internet, but I guess
>> it;s
>> >> time to go on email :)
>> >>
>> >>
>> >>
>> >> Here is the error that I am getting:
>> >>
>> >> 12/11/29 10:20:59 WARN mapred.JobClient: Use GenericOptionsParser for
>> >> parsing the arguments. Applications should implement Tool for the same.
>> >>
>> >> 12/11/29 10:20:59 WARN mapred.JobClient: No job jar file set. User
>> >> classes may not be found. See JobConf(Class) or JobConf#setJar(String).
>> >>
>> >> 12/11/29 10:20:59 INFO input.FileInputFormat: Total input paths to
>> process
>> >> : 1
>> >>
>> >> 12/11/29 10:20:59 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> >> library
>> >>
>> >> 12/11/29 10:20:59 WARN snappy.LoadSnappy: Snappy native library is
>> >> available
>> >>
>> >> 12/11/29 10:20:59 INFO snappy.LoadSnappy: Snappy native library loaded
>> >>
>> >> 12/11/29 10:21:00 INFO mapred.JobClient: Running job:
>> >> job_201210310210_0040
>> >>
>> >> 12/11/29 10:21:01 INFO mapred.JobClient: map 0% reduce 0%
>> >>
>> >> 12/11/29 10:21:07 INFO mapred.JobClient: Task Id :
>> >> attempt_201210310210_0040_m_000000_0, Status : FAILED
>> >>
>> >> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
>> >> WordCount2$Map not found
>> >>
>> >> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1439)
>> >>
>> >> at
>> >>
>> org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
>> >>
>> >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
>> >>
>> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>> >>
>> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
>> >>
>> >> at java.security.AccessController.doPrivileged(Native Method)
>> >>
>> >> at javax.security.auth.Subject.doAs(Subject.java:416)
>> >>
>> >> at
>> >>
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>> >>
>> >> at org.apache.hadoop.mapred.Child.main(Child.java:264)
>> >>
>> >> Caused by: java.lang.ClassNotFoundException: Class WordCount2$Map not
>> >> found
>> >>
>> >> at
>> >>
>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1350)
>> >>
>> >> at
>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1437)
>> >>
>> >> ... 8 more
>> >>
>> >>
>> >>
>> >> And here is the source code:
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> import org.apache.hadoop.conf.Configuration;
>> >>
>> >> import org.apache.hadoop.conf.Configured;
>> >>
>> >> import org.apache.hadoop.fs.Path;
>> >>
>> >> import org.apache.hadoop.io.IntWritable;
>> >>
>> >> import org.apache.hadoop.io.LongWritable;
>> >>
>> >> import org.apache.hadoop.io.Text;
>> >>
>> >> import org.apache.hadoop.mapreduce.Job;
>> >>
>> >> import org.apache.hadoop.mapreduce.Mapper;
>> >>
>> >> import org.apache.hadoop.mapreduce.Reducer;
>> >>
>> >> import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
>> >>
>> >> import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
>> >>
>> >> import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
>> >>
>> >> import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
>> >>
>> >> import org.apache.hadoop.util.Tool;
>> >>
>> >> import org.apache.hadoop.util.ToolRunner;
>> >>
>> >>
>> >>
>> >> import java.io.IOException;
>> >>
>> >> import java.util.StringTokenizer;
>> >>
>> >>
>> >>
>> >> public class WordCount2 extends Configured implements Tool {
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> public static class Map extends Mapper<LongWritable, Text, Text,
>> >> IntWritable> {
>> >>
>> >> private final static IntWritable one = new IntWritable(1);
>> >>
>> >> private Text word = new Text();
>> >>
>> >>
>> >>
>> >> @Override
>> >>
>> >> protected void map(LongWritable key, Text value, Context
>> context)
>> >> throws IOException, InterruptedException {
>> >>
>> >> String line = value.toString();
>> >>
>> >> StringTokenizer tokenizer = new StringTokenizer(line);
>> >>
>> >> while (tokenizer.hasMoreTokens()) {
>> >>
>> >> word.set(tokenizer.nextToken());
>> >>
>> >> context.write(word, one);
>> >>
>> >> }
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> public static class Reduce extends Reducer<Text, IntWritable, Text,
>> >> IntWritable> {
>> >>
>> >>
>> >>
>> >> @Override
>> >>
>> >> protected void reduce(Text key, Iterable<IntWritable> values,
>> >> Context context) throws IOException, InterruptedException {
>> >>
>> >>
>> >>
>> >> int sum = 0;
>> >>
>> >>
>> >>
>> >> for(IntWritable value : values) {
>> >>
>> >> sum += value.get();
>> >>
>> >> }
>> >>
>> >> // while (values.hasNext()) {
>> >>
>> >> // sum += values.next().get();
>> >>
>> >> // }
>> >>
>> >> context.write(key, new IntWritable(sum));
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >> @Override
>> >>
>> >> public int run(String[] args) throws Exception {
>> >>
>> >> Configuration conf = getConf();
>> >>
>> >> for (java.util.Map.Entry<String, String> entry: conf) {
>> >>
>> >> System.out.printf("%s=%s\n", entry.getKey(),
>> >> entry.getValue());
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >> System.out.println("arg[0]= "+args[0] + " args[1]= "+ args[1]);
>> >>
>> >>
>> >>
>> >> Job job = new Job(conf, WordCount2.class.getSimpleName());
>> >>
>> >> job.setJobName("wordcount2");
>> >>
>> >> job.setJarByClass(WordCount2.class);
>> >>
>> >>
>> >>
>> >> job.setMapOutputKeyClass(Text.class);
>> >>
>> >> job.setMapOutputValueClass(IntWritable.class);
>> >>
>> >>
>> >>
>> >> job.setOutputKeyClass(Text.class);
>> >>
>> >> job.setOutputValueClass(IntWritable.class);
>> >>
>> >>
>> >>
>> >> job.setMapperClass(Map.class);
>> >>
>> >> job.setCombinerClass(Reduce.class);
>> >>
>> >> job.setReducerClass(Reduce.class);
>> >>
>> >>
>> >>
>> >> job.setInputFormatClass(TextInputFormat.class);
>> >>
>> >> job.setOutputFormatClass(TextOutputFormat.class);
>> >>
>> >>
>> >>
>> >> FileInputFormat.setInputPaths(job, new Path(args[0]));
>> >>
>> >> FileOutputFormat.setOutputPath(job, new Path(args[1]));
>> >>
>> >>
>> >>
>> >> System.exit(job.waitForCompletion(true) ? 0 : 1);
>> >>
>> >>
>> >>
>> >> return 0;
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> public static void main(String[] args) throws Exception {
>> >>
>> >> int exitCode = ToolRunner.run(new WordCount2(), args);
>> >>
>> >> System.exit(exitCode);
>> >>
>> >> }
>> >>
>> >> }
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> NOTICE: This e-mail message and any attachments are confidential,
>> subject
>> >> to copyright and may be privileged. Any unauthorized use, copying or
>> >> disclosure is prohibited. If you are not the intended recipient, please
>> >> delete and contact the sender immediately. Please consider the
>> environment
>> >> before printing this e-mail. AVIS : le présent courriel et toute pièce
>> >> jointe qui l'accompagne sont confidentiels, protégés par le droit
>> d'auteur
>> >> et peuvent être couverts par le secret professionnel. Toute
>> utilisation,
>> >> copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le
>> >> destinataire prévu de ce courriel, supprimez-le et contactez
>> immédiatement
>> >> l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le
>> présent
>> >> courriel
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>
