I tried out by setting an enum to count no. of lines in output file from
job3.
But I am getting
14/06/12 10:12:30 INFO mapred.JobClient: Total committed heap usage
(bytes)=1238630400
conf3
Exception in thread "main" java.lang.IllegalStateException: Job in state
DEFINE instead of RUNNING
at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:116)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:491)
Below is my current code
*static enum UpdateCounter {*
* INCOMING_ATTR*
* }*
*public static void main(String[] args) throws Exception {*
* Configuration conf = new Configuration();*
* int res = ToolRunner.run(conf, new Driver(), args);*
* System.exit(res);*
*}*
*@Override*
*public int run(String[] args) throws Exception {*
*while(counter >= 0){*
* Configuration conf = getConf();*
* /**
* * Job 1: *
* */*
* Job job1 = new Job(conf, "");*
* //other configuration*
* job1.setMapperClass(ID3ClsLabelMapper.class);*
* job1.setReducerClass(ID3ClsLabelReducer.class);*
* Path in = new Path(args[0]);*
* Path out1 = new Path(CL);*
* if(counter == 0){*
* FileInputFormat.addInputPath(job1, in);*
* }*
* else{*
* FileInputFormat.addInputPath(job1, out5); *
* }*
* FileInputFormat.addInputPath(job1, in);*
* FileOutputFormat.setOutputPath(job1,out1);*
* job1.waitForCompletion(true);*
* /**
* * Job 2: *
* * *
* */*
* Configuration conf2 = getConf();*
* Job job2 = new Job(conf2, "");*
* Path out2 = new Path(ANC);*
* FileInputFormat.addInputPath(job2, in);*
* FileOutputFormat.setOutputPath(job2,out2);*
* job2.waitForCompletion(true);*
* /**
* * Job3*
* */*
* Configuration conf3 = getConf();*
* Job job3 = new Job(conf3, "");*
* System.out.println("conf3");*
* Path out5 = new Path(args[1]);*
* if(fs.exists(out5)){*
* fs.delete(out5, true);*
* }*
* FileInputFormat.addInputPath(job3,out2);*
* FileOutputFormat.setOutputPath(job3,out5);*
* job3.waitForCompletion(true);*
* FileInputFormat.addInputPath(job3,new Path(args[0]));*
* FileOutputFormat.setOutputPath(job3,out5);*
* job3.waitForCompletion(true);*
* counter =
job3.getCounters().findCounter(UpdateCounter.INCOMING_ATTR).getValue();*
* }*
* return 0;*
Am I doing anything wrong?
On Mon, Jun 9, 2014 at 4:37 PM, Krishna Kumar <[email protected]> wrote:
> You should use FileStatus to decide what files you want to include in the
> InputPath, and use the FileSystem class to delete or process the
> intermediate / final paths. Moving each job in your iteration logic into
> different methods would help keep things simple.
>
>
>
> From: unmesha sreeveni <[email protected]>
> Reply-To: "[email protected]" <[email protected]>
> Date: Monday, June 9, 2014 at 6:02 AM
> To: User Hadoop <[email protected]>
> Subject: Re: Counters in MapReduce
>
> Ok I will check out with counters.
> And after I st iteration the input file to job1 will be the output file of
> job 3.How to give that.
> *Inorder to satisfy 2 conditions*
> First iteration : users input file
> after first iteration :job 3 's output file as job 1 s input.
>
>
>
>> --
>> *Thanks & Regards*
>>
>>
>> *Unmesha Sreeveni U.B*
>> *Hadoop, Bigdata Developer*
>> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
>> http://www.unmeshasreeveni.blogspot.in/
>>
>>
>>
>> ------------------------------
>> *Kai Voigt* Am Germaniahafen 1 [email protected]
>> 24143 Kiel +49 160 96683050
>> Germany @KaiVoigt
>>
>>
>
>
> --
> *Thanks & Regards*
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>
--
*Thanks & Regards *
*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/
