> Maybe my mind is way off track - but I still sense a > problem with the mapper sending feedbacks to the job controller. That > is, when a mapper has reached the terminal condition, how can it tell > the job controller to stop?
I'm pretty sure that there is no way (within the Hadoop framework) for the mapper to send feedback to the job controller, and there is no way for the mapper to stop itself without the job being marked killed/failed. MapReduce is not really strong for that kind of interactive task. If I'm off base here, I'm sure someone will jump in... Any jobs you run should be allowed to finish properly, and then it should be up to your control code to start another job (or not, if the control decides that you have performed enough iterations). > I tried to find a way so that the job controller can open the file in > the output path at the end of the loop to read the contents; but thus > far, I haven't seen a way to achieve this. Yea, you will need to have your control code read the output from a job at the end of each iteration to decide whether to start another. See the FileSystem API (specifically the .open(Path) method): http://lucene.apache.org/hadoop/api/org/apache/hadoop/fs/FileSystem.html#open(org.apache.hadoop.fs.Path) Definitely not a dead-end... good luck! Thanks, Stu -----Original Message----- From: Jim the Standing Bear <[EMAIL PROTECTED]> Sent: Monday, October 29, 2007 9:57pm To: [email protected], [EMAIL PROTECTED] Subject: Re: can jobs be launched recursively within a mapper ? Thanks, Stu... Maybe my mind is way off track - but I still sense a problem with the mapper sending feedbacks to the job controller. That is, when a mapper has reached the terminal condition, how can it tell the job controller to stop? If I keep a JobConf object in the mapper, and set a property "stop.processing" to true when a mapping task has reached the terminal condition, will it cause synchronization problems? There could be other mapping tasks that still wish to go on? I tried to find a way so that the job controller can open the file in the output path at the end of the loop to read the contents; but thus far, I haven't seen a way to achieve this. Does this mean I have hit a dead-end? -- Jim On 10/29/07, Stu Hood <[EMAIL PROTECTED]> wrote: > The iteration would take place in your control code (your 'main' method, as > shown in the examples). > > In order to prevent records from looping infinitely, each iteration would > need to use a separate output/input directory. > > Thanks, > Stu > > > -----Original Message----- > From: Jim the Standing Bear <[EMAIL PROTECTED]> > Sent: Monday, October 29, 2007 5:45pm > To: [email protected] > Subject: Re: can jobs be launched recursively within a mapper ? > > thanks, Owen and David, > > I also thought of making a queue so that I can push catalog names to > the end of it, while the job control loop keeps removing items off the > queue until there is no more left > > However, the problem is I don't see how I can do so within the > map/reduce context. All the code examples are one-shot deals and > there is no iteration involved. > > Furthermore, what David said made sense, but to avoid infinite loop, > the code must remove the record it just read from the input file. How > do I do that using hadoop's fs? or does hadoop take care of it > automatically? > > -- Jim > > > > On 10/29/07, David Balatero <[EMAIL PROTECTED]> wrote: > > Aren't these questions a little advanced for a bear to be asking? > > I'll be here all night... > > > > But seriously, if your job is inherently recursive, one possible way > > to do it would be to make sure that you output in the same format > > that you input. Then you can keep re-reading the outputted file back > > into a new map/reduce job, until you hit some base case and you > > terminate. I've had a main method before that would kick off a bunch > > of jobs in a row -- but I wouldn't really recommend starting another > > map/reduce job in the scope of a running map() or reduce() method. > > > > - David > > > > > > On Oct 29, 2007, at 2:17 PM, Jim the Standing Bear wrote: > > > > > then > > > > > > > -- > -------------------------------------- > Standing Bear Has Spoken > -------------------------------------- > > > -- -------------------------------------- Standing Bear Has Spoken --------------------------------------
