Jim, Think like a web crawler.
You have a list of catalogs you already know about and you can create a list of catalogs as you discover them. Then you can merge your old list with your new list with suitable old/new marks and only keep new ones that do not have an old version. When you run out of new ones, you are done. You will have to do some file copying so that you have your new list where you want it and have all of your old generations handy as well, but it isn't hard. The outer loop of your program will be, as people have pointed out, in the main program. On 10/29/07 2:17 PM, "Jim the Standing Bear" <[EMAIL PROTECTED]> wrote: > Hi Lohit, > > Thank you for the response. If jobs cannot be recursively launched, > does it mean that I have to pre-process all the catalogs and putting > only filenames into a sequence file and then use hadoop from that > point on? > > -- Jim > > On 10/29/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: >>> Can I recursively launch a job within the map() method of a mapper? >> >> Interesting. If your launch is blocking, then I am thinking possibility of >> deadlock. Lets say you have N nodes and your initial map tasks are scheduled >> on all N nodes. Now if each of the map tasks launch few more jobs, then the >> new ones are sitting there waiting for free nodes. >> >> Thanks, >> Lohit >> ----- Original Message ---- >> From: Jim the Standing Bear <[EMAIL PROTECTED]> >> To: [email protected] >> Sent: Monday, October 29, 2007 1:28:17 PM >> Subject: can jobs be launched recursively within a mapper ? >> >> >> Hi, >> >> Can I recursively launch a job within the map() method of a mapper? >> The task i am facing involves extracting metadata for filenames and >> catalog names from hierarchical catalogs, and for each catalog name >> extract, I must recursively do the same thing until no more catalog >> names present. Therefore, it will be convenient if I can launch a job >> within the map(), but I want to make certain that it will work before >> investing too much efforts into the coding. >> >> If it won't work, could you suggest the proper way to go about doing >> so? Much appreciated in advance. >> >> -- Jim >> >> >> >> >> __________________________________________________ >> Do You Yahoo!? >> Tired of spam? Yahoo! Mail has the best spam protection around >> http://mail.yahoo.com >> >
