Thank you Paul, much appreciated. I will try this and let you know the result.
Bests, Nam On Aug 21, 2013, at 10:06 PM, Paul Chavez <[email protected]> wrote: > Hi Nam, > > I had the exact same issue a few weeks ago and asked this. Here is the > resoponse I got from Mona Chitnis: > > === > What you are looking for here is > "oozie.service.CoordMaterializeTriggerService.lookup.interval". Default value > = 300 sec or 5 min. This server-level setting governs how often the > materialization service is triggered to materialize the next set of actions. > > This service frequency works in conjunction with another setting - > "oozie.service.CoordMaterializeTriggerService.materialization.window". > Default value = 3600 sec or 1 hour. What this means is every 5 min, Oozie > will materialize actions falling in the next 1 hour window. > > For your use case, if you wish to materialize actions a) faster than every > 5 min and b) spanning beyond 1 hour window at a time, you should consider > changing these 2 values. > === > > In my case, all I had to do was update the first value (...lookup.interval) > and set it to 60s. > > Hope that helps, > Paul Chavez > > > -----Original Message----- > From: Nam Pham [mailto:[email protected]] > Sent: Wednesday, August 21, 2013 1:53 AM > To: [email protected] > Subject: Job creation rate > > Hello, > > I am new to this mailing list and would like to say hi to everybody. > > I am using oozie 3.3.2-chd4.3.0 to import about a year of data to hive. > Data are stored hourly, so the coordinator need to create about 8000 jobs to > import all the data. > > The problem is that oozie coordinator is too slow in creating jobs. It only > creates about 300 jobs per day with my setup. I tried to increase the > concurrency parameter of the coordinator job but it does not help. > > Is there a knob anywhere making the coordinator create jobs faster, > preferably multiple jobs at once since those can be executed in parallel ? > > Ideas and suggestions are much appreciated. > > Regards, > Nam
