Hi Nam, I had the exact same issue a few weeks ago and asked this. Here is the resoponse I got from Mona Chitnis:
=== What you are looking for here is "oozie.service.CoordMaterializeTriggerService.lookup.interval". Default value = 300 sec or 5 min. This server-level setting governs how often the materialization service is triggered to materialize the next set of actions. This service frequency works in conjunction with another setting - "oozie.service.CoordMaterializeTriggerService.materialization.window". Default value = 3600 sec or 1 hour. What this means is every 5 min, Oozie will materialize actions falling in the next 1 hour window. For your use case, if you wish to materialize actions a) faster than every 5 min and b) spanning beyond 1 hour window at a time, you should consider changing these 2 values. === In my case, all I had to do was update the first value (...lookup.interval) and set it to 60s. Hope that helps, Paul Chavez -----Original Message----- From: Nam Pham [mailto:[email protected]] Sent: Wednesday, August 21, 2013 1:53 AM To: [email protected] Subject: Job creation rate Hello, I am new to this mailing list and would like to say hi to everybody. I am using oozie 3.3.2-chd4.3.0 to import about a year of data to hive. Data are stored hourly, so the coordinator need to create about 8000 jobs to import all the data. The problem is that oozie coordinator is too slow in creating jobs. It only creates about 300 jobs per day with my setup. I tried to increase the concurrency parameter of the coordinator job but it does not help. Is there a knob anywhere making the coordinator create jobs faster, preferably multiple jobs at once since those can be executed in parallel ? Ideas and suggestions are much appreciated. Regards, Nam
