Thank you, Mona. The first setting worked for my current use case as I just reduced it to 60s.
However, I'm still curious about the second setting and the triggering logic. You say 'in the next hour' but how is that applied when running post-dated coordinators? If interpreted to mean 'up to an hour from the current trigger time' then all past actions should be available to materialize at the first trigger interval after submission, and as each trigger interval passes this seems to be true except only one action is materialized at a time. I would think this is related to concurrency but that seems to affect how many jobs are in the RUNNING state at once, not a limit on how many actions can be materialized at once. Can you explain a little more what prevents the coordinator from just materializing all available actions in a single interval? Thanks, Paul -----Original Message----- From: Mona Chitnis [mailto:[email protected]] Sent: Tuesday, July 30, 2013 10:47 AM To: [email protected] Subject: Re: Reduce delay between coordinator action materializations Hi Paul, What you are looking for here is "oozie.service.CoordMaterializeTriggerService.lookup.interval". Default value = 300 sec or 5 min. This server-level setting governs how often the materialization service is triggered to materialize the next set of actions. This service frequency works in conjunction with another setting - "oozie.service.CoordMaterializeTriggerService.materialization.window". Default value = 3600 sec or 1 hour. What this means is every 5 min, Oozie will materialize actions falling in the next 1 hour window. For your use case, if you wish to materialize actions a) faster than every 5 min and b) spanning beyond 1 hour window at a time, you should consider changing these 2 values. -- Mona On 7/30/13 10:38 AM, "Paul Chavez" <[email protected]> wrote: >I have noticed when setting up coordinators which will materialize >dozens of actions right away there is a delay of about 5 minutes before >each action is created. > >This is fine when the actions will take longer than 5 minutes on >average but sometimes the action only takes a minute or two so the >coordinator just sits there with no actions in the calendar for no apparent >reason. > >The specific scenario I have now is reprocessing a few months worth of >logs a day at a time and the coordinator has been submitted to process >the past 60 days. Each action executes in about 150 seconds so another >couple minutes are spent idle between each action. > >Would bumping up the concurrency help? If concurrency=2 would it >materialize two actions per interval? > >thanks, >Paul Chavez
