[
https://issues.apache.org/jira/browse/OOZIE-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy reopened OOZIE-1527:
---------------------------------------
Yes. If lookup interval was 1 hour and scheduling interval was 5 mins, it will
create lot of actions very early. But I don't think such a setting is even
sane. We are mostly looking at setting scheduling interval to 1 or 2 mins and
lookup to 5-8 mins. Lookup interval above 10 mins can cause more problems.
Noticed another issue. Previously lookup interval was scheduling interval. If
it was set to 2 mins and scheduling interval is not overriden (default 5 mins),
then actions can materialize slowly. Need to make scheduling
interval=min(lookup interval, scheduling interval). Reopening the jira to fix
that.
> Fix scalability issues with coordinator materialization
> -------------------------------------------------------
>
> Key: OOZIE-1527
> URL: https://issues.apache.org/jira/browse/OOZIE-1527
> Project: Oozie
> Issue Type: Bug
> Components: coordinator
> Affects Versions: trunk
> Reporter: Mona Chitnis
> Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1527-V2.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> In certain situations when there is a large number of coordinators in the
> system, they have been observed to create huge backlog in materialization,
> and progressing very slow compared to expected. This patch can be looked upon
> as both a bug-fix or an enhancement addressing following points:
> 1. 'materialization.system.limit' leads to bringing Coord jobs in LRU
> fashion, but some of them may already be maxing out at actions to materialize
> (= throttle), and < limit jobs may actually undergo materialization. This
> patch does a second iteration of loading jobs to get materialized to reduce
> backlog
> 2. 'materialization.window' being 1 hour may work in most cases, but hourly
> jobs are seen to face significant slowdown at times, by lot of other minute
> jobs getting materialized. Therefore, window can be doubled (i.e. 2 hours)
> when job is hourly/daily.
> 3. For hourly coordinators, it is consistently seen that materialization
> occurs only near the end of the hour. e.g. for action whose nominal time is
> 2:00, action creation time is 1:59, if nominal time - 3:00, creation time is
> 2:58 and so on. If window is an hour in the future, doesn't explain why
> materialization won't occur anytime in the middle of the preceding hour.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
