[
https://issues.apache.org/jira/browse/OOZIE-1527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohini Palaniswamy resolved OOZIE-1527.
---------------------------------------
Resolution: Fixed
Committed the additional fix OOZIE-1527-V3.patch to trunk. Thanks Puru.
> Fix scalability issues with coordinator materialization
> -------------------------------------------------------
>
> Key: OOZIE-1527
> URL: https://issues.apache.org/jira/browse/OOZIE-1527
> Project: Oozie
> Issue Type: Bug
> Components: coordinator
> Affects Versions: trunk
> Reporter: Mona Chitnis
> Assignee: Purshotam Shah
> Fix For: trunk
>
> Attachments: OOZIE-1527-V2.patch, OOZIE-1527-V3.patch
>
> Original Estimate: 96h
> Remaining Estimate: 96h
>
> In certain situations when there is a large number of coordinators in the
> system, they have been observed to create huge backlog in materialization,
> and progressing very slow compared to expected. This patch can be looked upon
> as both a bug-fix or an enhancement addressing following points:
> 1. 'materialization.system.limit' leads to bringing Coord jobs in LRU
> fashion, but some of them may already be maxing out at actions to materialize
> (= throttle), and < limit jobs may actually undergo materialization. This
> patch does a second iteration of loading jobs to get materialized to reduce
> backlog
> 2. 'materialization.window' being 1 hour may work in most cases, but hourly
> jobs are seen to face significant slowdown at times, by lot of other minute
> jobs getting materialized. Therefore, window can be doubled (i.e. 2 hours)
> when job is hourly/daily.
> 3. For hourly coordinators, it is consistently seen that materialization
> occurs only near the end of the hour. e.g. for action whose nominal time is
> 2:00, action creation time is 1:59, if nominal time - 3:00, creation time is
> 2:58 and so on. If window is an hour in the future, doesn't explain why
> materialization won't occur anytime in the middle of the preceding hour.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
