[
https://issues.apache.org/jira/browse/OOZIE-2175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14369891#comment-14369891
]
Purshotam Shah commented on OOZIE-2175:
---------------------------------------
{quote}
Robert Kanter, do you recall why we did it this way? Is it a valid bug?
{quote}
It is a valid bug, we should never create more actions then mat throttle, in
one materialization cycle.
This can cause multiple issues, queueing lots of inputcheck command,
registering lot of SLA operation which can cause SLA queue to full.
Lots of inputcheck command means oozie will be mostly doing input check and
hence other command will be reject ( as queue might be full).
This can bring down the system, unless we go and remove all those entries from
DB manually.
And if we are creating, we need to make sure that they are not queued. we can
synchronously call some command to change coord action status to SKIPPED or
while creating action itself we can change the coord action status and not
queue any input-check command ( this may still cause SLA queue issue).
Note : we don't have check on mat throttle, user can still specify large
throttle value, which can cause system to go down.
> Coord job with execution order LAST_ONLY/NONE can create enormous number of
> coord actions
> -----------------------------------------------------------------------------------------
>
> Key: OOZIE-2175
> URL: https://issues.apache.org/jira/browse/OOZIE-2175
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
>
> Coord job with execution order LAST_ONLY/NONE doesn't respect Mat Throttling
> and can create enormous number of coord action.
> If starttime is 2013 and current time is 2105 and job frequency is hourly,
> CoordMaterializeTransitionXCommand can create coord actions >10K in one
> materialization cycle.
> {code}
> // If LAST_ONLY and all materialization is in the past, ignore
> maxActionsToBeCreated
> boolean ignoreMaxActions =
>
> (coordJob.getExecutionOrder().equals(CoordinatorJob.Execution.LAST_ONLY) ||
>
> coordJob.getExecutionOrder().equals(CoordinatorJob.Execution.NONE))
> && endMatdTime.before(new Date());
> while (effStart.compareTo(end) < 0 && (ignoreMaxActions ||
> maxActionToBeCreated-- > 0)) {
> if (pause != null && effStart.compareTo(pause) >= 0) {
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
