[
https://issues.apache.org/jira/browse/OOZIE-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13988284#comment-13988284
]
Purshotam Shah commented on OOZIE-1818:
---------------------------------------
We noticed this with under utilized logic for CoordMaterializeTriggerService,
where if any coord is rejected and
coord to materialize > materializationLimit, we materialize again.
// account for under-utilization of limit due to jobs maxed out
// against mat_throttle. hence repeat
if (materializeCoordJobs(currDate, materializationLimit, LOG)) {
materializeCoordJobs(currDate, materializationLimit, LOG);
}
If materializationLimit = 50 and there are only 80 jobs to materialize, then
around 20 coord jobs will be queued twice.
With recent fix of OOZIE-1527. We have removed the under-utilization logic, so
this may not happen again.
But we can visualize many scenario where it can happen, like if system/DB is
highly loaded and CoordMaterializeTriggerService runs every 1 or 2 min. We can
have two services running simultaneously or immediately after other
(CoordMaterializeTriggerService runs before CoordMaterializeTransitionXCommand
DB update ), in that case CoordMaterializeTransitionXCommand command may be
queue twice.
> CoordMaterializeTransitionXCommand verifyPrecondition doesn't verify current
> time
> ---------------------------------------------------------------------------------
>
> Key: OOZIE-1818
> URL: https://issues.apache.org/jira/browse/OOZIE-1818
> Project: Oozie
> Issue Type: Bug
> Reporter: Purshotam Shah
>
> {code}
> Timestamp startTime = coordJob.getNextMaterializedTimestamp();
> if (startTime == null) {
> startTime = coordJob.getStartTimestamp();
> if (startTime.after(new Timestamp(System.currentTimeMillis() +
> lookAheadWindow * 1000))) {
> throw new PreconditionException(ErrorCode.E1100,
> "CoordMaterializeTransitionXCommand for jobId="
> + jobId + " job's start time is not reached yet -
> nothing to materialize");
> }
> }
> {code}
> It should be should be
> {code}
> Timestamp matTime = coordJob.getNextMaterializedTimestamp();
> if (matTime == null) {
> matTime = coordJob.getStartTimestamp();
> }
> if (matTime.after(new Timestamp(System.currentTimeMillis() +
> lookAheadWindow * 1000))) {
> throw new PreconditionException(ErrorCode.E1100,
> "CoordMaterializeTransitionXCommand for jobId="
> + jobId + " Request is for future time");
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
