[jira] [Commented] (OOZIE-3581) Callback does not applied in Oozie server, workflows stuk in RUNNING states.
[
https://issues.apache.org/jira/browse/OOZIE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228640#comment-17228640
]
Junfan Zhang commented on OOZIE-3581:
-
Of course. Please send me your wechat account to my email.
{{junfan.zh...@outlook.com}}. Thanks
> Callback does not applied in Oozie server, workflows stuk in RUNNING states.
>
>
> Key: OOZIE-3581
> URL: https://issues.apache.org/jira/browse/OOZIE-3581
> Project: Oozie
> Issue Type: Bug
> Components: action, workflow
>Affects Versions: 4.3.1
>Reporter: Kotsubinsky Victor
>Priority: Critical
>
> oozie version 4.3.1.3.1.0.0-78
> with HDP3.10 stack , release provides Oozie 4.3.1 and the additional Apache
> patches listed here:
> [https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_oozie.html]
> I use Hadoop kerberized cluster, run on OOZIE, YARN Mr jobs.
> 1. OOzie run mr-job via YARN
> 2. After YARN mr job completed,YYARN mr-job Successfully sent Callback
> request to OOzie,
> 3. in logs OOzie server, i can see this request, but OOZIE does not apply
> this callback request, so in WF-action-id i still see RUNNING state (until
> action.check process check wf-ids and swith action-id to SUCCESS state)
> LOGS in YARN:
> 2020-01-27 12:16:39,749 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification trying
> http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification to
> http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> succeeded
> 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification succeeded for job_1579778851579_31505
>
> Oozie logs about this event:
> 2020-01-27 12:16:39,770 DEBUG CallbackServlet:526 - SERVER[hdp3-oo-2] USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Received a CallbackServlet.doGet() with query string
> id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2]
> USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Execute command [callback] key [null]
> 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2]
> USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Queuing [1] commands with delay [0]ms
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3581) Callback does not applied in Oozie server, workflows stuk in RUNNING states.
[ https://issues.apache.org/jira/browse/OOZIE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228617#comment-17228617 ] gaofeng commented on OOZIE-3581: yes ,i know it ,but i have a problem in the external status's changing do you have wechat? can we talk about it further? > Callback does not applied in Oozie server, workflows stuk in RUNNING states. > > > Key: OOZIE-3581 > URL: https://issues.apache.org/jira/browse/OOZIE-3581 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.3.1 >Reporter: Kotsubinsky Victor >Priority: Critical > > oozie version 4.3.1.3.1.0.0-78 > with HDP3.10 stack , release provides Oozie 4.3.1 and the additional Apache > patches listed here: > [https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_oozie.html] > I use Hadoop kerberized cluster, run on OOZIE, YARN Mr jobs. > 1. OOzie run mr-job via YARN > 2. After YARN mr job completed,YYARN mr-job Successfully sent Callback > request to OOzie, > 3. in logs OOzie server, i can see this request, but OOZIE does not apply > this callback request, so in WF-action-id i still see RUNNING state (until > action.check process check wf-ids and swith action-id to SUCCESS state) > LOGS in YARN: > 2020-01-27 12:16:39,749 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification trying > http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification to > http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > succeeded > 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification succeeded for job_1579778851579_31505 > > Oozie logs about this event: > 2020-01-27 12:16:39,770 DEBUG CallbackServlet:526 - SERVER[hdp3-oo-2] USER[-] > GROUP[-] TOKEN[-] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Received a CallbackServlet.doGet() with query string > id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2] > USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Execute command [callback] key [null] > 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2] > USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Queuing [1] commands with delay [0]ms -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3581) Callback does not applied in Oozie server, workflows stuk in RUNNING states.
[ https://issues.apache.org/jira/browse/OOZIE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228611#comment-17228611 ] Junfan Zhang commented on OOZIE-3581: - Oozie uses two methods to detect the status of tasks, active and passive. Checkerservice is used to detect the status of the task. But callback will notify Oozie, and then trigger the detection. So if there is a delay, it means that the queue is backlogged. > Callback does not applied in Oozie server, workflows stuk in RUNNING states. > > > Key: OOZIE-3581 > URL: https://issues.apache.org/jira/browse/OOZIE-3581 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.3.1 >Reporter: Kotsubinsky Victor >Priority: Critical > > oozie version 4.3.1.3.1.0.0-78 > with HDP3.10 stack , release provides Oozie 4.3.1 and the additional Apache > patches listed here: > [https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_oozie.html] > I use Hadoop kerberized cluster, run on OOZIE, YARN Mr jobs. > 1. OOzie run mr-job via YARN > 2. After YARN mr job completed,YYARN mr-job Successfully sent Callback > request to OOzie, > 3. in logs OOzie server, i can see this request, but OOZIE does not apply > this callback request, so in WF-action-id i still see RUNNING state (until > action.check process check wf-ids and swith action-id to SUCCESS state) > LOGS in YARN: > 2020-01-27 12:16:39,749 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification trying > http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification to > http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > succeeded > 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification succeeded for job_1579778851579_31505 > > Oozie logs about this event: > 2020-01-27 12:16:39,770 DEBUG CallbackServlet:526 - SERVER[hdp3-oo-2] USER[-] > GROUP[-] TOKEN[-] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Received a CallbackServlet.doGet() with query string > id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2] > USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Execute command [callback] key [null] > 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2] > USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Queuing [1] commands with delay [0]ms -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3581) Callback does not applied in Oozie server, workflows stuk in RUNNING states.
[ https://issues.apache.org/jira/browse/OOZIE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228502#comment-17228502 ] Frank Top Frank commented on OOZIE-3581: [~zuston] [~vkotsubinskiy] > Callback does not applied in Oozie server, workflows stuk in RUNNING states. > > > Key: OOZIE-3581 > URL: https://issues.apache.org/jira/browse/OOZIE-3581 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.3.1 >Reporter: Kotsubinsky Victor >Priority: Critical > > oozie version 4.3.1.3.1.0.0-78 > with HDP3.10 stack , release provides Oozie 4.3.1 and the additional Apache > patches listed here: > [https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_oozie.html] > I use Hadoop kerberized cluster, run on OOZIE, YARN Mr jobs. > 1. OOzie run mr-job via YARN > 2. After YARN mr job completed,YYARN mr-job Successfully sent Callback > request to OOzie, > 3. in logs OOzie server, i can see this request, but OOZIE does not apply > this callback request, so in WF-action-id i still see RUNNING state (until > action.check process check wf-ids and swith action-id to SUCCESS state) > LOGS in YARN: > 2020-01-27 12:16:39,749 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification trying > http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification to > http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > succeeded > 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification succeeded for job_1579778851579_31505 > > Oozie logs about this event: > 2020-01-27 12:16:39,770 DEBUG CallbackServlet:526 - SERVER[hdp3-oo-2] USER[-] > GROUP[-] TOKEN[-] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Received a CallbackServlet.doGet() with query string > id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2] > USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Execute command [callback] key [null] > 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2] > USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Queuing [1] commands with delay [0]ms -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3581) Callback does not applied in Oozie server, workflows stuk in RUNNING states.
[
https://issues.apache.org/jira/browse/OOZIE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17228488#comment-17228488
]
Frank Top Frank commented on OOZIE-3581:
Hello, I also ran into the same problem with your production environment under
the same version.Hello, I also ran into the same problem with your production
environment under the same version.Initially, I modified the following
parameters
#oozie.service.ActionCheckerService.action.check.interval ->10
#oozie.service.ActionCheckerService.action.check.delay ->20
#oozie.service.ActionCheckerService.callable.batch.size ->5
All oozie tasks are up and running without inconsistent state issues. However,
after several hours of running, oozie again has a large number of task states
running but tasks on Yarn have succeededI combed through the logic
Then, with Debugg mode enabled, it is found through oozie logs that the code
does not continue here
{code:java}
protected Void execute() throws CommandException {
// If the action is still in PREP, we probably received a callback
before Oozie was able to update from PREP to RUNNING;
// we'll requeue this command a few times and hope that it switches to
RUNNING before giving up
if (this.wfactionBean.getStatus() == WorkflowActionBean.Status.PREP) {
int maxEarlyRequeueCount =
Services.get().get(CallbackService.class).getEarlyRequeueMaxRetries();
if (this.earlyRequeueCount < maxEarlyRequeueCount) {
long delay = getRequeueDelay();
LOG.warn("Received early callback for action still in PREP
state; will wait [{0}]ms and requeue up to [{1}] more"
+ " times", delay, (maxEarlyRequeueCount -
earlyRequeueCount));
queue(new CompletedActionXCommand(this.actionId,
this.externalStatus, null, this.getPriority(),
this.earlyRequeueCount + 1), delay);
} else {
throw new CommandException(ErrorCode.E0822, actionId);
}
} else {// RUNNING
ActionExecutor executor =
Services.get().get(ActionService.class).getExecutor(this.wfactionBean.getType());
// this is done because oozie notifications (of sub-wfs) is send
// every status change, not only on completion.
if (executor.isCompleted(externalStatus)) {//***Here!!code
does not run into queue()*
queue(new ActionCheckXCommand(this.wfactionBean.getId(),
getPriority(), -1));
}
}
return null;
}
{code}
> Callback does not applied in Oozie server, workflows stuk in RUNNING states.
>
>
> Key: OOZIE-3581
> URL: https://issues.apache.org/jira/browse/OOZIE-3581
> Project: Oozie
> Issue Type: Bug
> Components: action, workflow
>Affects Versions: 4.3.1
>Reporter: Kotsubinsky Victor
>Priority: Critical
>
> oozie version 4.3.1.3.1.0.0-78
> with HDP3.10 stack , release provides Oozie 4.3.1 and the additional Apache
> patches listed here:
> [https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_oozie.html]
> I use Hadoop kerberized cluster, run on OOZIE, YARN Mr jobs.
> 1. OOzie run mr-job via YARN
> 2. After YARN mr job completed,YYARN mr-job Successfully sent Callback
> request to OOzie,
> 3. in logs OOzie server, i can see this request, but OOZIE does not apply
> this callback request, so in WF-action-id i still see RUNNING state (until
> action.check process check wf-ids and swith action-id to SUCCESS state)
> LOGS in YARN:
> 2020-01-27 12:16:39,749 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification trying
> http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification to
> http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> succeeded
> 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification succeeded for job_1579778851579_31505
>
> Oozie logs about this event:
> 2020-01-27 12:16:39,770 DEBUG CallbackServlet:526 - SERVER[hdp3-oo-2] USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Received a CallbackServlet.doGet() with query string
> id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2]
> USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-
[jira] [Commented] (OOZIE-3581) Callback does not applied in Oozie server, workflows stuk in RUNNING states.
[
https://issues.apache.org/jira/browse/OOZIE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17027193#comment-17027193
]
Junfan Zhang commented on OOZIE-3581:
-
Yes, this callback will trigger immediately event, firstly trigger
{{CompletedActionXCommand}} and then {{ActionCheckXCommand}}.
If there is a delay problem, there may be a bug in it. The code is
[here|https://github.com/apache/oozie/blob/5b55169c6743194223a0756756fc7247eb673aca/core/src/main/java/org/apache/oozie/servlet/CallbackServlet.java#L121].
> Callback does not applied in Oozie server, workflows stuk in RUNNING states.
>
>
> Key: OOZIE-3581
> URL: https://issues.apache.org/jira/browse/OOZIE-3581
> Project: Oozie
> Issue Type: Bug
> Components: action, workflow
>Affects Versions: 4.3.1
>Reporter: Kotsubinsky Victor
>Priority: Critical
>
> oozie version 4.3.1.3.1.0.0-78
> with HDP3.10 stack , release provides Oozie 4.3.1 and the additional Apache
> patches listed here:
> [https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_oozie.html]
> I use Hadoop kerberized cluster, run on OOZIE, YARN Mr jobs.
> 1. OOzie run mr-job via YARN
> 2. After YARN mr job completed,YYARN mr-job Successfully sent Callback
> request to OOzie,
> 3. in logs OOzie server, i can see this request, but OOZIE does not apply
> this callback request, so in WF-action-id i still see RUNNING state (until
> action.check process check wf-ids and swith action-id to SUCCESS state)
> LOGS in YARN:
> 2020-01-27 12:16:39,749 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification trying
> http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification to
> http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> succeeded
> 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification succeeded for job_1579778851579_31505
>
> Oozie logs about this event:
> 2020-01-27 12:16:39,770 DEBUG CallbackServlet:526 - SERVER[hdp3-oo-2] USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Received a CallbackServlet.doGet() with query string
> id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2]
> USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Execute command [callback] key [null]
> 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2]
> USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Queuing [1] commands with delay [0]ms
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (OOZIE-3581) Callback does not applied in Oozie server, workflows stuk in RUNNING states.
[ https://issues.apache.org/jira/browse/OOZIE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026706#comment-17026706 ] Kotsubinsky Victor commented on OOZIE-3581: --- Hi [~zuston], thank you for you information. If possible, can you sent me, link of oozie documentation or source-code where this case described? check.action process runs periodically, and recheck after some periods. It is not synchronous operation. I guest work callback as immediately event, not as scheduled. It is possible make patch for immediate process? > Callback does not applied in Oozie server, workflows stuk in RUNNING states. > > > Key: OOZIE-3581 > URL: https://issues.apache.org/jira/browse/OOZIE-3581 > Project: Oozie > Issue Type: Bug > Components: action, workflow >Affects Versions: 4.3.1 >Reporter: Kotsubinsky Victor >Priority: Critical > > oozie version 4.3.1.3.1.0.0-78 > with HDP3.10 stack , release provides Oozie 4.3.1 and the additional Apache > patches listed here: > [https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_oozie.html] > I use Hadoop kerberized cluster, run on OOZIE, YARN Mr jobs. > 1. OOzie run mr-job via YARN > 2. After YARN mr job completed,YYARN mr-job Successfully sent Callback > request to OOzie, > 3. in logs OOzie server, i can see this request, but OOZIE does not apply > this callback request, so in WF-action-id i still see RUNNING state (until > action.check process check wf-ids and swith action-id to SUCCESS state) > LOGS in YARN: > 2020-01-27 12:16:39,749 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification trying > http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification to > http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > succeeded > 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end > notification succeeded for job_1579778851579_31505 > > Oozie logs about this event: > 2020-01-27 12:16:39,770 DEBUG CallbackServlet:526 - SERVER[hdp3-oo-2] USER[-] > GROUP[-] TOKEN[-] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Received a CallbackServlet.doGet() with query string > id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED > 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2] > USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Execute command [callback] key [null] > 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2] > USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W] > ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java] > Queuing [1] commands with delay [0]ms -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (OOZIE-3581) Callback does not applied in Oozie server, workflows stuk in RUNNING states.
[
https://issues.apache.org/jira/browse/OOZIE-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17026653#comment-17026653
]
Junfan Zhang commented on OOZIE-3581:
-
Hey [~vkotsubinskiy], {{callback}} will trigger {{checkCommand}} instead of
directly determining action status based on {{callback}}.
> Callback does not applied in Oozie server, workflows stuk in RUNNING states.
>
>
> Key: OOZIE-3581
> URL: https://issues.apache.org/jira/browse/OOZIE-3581
> Project: Oozie
> Issue Type: Bug
> Components: action, workflow
>Affects Versions: 4.3.1
>Reporter: Kotsubinsky Victor
>Priority: Critical
>
> oozie version 4.3.1.3.1.0.0-78
> with HDP3.10 stack , release provides Oozie 4.3.1 and the additional Apache
> patches listed here:
> [https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.0/release-notes/content/patch_oozie.html]
> I use Hadoop kerberized cluster, run on OOZIE, YARN Mr jobs.
> 1. OOzie run mr-job via YARN
> 2. After YARN mr job completed,YYARN mr-job Successfully sent Callback
> request to OOzie,
> 3. in logs OOzie server, i can see this request, but OOZIE does not apply
> this callback request, so in WF-action-id i still see RUNNING state (until
> action.check process check wf-ids and swith action-id to SUCCESS state)
> LOGS in YARN:
> 2020-01-27 12:16:39,749 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification trying
> http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification to
> http://hdp3-oozie:11000/oozie/callback?id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> succeeded
> 2020-01-27 12:16:39,772 INFO [Thread-78] org.eclipse.jetty.util.log: Job end
> notification succeeded for job_1579778851579_31505
>
> Oozie logs about this event:
> 2020-01-27 12:16:39,770 DEBUG CallbackServlet:526 - SERVER[hdp3-oo-2] USER[-]
> GROUP[-] TOKEN[-] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Received a CallbackServlet.doGet() with query string
> id=0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java&status=SUCCEEDED
> 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2]
> USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Execute command [callback] key [null]
> 2020-01-27 12:16:39,776 DEBUG CompletedActionXCommand:526 - SERVER[hdp3-oo-2]
> USER[-] GROUP[-] TOKEN[] APP[-] JOB[0005607-200123121357414-oozie-oozi-W]
> ACTION[0005607-200123121357414-oozie-oozi-W@rdb-full-table-extract-java]
> Queuing [1] commands with delay [0]ms
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
