[
https://issues.apache.org/jira/browse/SQOOP-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188443#comment-14188443
]
Jarek Jarcec Cecho commented on SQOOP-1638:
-------------------------------------------
We have a loop in
[JobManager|https://github.com/apache/sqoop/blob/sqoop2/core/src/main/java/org/apache/sqoop/driver/JobManager.java#L669]
that is periodically checking status of currently running jobs and updating
their status. The problem with that loop is that it do have a delay - it's
configurable but yet if we check the job's status every 5 minutes, we can end
up having up to five minutes of delay where Sqoop thinks that the job is still
running when it's not.
Hadoop do have ability to send a HTTP request to given configured URL when a
job has been finished (both successfully or on failure) - it's sort of a
notification mechanism. I'm not sure whether other execution engines (namely
spark) do have this ability, but I would expect so. We're using this ability to
cut the delay on update loop. Once Hadoop execution is done, we get called on
the notification end point, we will get the latest status (hence the call to
{{status()}}) and persist it in our repository. As Hadoop doesn't care about
return value, we are returning {{EMPTY_BEAN}} back.
Obviously this end point is more an internal of Sqoop 2 that is exposed to
execution engine rather then endpoint that is meant to be used by a user.
> Notification on Submission.
> ---------------------------
>
> Key: SQOOP-1638
> URL: https://issues.apache.org/jira/browse/SQOOP-1638
> Project: Sqoop
> Issue Type: Sub-task
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Fix For: 2.0.0
>
>
> What is this really supposed to do?
> There was some hacky code to set this URL
> {code}
> // TODO: This should be outsourced somewhere more suitable than here
> if(JobManager.getInstance().getNotificationBaseUrl() == null) {
> String url = ctx.getRequest().getRequestURL().toString();
> JobManager.getInstance().setNotificationBaseUrl(
> url.split("v1")[0] + "/v1/submission/notification/");
> }
> {code}
> But all it did was return status
> {code}
> private JsonBean handleNotification(RequestContext ctx, String sjid) {
> LOG.debug("Received notification request for job " + sjid);
> JobManager.getInstance().status(Long.parseLong(sjid));
> return JsonBean.EMPTY_BEAN;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
