[
https://issues.apache.org/jira/browse/MAPREDUCE-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129864#comment-13129864
]
Robert Joseph Evans commented on MAPREDUCE-3028:
------------------------------------------------
I don't like the job history server doing it because launching the job history
server is some what optional (The default start scripts do not launch it
currently). Also it will add in a new delay, and a lot more churn on the Job
history server. Currently the JobHistory server polls HDFS for new jobs that
have finished. If the job has finished it will move it to its own directory,
and then cache some info about the app based only on the name of the jhist
file. If we want this to do notification also then we need to either store the
URL in the jhist file name, or force the server to read in the config file
every time it pulls in a new Application.
I agree that having applications poll for status with something like a very low
latency RSS is the preferable solution and as such I would like to see the
minimal done to support backwards compatibility, which would be to put the
notification in the ApplicationMaster. But it is Ravi that is doing the fix
and if he wants to do more then that and it is not something that is going to
cause more problems then it fixes then I would say let him do it.
What I mean by cause more problems then it fixes is that we have had lots of
issues in production with having the notification be in a single thread
blocking other notifications. What ever solution it is it cannot allow other
notifications to become blocked or severely delayed by a single bad URL. (Or
our case of a remote server crashing in the middle of the request and some how
causing the socket be in a psudo state where the read blocks indefinitely and
there is no one on the other side that is ever going to write)
> Support job end notification in .next /0.23
> -------------------------------------------
>
> Key: MAPREDUCE-3028
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3028
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Ravi Prakash
> Priority: Blocker
> Fix For: 0.23.0
>
>
> Oozie primarily depends on the job end notification to determine when the
> job finishes. In the current version, job end notification is implemented in
> job tracker. Since job tracker will be removed in the upcoming hadoop release
> (.next), we wander where this support will move. I think this best effort
> notification could be implemented in the new Application Manager as one of
> the last step of job completion.
> Whatever implementation will it be, Oozie badly needs this feature to be
> continued in next releases as well.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
