[
https://issues.apache.org/jira/browse/SPARK-18225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15635846#comment-15635846
]
Saisai Shao commented on SPARK-18225:
-------------------------------------
I think the problem [~liujianhui] trying to address is that when using web UI's
kill function to kill the streaming application, though SparkContext is
aborted, other Streaming specific stuffs like checkpointing is still working,
so when the streaming application is recovered from last checkpoint, some jobs
are missing (though checkpointed, but actually not run).
The problem here is that Web UI (core part) is not aware of Streaming specific
things, what it could only do is to stop SparkContext related stuff, but it
cannot fully stop the things out of Spark core, like streaming.
I'm not sure is it necessary to fix this issue, since we don't encourage user
to stop streaming app in this way. Also it may not be easy to fix (the reason
is mentioned above, core part cannot be aware of Streaming specific things),
and the problem is not a Streaming specific problem, other Context like SQL may
also meet this problem if it has its own context specific things out of core.
> job will miss when driver removed by master in spark streaming
> ---------------------------------------------------------------
>
> Key: SPARK-18225
> URL: https://issues.apache.org/jira/browse/SPARK-18225
> Project: Spark
> Issue Type: Bug
> Components: DStreams, Scheduler
> Affects Versions: 1.6.1, 1.6.2
> Reporter: liujianhui
>
> kill the application on spark ui, the master will send an ApplicationRemoved
> to driver, driver will abort the all pending job,and then the job finish with
> exception "Master removed our application:Killed",and then Jobscheduler will
> remove the job from jobsets, but the jobgenerator still docheckpoint without
> the job which removed before, and then driver stop；when recover from the
> check point file，it miss all jobs which aborted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
