[
https://issues.apache.org/jira/browse/SPARK-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13967228#comment-13967228
]
Kan Zhang edited comment on SPARK-1475 at 4/17/14 2:08 AM:
-----------------------------------------------------------
When event queue is not drained, users may observe similar issues as those
reported in SPARK-1407 (when sc.stop() is not called).
https://github.com/apache/spark/pull/366
The above PR fixes this issue. It does require applications to call sc.stop()
to properly stop SparkListenerBus and event logger.
was (Author: kzhang):
When event queue is not drained, users may observe similar issues as those
reported in SPARK-1407 (when sc.stop() is not called).
https://github.com/apache/spark/pull/366
The above PR fixes this issue. It does require applications to call
SparkContext.stop() to properly stop SparkListenerBus and event logger.
> Draining event logging queue before stopping event logger
> ---------------------------------------------------------
>
> Key: SPARK-1475
> URL: https://issues.apache.org/jira/browse/SPARK-1475
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Kan Zhang
> Assignee: Kan Zhang
> Priority: Blocker
> Fix For: 1.0.0
>
>
> When stopping SparkListenerBus, its event queue needs to be drained. And this
> needs to happen before event logger is stopped. Otherwise, any event still
> waiting to be processed in the queue may be lost and consequently event log
> file may be incomplete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
