[
https://issues.apache.org/jira/browse/SPARK-2105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14184613#comment-14184613
]
Josh Rosen commented on SPARK-2105:
-----------------------------------
I tried and failed to reproduce this:
https://github.com/apache/spark/commit/bf589fc717c842d1998e3c3a523bc8775cb30269#diff-f346ada4cd59416756b6dd36b6c2605aR97
That doesn't mean that we've fixed the issue, though. In my tests, the stage
never becomes active because the ClosureCleaner detects that the task isn't
serializable. Maybe there's some UDF that manages to slip through the closure
cleaning step and fails once the stage is submitted to the scheduler, so it's
still possible that we could hit this bug.
> SparkUI doesn't remove active stages that failed
> ------------------------------------------------
>
> Key: SPARK-2105
> URL: https://issues.apache.org/jira/browse/SPARK-2105
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Web UI
> Affects Versions: 1.0.0
> Reporter: Andrew Or
>
> If a stage fails because its tasks cannot be serialized, for instance, the
> failed stage remains in the Active Stages section forever. This is because
> the StageCompleted event is never posted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
