[
https://issues.apache.org/jira/browse/SPARK-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14637632#comment-14637632
]
Sean Owen commented on SPARK-9252:
----------------------------------
I think this is more of a question for user@ rather than a JIRA.
The model for a Spark app is that the lifetime of the context is basically the
lifetime of the driver JVM. If you're manually embedding Spark, yes you have to
try to manage the context yourself. Although it should be possible to make a
new context like that, I don't know that this is supported.
Try 1.4.1 instead. It's possible something was improved along the way to
enabled this. If so it would be useful if you can provide any info about what
the problem is, like what error you see, and therefore what is not being
reinitialized.
> Spark client application must be restarted if the cluster (yarn) gets
> restarted
> -------------------------------------------------------------------------------
>
> Key: SPARK-9252
> URL: https://issues.apache.org/jira/browse/SPARK-9252
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 1.3.0
> Environment: Spark 1.3.0, Apache Hadoop 2.6
> Reporter: Sarto Mihai
>
> We have a Java application that is building and sending successful RDDs. But
> if the cluster gets restarted, even we detect that from the application and
> rebuild the JavaSparkContext the execution will all fail until we restart the
> application too.
> We are suspecting there is something static with the JavaSparkContext that
> does not get reinitialized - because we build new JavaSparkContext objects if
> we detect the oldSparkContext.env().isStopped().
> If we also restart the 'client' application then the RDD executions will work
> just fine.
> Therefore, we would like to not restart our application in case the Hadoop
> cluster get restarted and be able to make new JavaSparkContext in case the
> old Yarn application (Spark) was stopped.
> Let me know should you need any more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
