[
https://issues.apache.org/jira/browse/SPARK-10812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945492#comment-14945492
]
Apache Spark commented on SPARK-10812:
--------------------------------------
User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/8996
> Spark Hadoop Util does not support stopping a non-yarn Spark Context &
> starting a Yarn spark context.
> -----------------------------------------------------------------------------------------------------
>
> Key: SPARK-10812
> URL: https://issues.apache.org/jira/browse/SPARK-10812
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Reporter: holdenk
> Assignee: Holden Karau
> Priority: Minor
> Fix For: 1.6.0
>
>
> While this is likely not a huge issue for real production systems, for test
> systems which may setup a Spark Context and tear it down and stand up a Spark
> Context with a different master (e.g. some local mode & some yarn mode) tests
> this cane be an issue. Discovered during work on spark-testing-base on Spark
> 1.4.1, but seems like the logic that triggers it is present in master (see
> SparkHadoopUtil object). A valid work around for users encountering this
> issue is to fork a different JVM, however this can be heavy weight.
> {quote}
> [info] SampleMiniClusterTest:
> [info] Exception encountered when attempting to run a suite with class name:
> com.holdenkarau.spark.testing.SampleMiniClusterTest *** ABORTED ***
> [info] java.lang.ClassCastException:
> org.apache.spark.deploy.SparkHadoopUtil cannot be cast to
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil
> [info] at
> org.apache.spark.deploy.yarn.YarnSparkHadoopUtil$.get(YarnSparkHadoopUtil.scala:163)
> [info] at
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:257)
> [info] at
> org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:561)
> [info] at
> org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:115)
> [info] at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
> [info] at
> org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
> [info] at org.apache.spark.SparkContext.<init>(SparkContext.scala:497)
> [info] at
> com.holdenkarau.spark.testing.SharedMiniCluster$class.setup(SharedMiniCluster.scala:186)
> [info] at
> com.holdenkarau.spark.testing.SampleMiniClusterTest.setup(SampleMiniClusterTest.scala:26)
> [info] at
> com.holdenkarau.spark.testing.SharedMiniCluster$class.beforeAll(SharedMiniCluster.scala:103)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
