[
https://issues.apache.org/jira/browse/SPARK-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337774#comment-15337774
]
Liam Fisk commented on SPARK-15909:
-----------------------------------
Cluster mode isn't used here, I have a mesos cluster (and therefore am in
client mode, as you said).
In client mode, the remote mesos executors need to be able to retrieve any
dependencies, and they can't do that if they are attempting to contact
localhost.
The bug here is that there is completely different behaviour on startup vs
within the REPL. If I stop the spark context, clone the config, and construct a
new spark context it will no longer work.
> PySpark classpath uri incorrectly set
> -------------------------------------
>
> Key: SPARK-15909
> URL: https://issues.apache.org/jira/browse/SPARK-15909
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 1.6.1
> Reporter: Liam Fisk
>
> PySpark behaves differently if the SparkContext is created within the REPL
> (vs initialised by the shell).
> My conf/spark-env.sh file contains:
> {code}
> #!/bin/bash
> export SPARK_LOCAL_IP=172.20.30.158
> export LIBPROCESS_IP=172.20.30.158
> export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
> {code}
> And when running pyspark it will correctly initialize my SparkContext.
> However, when I run:
> {code}
> from pyspark import SparkContext, SparkConf
> sc.stop()
> conf = (
> SparkConf()
> .setMaster("mesos://zk://foo:2181/mesos")
> .setAppName("Jupyter PySpark")
> )
> sc = SparkContext(conf=conf)
> {code}
> my _spark.driver.uri_ and URL classpath will point to localhost (preventing
> my mesos cluster from accessing the appropriate files)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
