[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15480816#comment-15480816 ] Liam Fisk commented on SPARK-4563: -- There was a proposed patch in SPARK-11638, unfortunately the PR (and subsequently the ticket) was closed. Perhaps that patch could be investigated? > Allow spark driver to bind to different ip then advertise ip > > > Key: SPARK-4563 > URL: https://issues.apache.org/jira/browse/SPARK-4563 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Long Nguyen >Priority: Minor > > Spark driver bind ip and advertise is not configurable. spark.driver.host is > only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option > to set advertised ip/hostname -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11638) Run Spark on Mesos with bridge networking
[
https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432163#comment-15432163
]
Liam Fisk commented on SPARK-11638:
---
Just mirroring what I said on SPARK-4563 about the lack of support for bridged
networking:
{quote}
It also makes life difficult for OSX users. Docker for Mac uses xhyve to
virtualize the docker engine
(https://docs.docker.com/engine/installation/mac/), and thus `--net=host` binds
to the VM's network instead of the true OSX host. The SPARK_LOCAL_IP ends up as
172.17.0.2, which is not externally contactable.
The end result is OSX users cannot containerize Spark if Spark needs to contact
a mesos cluster.
{quote}
While you are unlikely to have Spark running on an OSX machine in production,
the development experience is a bit painful if you have to run a separate VM
with public networking.
> Run Spark on Mesos with bridge networking
> -
>
> Key: SPARK-11638
> URL: https://issues.apache.org/jira/browse/SPARK-11638
> Project: Spark
> Issue Type: Improvement
> Components: Mesos, Spark Core
>Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0
>Reporter: Radoslaw Gruchalski
> Attachments: 1.4.0.patch, 1.4.1.patch, 1.5.0.patch, 1.5.1.patch,
> 1.5.2.patch, 1.6.0.patch, 2.3.11.patch, 2.3.4.patch
>
>
> h4. Summary
> Provides {{spark.driver.advertisedPort}},
> {{spark.fileserver.advertisedPort}}, {{spark.broadcast.advertisedPort}} and
> {{spark.replClassServer.advertisedPort}} settings to enable running Spark in
> Mesos on Docker with Bridge networking. Provides patches for Akka Remote to
> enable Spark driver advertisement using alternative host and port.
> With these settings, it is possible to run Spark Master in a Docker container
> and have the executors running on Mesos talk back correctly to such Master.
> The problem is discussed on the Mesos mailing list here:
> https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bfotj5ljzfrh5u7ix-ghppfqknvg9mkkc...@mail.gmail.com%3E
> h4. Running Spark on Mesos - LIBPROCESS_ADVERTISE_IP opens the door
> In order for the framework to receive orders in the bridged container, Mesos
> in the container has to register for offers using the IP address of the
> Agent. Offers are sent by Mesos Master to the Docker container running on a
> different host, an Agent. Normally, prior to Mesos 0.24.0, {{libprocess}}
> would advertise itself using the IP address of the container, something like
> {{172.x.x.x}}. Obviously, Mesos Master can't reach that address, it's a
> different host, it's a different machine. Mesos 0.24.0 introduced two new
> properties for {{libprocess}} - {{LIBPROCESS_ADVERTISE_IP}} and
> {{LIBPROCESS_ADVERTISE_PORT}}. This allows the container to use the Agent's
> address to register for offers. This was provided mainly for running Mesos in
> Docker on Mesos.
> h4. Spark - how does the above relate and what is being addressed here?
> Similar to Mesos, out of the box, Spark does not allow to advertise its
> services on ports different than bind ports. Consider following scenario:
> Spark is running inside a Docker container on Mesos, it's a bridge networking
> mode. Assuming a port {{}} for the {{spark.driver.port}}, {{6677}} for
> the {{spark.fileserver.port}}, {{6688}} for the {{spark.broadcast.port}} and
> {{23456}} for the {{spark.replClassServer.port}}. If such task is posted to
> Marathon, Mesos will give 4 ports in range {{31000-32000}} mapping to the
> container ports. Starting the executors from such container results in
> executors not being able to communicate back to the Spark Master.
> This happens because of 2 things:
> Spark driver is effectively an {{akka-remote}} system with {{akka.tcp}}
> transport. {{akka-remote}} prior to version {{2.4}} can't advertise a port
> different to what it bound to. The settings discussed are here:
> https://github.com/akka/akka/blob/f8c1671903923837f22d0726a955e0893add5e9f/akka-remote/src/main/resources/reference.conf#L345-L376.
> These do not exist in Akka {{2.3.x}}. Spark driver will always advertise
> port {{}} as this is the one {{akka-remote}} is bound to.
> Any URIs the executors contact the Spark Master on, are prepared by Spark
> Master and handed over to executors. These always contain the port number
> used by the Master to find the service on. The services are:
> - {{spark.broadcast.port}}
> - {{spark.fileserver.port}}
> - {{spark.replClassServer.port}}
> all above ports are by default {{0}} (random assignment) but can be specified
> using Spark configuration ( {{-Dspark...port}} ). However, they are limited
> in the same way as the {{spark.driver.port}}; in the above example, an
> executor should not contact the file server on port {{6677}} but rather on
> the respective
[jira] [Commented] (SPARK-11638) Run Spark on Mesos with bridge networking
[
https://issues.apache.org/jira/browse/SPARK-11638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432151#comment-15432151
]
Liam Fisk commented on SPARK-11638:
---
This ticket appears to be related to SPARK-4563.
> Run Spark on Mesos with bridge networking
> -
>
> Key: SPARK-11638
> URL: https://issues.apache.org/jira/browse/SPARK-11638
> Project: Spark
> Issue Type: Improvement
> Components: Mesos, Spark Core
>Affects Versions: 1.4.0, 1.4.1, 1.5.0, 1.5.1, 1.5.2, 1.6.0
>Reporter: Radoslaw Gruchalski
> Attachments: 1.4.0.patch, 1.4.1.patch, 1.5.0.patch, 1.5.1.patch,
> 1.5.2.patch, 1.6.0.patch, 2.3.11.patch, 2.3.4.patch
>
>
> h4. Summary
> Provides {{spark.driver.advertisedPort}},
> {{spark.fileserver.advertisedPort}}, {{spark.broadcast.advertisedPort}} and
> {{spark.replClassServer.advertisedPort}} settings to enable running Spark in
> Mesos on Docker with Bridge networking. Provides patches for Akka Remote to
> enable Spark driver advertisement using alternative host and port.
> With these settings, it is possible to run Spark Master in a Docker container
> and have the executors running on Mesos talk back correctly to such Master.
> The problem is discussed on the Mesos mailing list here:
> https://mail-archives.apache.org/mod_mbox/mesos-user/201510.mbox/%3CCACTd3c9vjAMXk=bfotj5ljzfrh5u7ix-ghppfqknvg9mkkc...@mail.gmail.com%3E
> h4. Running Spark on Mesos - LIBPROCESS_ADVERTISE_IP opens the door
> In order for the framework to receive orders in the bridged container, Mesos
> in the container has to register for offers using the IP address of the
> Agent. Offers are sent by Mesos Master to the Docker container running on a
> different host, an Agent. Normally, prior to Mesos 0.24.0, {{libprocess}}
> would advertise itself using the IP address of the container, something like
> {{172.x.x.x}}. Obviously, Mesos Master can't reach that address, it's a
> different host, it's a different machine. Mesos 0.24.0 introduced two new
> properties for {{libprocess}} - {{LIBPROCESS_ADVERTISE_IP}} and
> {{LIBPROCESS_ADVERTISE_PORT}}. This allows the container to use the Agent's
> address to register for offers. This was provided mainly for running Mesos in
> Docker on Mesos.
> h4. Spark - how does the above relate and what is being addressed here?
> Similar to Mesos, out of the box, Spark does not allow to advertise its
> services on ports different than bind ports. Consider following scenario:
> Spark is running inside a Docker container on Mesos, it's a bridge networking
> mode. Assuming a port {{}} for the {{spark.driver.port}}, {{6677}} for
> the {{spark.fileserver.port}}, {{6688}} for the {{spark.broadcast.port}} and
> {{23456}} for the {{spark.replClassServer.port}}. If such task is posted to
> Marathon, Mesos will give 4 ports in range {{31000-32000}} mapping to the
> container ports. Starting the executors from such container results in
> executors not being able to communicate back to the Spark Master.
> This happens because of 2 things:
> Spark driver is effectively an {{akka-remote}} system with {{akka.tcp}}
> transport. {{akka-remote}} prior to version {{2.4}} can't advertise a port
> different to what it bound to. The settings discussed are here:
> https://github.com/akka/akka/blob/f8c1671903923837f22d0726a955e0893add5e9f/akka-remote/src/main/resources/reference.conf#L345-L376.
> These do not exist in Akka {{2.3.x}}. Spark driver will always advertise
> port {{}} as this is the one {{akka-remote}} is bound to.
> Any URIs the executors contact the Spark Master on, are prepared by Spark
> Master and handed over to executors. These always contain the port number
> used by the Master to find the service on. The services are:
> - {{spark.broadcast.port}}
> - {{spark.fileserver.port}}
> - {{spark.replClassServer.port}}
> all above ports are by default {{0}} (random assignment) but can be specified
> using Spark configuration ( {{-Dspark...port}} ). However, they are limited
> in the same way as the {{spark.driver.port}}; in the above example, an
> executor should not contact the file server on port {{6677}} but rather on
> the respective 31xxx assigned by Mesos.
> Spark currently does not allow any of that.
> h4. Taking on the problem, step 1: Spark Driver
> As mentioned above, Spark Driver is based on {{akka-remote}}. In order to
> take on the problem, the {{akka.remote.net.tcp.bind-hostname}} and
> {{akka.remote.net.tcp.bind-port}} settings are a must. Spark does not compile
> with Akka 2.4.x yet.
> What we want is the back port of mentioned {{akka-remote}} settings to
> {{2.3.x}} versions. These patches are attached to this ticket -
> {{2.3.4.patch}} and {{2.3.11.patch}} files provide patches for respective
> akka versions. These add mentioned settings and ensure they
[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432149#comment-15432149 ] Liam Fisk commented on SPARK-4563: -- This ticket appears to be related to SPARK-11638 > Allow spark driver to bind to different ip then advertise ip > > > Key: SPARK-4563 > URL: https://issues.apache.org/jira/browse/SPARK-4563 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Long Nguyen >Priority: Minor > > Spark driver bind ip and advertise is not configurable. spark.driver.host is > only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option > to set advertised ip/hostname -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432128#comment-15432128 ] Liam Fisk commented on SPARK-4563: -- It also makes life difficult for OSX users. Docker for Mac uses xhyve to virtualize the docker engine (https://docs.docker.com/engine/installation/mac/), and thus `--net=host` binds to the VM's network instead of the true OSX host. The SPARK_LOCAL_IP ends up as 172.17.0.2, which is not externally contactable. The end result is OSX users cannot containerize Spark if Spark needs to contact a mesos cluster. > Allow spark driver to bind to different ip then advertise ip > > > Key: SPARK-4563 > URL: https://issues.apache.org/jira/browse/SPARK-4563 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Long Nguyen >Priority: Minor > > Spark driver bind ip and advertise is not configurable. spark.driver.host is > only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option > to set advertised ip/hostname -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432089#comment-15432089 ] Liam Fisk commented on SPARK-4563: -- Further to my comment, I disagree with the "minor" rating of this issue. Without this feature, Spark cannot be containerized in a production environment, as --net=host is not an option when multiple containers exist. > Allow spark driver to bind to different ip then advertise ip > > > Key: SPARK-4563 > URL: https://issues.apache.org/jira/browse/SPARK-4563 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Long Nguyen >Priority: Minor > > Spark driver bind ip and advertise is not configurable. spark.driver.host is > only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option > to set advertised ip/hostname -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4563) Allow spark driver to bind to different ip then advertise ip
[ https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432086#comment-15432086 ] Liam Fisk commented on SPARK-4563: -- +1 for me I am running into this problem when I run `spark-submit --master mesos://zk://foo:2181/mesos ` in a Docker container. As Spark is in the container, the SPARK_LOCAL_IP will resolve to 172.17.0.3 or similar, and the mesos executors will fail to contact this address. If SPARK_ADVERTISED_IP existed then I would broadcast the ip of the host system. I cannot use host networking (as this container will inhabit multi-tenanted infrastructure). > Allow spark driver to bind to different ip then advertise ip > > > Key: SPARK-4563 > URL: https://issues.apache.org/jira/browse/SPARK-4563 > Project: Spark > Issue Type: Improvement > Components: Deploy >Reporter: Long Nguyen >Priority: Minor > > Spark driver bind ip and advertise is not configurable. spark.driver.host is > only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option > to set advertised ip/hostname -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15909) PySpark classpath uri incorrectly set
[
https://issues.apache.org/jira/browse/SPARK-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15337774#comment-15337774
]
Liam Fisk commented on SPARK-15909:
---
Cluster mode isn't used here, I have a mesos cluster (and therefore am in
client mode, as you said).
In client mode, the remote mesos executors need to be able to retrieve any
dependencies, and they can't do that if they are attempting to contact
localhost.
The bug here is that there is completely different behaviour on startup vs
within the REPL. If I stop the spark context, clone the config, and construct a
new spark context it will no longer work.
> PySpark classpath uri incorrectly set
> -
>
> Key: SPARK-15909
> URL: https://issues.apache.org/jira/browse/SPARK-15909
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.6.1
>Reporter: Liam Fisk
>
> PySpark behaves differently if the SparkContext is created within the REPL
> (vs initialised by the shell).
> My conf/spark-env.sh file contains:
> {code}
> #!/bin/bash
> export SPARK_LOCAL_IP=172.20.30.158
> export LIBPROCESS_IP=172.20.30.158
> export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
> {code}
> And when running pyspark it will correctly initialize my SparkContext.
> However, when I run:
> {code}
> from pyspark import SparkContext, SparkConf
> sc.stop()
> conf = (
> SparkConf()
> .setMaster("mesos://zk://foo:2181/mesos")
> .setAppName("Jupyter PySpark")
> )
> sc = SparkContext(conf=conf)
> {code}
> my _spark.driver.uri_ and URL classpath will point to localhost (preventing
> my mesos cluster from accessing the appropriate files)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15909) PySpark classpath uri incorrectly set
[
https://issues.apache.org/jira/browse/SPARK-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liam Fisk updated SPARK-15909:
--
Description:
PySpark behaves differently if the SparkContext is created within the REPL (vs
initialised by the shell).
My conf/spark-env.sh file contains:
{code}
#!/bin/bash
export SPARK_LOCAL_IP=172.20.30.158
export LIBPROCESS_IP=172.20.30.158
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
{code}
And when running pyspark it will correctly initialize my SparkContext. However,
when I run:
{code}
from pyspark import SparkContext, SparkConf
sc.stop()
conf = (
SparkConf()
.setMaster("mesos://zk://foo:2181/mesos")
.setAppName("Jupyter PySpark")
)
sc = SparkContext(conf=conf)
{code}
my _spark.driver.uri_ and URL classpath will point to localhost (preventing my
mesos cluster from accessing the appropriate files)
was:
PySpark behaves differently if the SparkContext is created within the REPL (vs
initialised by the shell).
My conf/spark-env.sh file contains:
{code}
#!/bin/bash
export SPARK_LOCAL_IP=172.20.30.158
export LIBPROCESS_IP=172.20.30.158
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
{code}
And when running pyspark it will correctly initialize my SparkContext. However,
when I run:
{code}
from pyspark import SparkContext, SparkConf
sc.stop()
conf = (
SparkConf()
.setMaster("mesos://zk://foo:2181/mesos")
.setAppName("Jupyter PySpark")
)
sc = SparkContext(conf=conf)
{code}
my `spark.driver.uri` and URL classpath will point to localhost (preventing my
mesos cluster from accessing the appropriate files)
> PySpark classpath uri incorrectly set
> -
>
> Key: SPARK-15909
> URL: https://issues.apache.org/jira/browse/SPARK-15909
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.6.1
>Reporter: Liam Fisk
>
> PySpark behaves differently if the SparkContext is created within the REPL
> (vs initialised by the shell).
> My conf/spark-env.sh file contains:
> {code}
> #!/bin/bash
> export SPARK_LOCAL_IP=172.20.30.158
> export LIBPROCESS_IP=172.20.30.158
> export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
> {code}
> And when running pyspark it will correctly initialize my SparkContext.
> However, when I run:
> {code}
> from pyspark import SparkContext, SparkConf
> sc.stop()
> conf = (
> SparkConf()
> .setMaster("mesos://zk://foo:2181/mesos")
> .setAppName("Jupyter PySpark")
> )
> sc = SparkContext(conf=conf)
> {code}
> my _spark.driver.uri_ and URL classpath will point to localhost (preventing
> my mesos cluster from accessing the appropriate files)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15909) PySpark classpath uri incorrectly set
[
https://issues.apache.org/jira/browse/SPARK-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liam Fisk updated SPARK-15909:
--
Description:
PySpark behaves differently if the SparkContext is created within the REPL (vs
initialised by the shell).
My conf/spark-env.sh file contains:
{code}
#!/bin/bash
export SPARK_LOCAL_IP=172.20.30.158
export LIBPROCESS_IP=172.20.30.158
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
{code}
And when running pyspark it will correctly initialize my SparkContext. However,
when I run:
{code}
from pyspark import SparkContext, SparkConf
sc.stop()
conf = (
SparkConf()
.setMaster("mesos://zk://foo:2181/mesos")
.setAppName("Jupyter PySpark")
)
sc = SparkContext(conf=conf)
{code}
my `spark.driver.uri` and URL classpath will point to localhost (preventing my
mesos cluster from accessing the appropriate files)
was:
PySpark behaves differently if the SparkContext is created within the REPL (vs
initialised by the shell).
My conf/spark-env.sh file contains:
{code}
#!/bin/bash
export SPARK_LOCAL_IP=172.20.30.158
export LIBPROCESS_IP=172.20.30.158
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
{code}
And when running pyspark it will correctly initialize my SparkContext. However,
when I run:
{code}
from pyspark import SparkContext, SparkConf
sc.stop()
conf = (
SparkConf()
.setMaster("mesos://zk://foo:2181/mesos")
.setAppName("Jupyter PySpark")
)
sc = SparkContext(conf=conf)
{code}
it will point to localhost (preventing my mesos cluster from accessing the
appropriate files)
> PySpark classpath uri incorrectly set
> -
>
> Key: SPARK-15909
> URL: https://issues.apache.org/jira/browse/SPARK-15909
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.6.1
>Reporter: Liam Fisk
>
> PySpark behaves differently if the SparkContext is created within the REPL
> (vs initialised by the shell).
> My conf/spark-env.sh file contains:
> {code}
> #!/bin/bash
> export SPARK_LOCAL_IP=172.20.30.158
> export LIBPROCESS_IP=172.20.30.158
> export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
> {code}
> And when running pyspark it will correctly initialize my SparkContext.
> However, when I run:
> {code}
> from pyspark import SparkContext, SparkConf
> sc.stop()
> conf = (
> SparkConf()
> .setMaster("mesos://zk://foo:2181/mesos")
> .setAppName("Jupyter PySpark")
> )
> sc = SparkContext(conf=conf)
> {code}
> my `spark.driver.uri` and URL classpath will point to localhost (preventing
> my mesos cluster from accessing the appropriate files)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15909) PySpark classpath uri incorrectly set
[
https://issues.apache.org/jira/browse/SPARK-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liam Fisk updated SPARK-15909:
--
Description:
PySpark behaves differently if the SparkContext is created within the REPL (vs
initialised by the shell).
My conf/spark-env.sh file contains:
{code}
#!/bin/bash
export SPARK_LOCAL_IP=172.20.30.158
export LIBPROCESS_IP=172.20.30.158
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
{code}
And when running pyspark it will correctly initialize my SparkContext. However,
when I run:
{code}
from pyspark import SparkContext, SparkConf
sc.stop()
conf = (
SparkConf()
.setMaster("mesos://zk://foo:2181/mesos")
.setAppName("Jupyter PySpark")
)
sc = SparkContext(conf=conf)
{code}
it will point to localhost (preventing my mesos cluster from accessing the
appropriate files)
was:
PySpark behaves differently if the SparkContext is created within the REPL (vs
initialised by the shell).
My conf/spark-env.sh file contains:
{code}
#!/bin/bash
export SPARK_LOCAL_IP=172.20.30.158
export LIBPROCESS_IP=172.20.30.158
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
{code}
> PySpark classpath uri incorrectly set
> -
>
> Key: SPARK-15909
> URL: https://issues.apache.org/jira/browse/SPARK-15909
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.6.1
>Reporter: Liam Fisk
>
> PySpark behaves differently if the SparkContext is created within the REPL
> (vs initialised by the shell).
> My conf/spark-env.sh file contains:
> {code}
> #!/bin/bash
> export SPARK_LOCAL_IP=172.20.30.158
> export LIBPROCESS_IP=172.20.30.158
> export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
> {code}
> And when running pyspark it will correctly initialize my SparkContext.
> However, when I run:
> {code}
> from pyspark import SparkContext, SparkConf
> sc.stop()
> conf = (
> SparkConf()
> .setMaster("mesos://zk://foo:2181/mesos")
> .setAppName("Jupyter PySpark")
> )
> sc = SparkContext(conf=conf)
> {code}
> it will point to localhost (preventing my mesos cluster from accessing the
> appropriate files)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-15909) PySpark classpath uri incorrectly set
[
https://issues.apache.org/jira/browse/SPARK-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liam Fisk updated SPARK-15909:
--
Description:
PySpark behaves differently if the SparkContext is created within the REPL (vs
initialised by the shell).
My conf/spark-env.sh file contains:
{code}
#!/bin/bash
export SPARK_LOCAL_IP=172.20.30.158
export LIBPROCESS_IP=172.20.30.158
export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
{code}
was:PySpark behaves differently if the SparkContext is created within the
REPL (vs initialised by the shell).
> PySpark classpath uri incorrectly set
> -
>
> Key: SPARK-15909
> URL: https://issues.apache.org/jira/browse/SPARK-15909
> Project: Spark
> Issue Type: Bug
> Components: PySpark
>Affects Versions: 1.6.1
>Reporter: Liam Fisk
>
> PySpark behaves differently if the SparkContext is created within the REPL
> (vs initialised by the shell).
> My conf/spark-env.sh file contains:
> {code}
> #!/bin/bash
> export SPARK_LOCAL_IP=172.20.30.158
> export LIBPROCESS_IP=172.20.30.158
> export MESOS_NATIVE_JAVA_LIBRARY=/usr/local/lib/libmesos.so
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-15909) PySpark classpath uri incorrectly set
Liam Fisk created SPARK-15909: - Summary: PySpark classpath uri incorrectly set Key: SPARK-15909 URL: https://issues.apache.org/jira/browse/SPARK-15909 Project: Spark Issue Type: Bug Components: PySpark Affects Versions: 1.6.1 Reporter: Liam Fisk PySpark behaves differently if the SparkContext is created within the REPL (vs initialised by the shell). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
