[
https://issues.apache.org/jira/browse/SPARK-4563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15636802#comment-15636802
]
György Süveges edited comment on SPARK-4563 at 11/4/16 4:01 PM:
----------------------------------------------------------------
+1
However, it's related to SPARK_LOCAL_IP and Spark UI, I think it's similar.
We use AWS EMR5 (Spark 2.0 over YARN)
While developing we use spark from intellij in yarn-client mode. To connect to
AWS, OpenVPN is applied with biderectional routing so the executors in amazon
can access the driver on the local workstation. With OpenVPN there are multiple
network interfaces then, so SPARK_LOCAL_IP should be set. But it's really
inconvenient to set SPARK_LOCAL_IP each time we connect to OpenVPN and get
different IP.
So we decided to set it automatically from code. Since SPARK_LOCAL_IP is env
var and env vars are inmutable in java, we change the spark.driver.host
corresponding java property. But as noticed spark.driver.host is only second
order citizen in spark, for some features only the SPARK_LOCAL_IP is taken into
account. For example Spark UI listens on the host set in SPARK_LOCAL_IP, but
ignores the spark.driver.host totally. I hope this improvement will fix this
issue, as I saw in
https://github.com/apache/spark/commit/2cd1bfa4f0c6625b0ab1dbeba2b9586b9a6a9f42
I think SPARK_LOCAL_IP and spark.driver.host should be equivalent counterparts,
shouldn't they?
was (Author: [email protected]):
+1
However, it's related to SPARK_LOCAL_IP and Spark UI, I think it's similar.
We use AWS EMR5 (Spark 2.0 over YARN)
While developing we use spark from intellij in yarn-client mode. To connect to
AWS, OpenVPN is applied with biderectional routing so the executors in amazon
can access the driver on the local workstation. With OpenVPN there are multiple
network interfaces then, so SPARK_LOCAL_IP should be set. But it's really
inconvenient to set SPARK_LOCAL_IP each time we connect to OpenVPN and get
different IP.
So we decided to set it automatically from code. Since SPARK_LOCAL_IP is env
var and env vars are inmutable in java, then we change the spark.driver.host
corresponding java property. But as noticed spark.driver.host is only second
order citizen in spark, for some features only the SPARK_LOCAL_IP is taken into
account. For example Spark UI listens on the host set in SPARK_LOCAL_IP, but
ignores the spark.driver.host totally. I hope this improvement will fix this
issue, as I saw in
https://github.com/apache/spark/commit/2cd1bfa4f0c6625b0ab1dbeba2b9586b9a6a9f42
I think SPARK_LOCAL_IP and spark.driver.host should be equivalent counterparts,
shouldn't they?
> Allow spark driver to bind to different ip then advertise ip
> ------------------------------------------------------------
>
> Key: SPARK-4563
> URL: https://issues.apache.org/jira/browse/SPARK-4563
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Reporter: Long Nguyen
> Assignee: Marcelo Vanzin
> Priority: Minor
> Fix For: 2.1.0
>
>
> Spark driver bind ip and advertise is not configurable. spark.driver.host is
> only bind ip. SPARK_PUBLIC_DNS does not work for spark driver. Allow option
> to set advertised ip/hostname
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
