GitHub user tashoyan opened a pull request:
https://github.com/apache/spark/pull/18885
[SPARK-21668][CORE] Ability to run driver programs within a container
## What changes were proposed in this pull request?
When running inside a container, driver program provides a driver host set
to the container IP address. This IP address is visible only on the machine
where the container is running. Spark executors running on other machines are
not able to communicate to the driver program.
Now driver program may use standard SPARK_PUBLIC_DNS variable in order to
expose driver host to executors. Just declare SPARK_PUBLIC_DNS=<host machine
where the container is running> in spark-env.sh within the container. Thanks to
exposed ports, all requests from executors are forwarded to the driver program
within the container.
## How was this patch tested?
I have tested this modification manually. I have a Spark cluster on 3
machines. I run my Spark application in a Docker container; the host machine
belongs to the same network as my Spark cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tashoyan/spark SPARK-21668
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18885.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18885
----
commit 6b16afb3dbfed3c745820bc3e727b4c9a13017f7
Author: Arseniy Tashoyan <[email protected]>
Date: 2017-08-08T11:58:45Z
Driver program should advertise the hostname specified in SPARK_PUBLIC_DNS
if specified
commit 12c0b901ea109f7389d1c83afdc16817c2cb0cfd
Author: Arseniy Tashoyan <[email protected]>
Date: 2017-08-08T12:08:28Z
Worker keeps driver on the host provided in the driverUrl. It may differ
from the original spark.driver.host value if the driver specified
SPARK_PUBLIC_DNS.
commit bd7399c1552768d54ab7c8cfc1dfeb27667c7f95
Author: Arseniy Tashoyan <[email protected]>
Date: 2017-08-08T12:11:27Z
When starting executor, take SPARK_PUBLIC_DNS into account for the driver
url
commit e16d334eb58efe2375f4c85d77739ca3bacccecd
Author: Arseniy Tashoyan <[email protected]>
Date: 2017-08-08T12:49:54Z
Honor checkstyle
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
