GitHub user tashoyan opened a pull request:

    [SPARK-21668][CORE] Ability to run driver programs within a container

    ## What changes were proposed in this pull request?
    When running inside a container, driver program provides a driver host set 
to the container IP address. This IP address is visible only on the machine 
where the container is running. Spark executors running on other machines are 
not able to communicate to the driver program.
    Now driver program may use standard SPARK_PUBLIC_DNS variable in order to 
expose driver host to executors. Just declare SPARK_PUBLIC_DNS=<host machine 
where the container is running> in within the container. Thanks to 
exposed ports, all requests from executors are forwarded to the driver program 
within the container.
    ## How was this patch tested?
    I have tested this modification manually. I have a Spark cluster on 3 
machines. I run my Spark application in a Docker container; the host machine 
belongs to the same network as my Spark cluster.

You can merge this pull request into a Git repository by running:

    $ git pull SPARK-21668

Alternatively you can review and apply these changes as the patch at:

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #18885
commit 6b16afb3dbfed3c745820bc3e727b4c9a13017f7
Author: Arseniy Tashoyan <>
Date:   2017-08-08T11:58:45Z

    Driver program should advertise the hostname specified in SPARK_PUBLIC_DNS 
if specified

commit 12c0b901ea109f7389d1c83afdc16817c2cb0cfd
Author: Arseniy Tashoyan <>
Date:   2017-08-08T12:08:28Z

    Worker keeps driver on the host provided in the driverUrl. It may differ 
from the original value if the driver specified 

commit bd7399c1552768d54ab7c8cfc1dfeb27667c7f95
Author: Arseniy Tashoyan <>
Date:   2017-08-08T12:11:27Z

    When starting executor, take SPARK_PUBLIC_DNS into account for the driver 

commit e16d334eb58efe2375f4c85d77739ca3bacccecd
Author: Arseniy Tashoyan <>
Date:   2017-08-08T12:49:54Z

    Honor checkstyle


If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at or file a JIRA ticket
with INFRA.

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to