Arseniy Tashoyan created SPARK-21668:

             Summary: Ability to run driver programs within a container
                 Key: SPARK-21668
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 2.2.0, 2.1.1
            Reporter: Arseniy Tashoyan
            Priority: Minor

When a driver program in Client mode runs in a Docker container, it binds to 
the IP address of the container, not the host machine. This container IP 
address is accessible only within the host machine, it is inaccessible for 
master and worker nodes.
For example, the host machine has IP address When Docker 
machine starts a container, it places it to a special bridged network and 
assigns it an IP address like All Spark nodes belonging to the network cannot access the bridged network with the container. 
Therefore, the driver program is not able to communicate with the Spark cluster.
Spark already provides SPARK_PUBLIC_DNS environment variable for this purpose. 
However, in this scenario setting SPARK_PUBLIC_DNS to the host machine IP 
address does not work.

Topic on StackOverflow: 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to