GitHub user vanzin opened a pull request:

    https://github.com/apache/spark/pull/15120

    [SPARK-4563][core] Allow driver to advertise a different network address.

    The goal of this feature is to allow the Spark driver to run in an
    isolated environment, such as a docker container, and be able to use
    the host's port forwarding mechanism to be able to accept connections
    from the outside world.
    
    The change is restricted to the driver: there is no support for achieving
    the same thing on executors (or the YARN AM for that matter). Those still
    need full access to the outside world so that, for example, connections
    can be made to an executor's block manager.
    
    The core of the change is simple: add a new configuration that tells what's
    the address the driver should bind to, which can be different than the 
address
    it advertises to executors (spark.driver.host). Everything else is plumbing
    the new configuration where it's needed.
    
    To use the feature, the host starting the container needs to set up the
    driver's port range to fall into a range that is being forwarded; this
    required the block manager port to need a special configuration just for
    the driver, which falls back to the existing spark.blockManager.port when
    not set. This way, users can modify the driver settings without affecting
    the executors; it would theoretically be nice to also have different
    retry counts for driver and executors, but given that docker (at least)
    allows forwarding port ranges, we can probably live without that for now.
    
    Because of the nature of the feature it's kinda hard to add unit tests;
    I just added a simple one to make sure the configuration works.
    
    This was tested with a docker image running spark-shell with the following
    command:
    
     docker blah blah blah \
       -p 38000-38100:38000-38100 \
       [image] \
       spark-shell \
         --num-executors 3 \
         --conf spark.shuffle.service.enabled=false \
         --conf spark.dynamicAllocation.enabled=false \
         --conf spark.driver.host=[host's address] \
         --conf spark.driver.port=38000 \
         --conf spark.driver.blockManager.port=38020 \
         --conf spark.ui.port=38040
    
    Running on YARN; verified the driver works, executors start up and listen
    on ephemeral ports (instead of using the driver's config), and that caching
    and shuffling (without the shuffle service) works. Clicked through the UI
    to make sure all pages (including executor thread dumps) worked. Also tested
    apps without docker, and ran unit tests.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/vanzin/spark SPARK-4563

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/15120.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #15120
    
----
commit 40f23a48a8eca3a7d20a193a7998413090ea695c
Author: Marcelo Vanzin <van...@cloudera.com>
Date:   2016-09-15T19:46:10Z

    [SPARK-4563][core] Allow driver to advertise a different network address.
    
    The goal of this feature is to allow the Spark driver to run in an
    isolated environment, such as a docker container, and be able to use
    the host's port forwarding mechanism to be able to accept connections
    from the outside world.
    
    The change is restricted to the driver: there is no support for achieving
    the same thing on executors (or the YARN AM for that matter). Those still
    need full access to the outside world so that, for example, connections
    can be made to an executor's block manager.
    
    The core of the change is simple: add a new configuration that tells what's
    the address the driver should bind to, which can be different than the 
address
    it advertises to executors (spark.driver.host). Everything else is plumbing
    the new configuration where it's needed.
    
    To use the feature, the host starting the container needs to set up the
    driver's port range to fall into a range that is being forwarded; this
    required the block manager port to need a special configuration just for
    the driver, which falls back to the existing spark.blockManager.port when
    not set. This way, users can modify the driver settings without affecting
    the executors; it would theoretically be nice to also have different
    retry counts for driver and executors, but given that docker (at least)
    allows forwarding port ranges, we can probably live without that for now.
    
    Because of the nature of the feature it's kinda hard to add unit tests;
    I just added a simple one to make sure the configuration works.
    
    This was tested with a docker image running spark-shell with the following
    command:
    
     docker blah blah blah \
       -p 38000-38100:38000-38100 \
       [image] \
       spark-shell \
         --num-executors 3 \
         --conf spark.shuffle.service.enabled=false \
         --conf spark.dynamicAllocation.enabled=false \
         --conf spark.driver.host=[host's address] \
         --conf spark.driver.port=38000 \
         --conf spark.driver.blockManager.port=38020 \
         --conf spark.ui.port=38040
    
    Running on YARN; verified the driver works, executors start up and listen
    on ephemeral ports (instead of using the driver's config), and that caching
    and shuffling (without the shuffle service) works. Clicked through the UI
    to make sure all pages (including executor thread dumps) worked. Also tested
    apps without docker, and ran unit tests.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to