GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/17821
[SPARK-20529][Core]Allow worker and master work with a proxy server
## What changes were proposed in this pull request?
In the current codes, when worker connects to master, master will send its
address to the worker. Then worker will save this address and use it to
reconnect in case of failure. However, sometimes, this address is not correct.
If there is a proxy between master and worker, the address master sent is not
the address of proxy.
In this PR, the master address used by the worker will be sent to the
master, then master just replies this address back, worker will use this
address to reconnect in case of failure. In other words, the worker will use
the config master address set in the worker side if possible rather than the
master address set in the master side.
There is still one potential issue though. When a master is restarted or
takes over leadership, the work will use the address sent from the master to
connect. If there is still a proxy between master and worker, the address may
be wrong. However, there is no way to figure it out just in the worker.
## How was this patch tested?
The new added unit test.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark SPARK-20529
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/17821.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #17821
----
commit 8ded9b197cc7ef3cdb32858da385cf9f900deb7d
Author: Shixiong Zhu <[email protected]>
Date: 2017-04-28T23:04:48Z
Fix SPARK-20529
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
