[
https://issues.apache.org/jira/browse/SPARK-30850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-30850:
----------------------------------
Affects Version/s: (was: 2.4.5)
3.1.0
> RPC listener wildcard bindings (IPv6)
> -------------------------------------
>
> Key: SPARK-30850
> URL: https://issues.apache.org/jira/browse/SPARK-30850
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.1.0
> Reporter: Ben Steadman
> Priority: Major
>
> Running Spark Standalone on IPv6-first infrastructure requires that RPC
> listeners bind IPv6 sockets e.g. {{:::7077}} instead of {{127.0.0.1:7077}}
> whilst advertising a public DNS name using {{SPARK_PUBLIC_DNS}}. Currently,
> the various RPC listeners can be configured with a *hostname/IPv4 address*
> (enforced via {{Utils.checkHost}}) only e.g. via
> {{spark.driver.bindAddress}}, {{spark.driver.host}}, {{SPARK_LOCAL_HOST}}
> e.t.c. which are eventually resolved using {{InetSocketAddress}} in
> [{{TransportServer}}|https://github.com/apache/spark/blob/84bd8085c74b5292cdccaa287023e39703062b07/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L136]
> Initial experimentation led to a *hack* (absolutely not a real solution) to
> force this behaviour by setting {{bindAddress = null}} when the different
> `RPCEnv`s are created ([link to
> diff|https://gist.github.com/SteadBytes/b206335c8e2429486a13421b7cfcdc88]).
> Since {{TransportServer}} does support the wildcard variant of
> {{InetSocketAddress}} if {{hostToBind = null}}
> [here|https://github.com/apache/spark/blob/84bd8085c74b5292cdccaa287023e39703062b07/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L137]
> this should be possible.
> Adding additional options similar to {{spark.driver.bindAddress}} but for
> other RPC components (master, driver etc) and allowing these to have
> wildcard/IPv6 values might be an option that avoids breaking existing
> configurations?
> I'm prepared to contribute the work required and upstream any changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
