Ben Steadman created SPARK-30850:
------------------------------------
Summary: RPC listener wildcard bindings (IPv6)
Key: SPARK-30850
URL: https://issues.apache.org/jira/browse/SPARK-30850
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.4.5
Reporter: Ben Steadman
Running Spark Standalone on IPv6-first infrastructure requires that RPC
listeners bind IPv6 sockets e.g. {{:::7077}} instead of {{127.0.0.1:7077}}
whilst advertising a public DNS name using {{SPARK_PUBLIC_DNS}}. Currently, the
various RPC listeners can be configured with a *hostname/IPv4 address* only
e.g. via {{spark.driver.bindAddress}}, {{spark.driver.host}},
{{SPARK_LOCAL_IP}} e.t.c. which are eventually resolved using
{{InetSocketAddress}} in
[{{TransportServer}}|https://github.com/apache/spark/blob/84bd8085c74b5292cdccaa287023e39703062b07/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L136]
Initial experimentation led to a *hack* (not a real solution) to force this
behaviour by setting {{bindAddress = null}} when the different `RPCEnv`s are
created ([link to
diff|https://gist.github.com/SteadBytes/b206335c8e2429486a13421b7cfcdc88]).
Since {{TransportServer}} does support the wildcard variant of
{{InetSocketAddress}} if {{hostToBind = null}}
[here|https://github.com/apache/spark/blob/84bd8085c74b5292cdccaa287023e39703062b07/common/network-common/src/main/java/org/apache/spark/network/server/TransportServer.java#L137]
this should be possible.
Would using the wildcard as a default instead of the local hostname be an
option?
I'm prepared to contribute the work required and upstream any changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
