Santiago M. Mola created SPARK-4799:
---------------------------------------
Summary: Spark should not rely on local host being resolvable on
every node
Key: SPARK-4799
URL: https://issues.apache.org/jira/browse/SPARK-4799
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.1.0
Environment: Tested a Spark+Mesos cluster on top of Docker to
reproduce the issue.
Reporter: Santiago M. Mola
Spark fails when a node hostname is not resolvable by other nodes.
See an example trace:
{code}
14/12/09 17:02:41 ERROR SendingConnection: Error connecting to
27e434cf36ac:35093
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:127)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644)
at
org.apache.spark.network.SendingConnection.connect(Connection.scala:299)
at
org.apache.spark.network.ConnectionManager.run(ConnectionManager.scala:278)
at
org.apache.spark.network.ConnectionManager$$anon$4.run(ConnectionManager.scala:139)
{code}
The relevant code is here:
https://github.com/apache/spark/blob/bcb5cdad614d4fce43725dfec3ce88172d2f8c11/core/src/main/scala/org/apache/spark/network/nio/ConnectionManager.scala#L170
{code}
val id = new ConnectionManagerId(Utils.localHostName,
serverChannel.socket.getLocalPort)
{code}
This piece of code should use the host IP with Utils.localIpAddress or a method
that acknowleges user settings (e.g. SPARK_LOCAL_IP). Since I cannot think
about a use case for using hostname here, I'm creating a PR with the former
solution, but if you think the later is better, I'm willing to create a new PR
with a more elaborate fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
