Xia Hu created SPARK-6135:
-----------------------------
Summary: No checks of illegal hostname when runing spark on yarn.
Key: SPARK-6135
URL: https://issues.apache.org/jira/browse/SPARK-6135
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 1.2.1
Environment: spark on yarn cluster.
Reporter: Xia Hu
I submit spark application to yarn cluster from a spark client machine, then I
find this problem. When using spark on yarn-client model, driver is running in
client, then the yarn cluster need to remotely connect to driver. But spark use
InetAddress to read out the hostname of client machine,without checking if the
hostname is legal or useful. So in my condition, some client machine have
hostnames like "sjs_1_2", then this application fail because of cannot connect
to driver on "sjs_1_2".
I suppose there should be a check for if a hostname is legal, and if not, using
the IP instead.
And for this problem, I found an env. configuration "SPARK_LOCAL_HOSTNAME" can
be used. if I set "SPARK_LOCAL_HOSTNAME" to be IP address in spark-env.sh,
then this problem is solved. But it seems this configuration isn't introduced
in any introductions or references and I found it when reading codes.
But I still think a check of if hostname is illegal is needed,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
