[
https://issues.apache.org/jira/browse/SPARK-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344746#comment-14344746
]
Sean Owen commented on SPARK-6135:
----------------------------------
Hm, what are you asking for here? why is {{sjs_1_2}} invalid? it may not be the
right one your workers need to contact, but how does Spark know that to check
for it? Or, if you mean it's "invalid", isn't that a network configuration
problem in your cluster?
There are {{SPARK_LOCAL_HOSTNAME}} and {{SPARK_LOCAL_IP}} too. Yes, you can
override with these. Does that solve your problem? or are you saying they need
to be documented to resolve this?
> No checks of illegal hostname when runing spark on yarn.
> --------------------------------------------------------
>
> Key: SPARK-6135
> URL: https://issues.apache.org/jira/browse/SPARK-6135
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.2.1
> Environment: spark on yarn cluster.
> Reporter: Xia Hu
> Labels: spark-submit, yarn-client
>
> I submit spark application to yarn cluster from a spark client machine, then
> I find this problem. When using spark on yarn-client model, driver is running
> in client, then the yarn cluster need to remotely connect to driver. But
> spark use InetAddress to read out the hostname of client machine,without
> checking if the hostname is legal or useful. So in my condition, some client
> machine have hostnames like "sjs_1_2", then this application fail because of
> cannot connect to driver on "sjs_1_2".
> I suppose there should be a check for if a hostname is legal, and if not,
> using the IP instead.
> And for this problem, I found an env. configuration "SPARK_LOCAL_HOSTNAME"
> can be used. if I set "SPARK_LOCAL_HOSTNAME" to be IP address in
> spark-env.sh, then this problem is solved. But it seems this configuration
> isn't introduced in any introductions or references and I found it when
> reading codes.
> But I still think a check of if hostname is illegal is needed,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
