Hi Andrew
Thanks Andrew for your suggestion. I updated the hdfs-site on server side
and also on client side to use hostname instead of IP as mentioned here =
http://rainerpeter.wordpress.com/2014/02/12/connect-to-hdfs-running-in-ec2-using-public-ip-addresses/
. Now, I could see that the client is
I am trying to run Spark on YARN. I have a hadoop 2.2 cluster (YARN +
HDFS) in EC2. Then, I compiled Spark using Maven with 2.2 hadoop profiles.
Now am trying to run the example Spark job . (In Yarn-cluster mode).
From my *local machine. *I have setup HADOOP_CONF_DIR environment variable
Hi Praveen,
Yes, the fact that it is trying to use a private IP from outside of the
cluster is suspicious.
My guess is that your HDFS is configured to use internal IPs rather than
external IPs.
This means even though the hadoop confs on your local machine only use
external IPs,
the
(Also, an easier workaround is to simply submit the application from within
your
cluster, thus saving you all the manual labor of reconfiguring everything
to use
public hostnames. This may or may not be applicable to your use case.)
2014-06-19 14:04 GMT-07:00 Andrew Or and...@databricks.com:
