Hi Guys,
I am having trouble connecting through the whirr proxy. I am using
whirr 0.8.0 and hadoop 0.23.3.
I can launch my cluster and ssh into nodes in it, but when I try to us
the proxy I get:
channel 2: open failed: connect failed: Connection refused.
At one point the proxy was working for me, but now it no longer does (no
idea what happened).
To set my hadoop configuration options I use:
export HADOOP_CONF_DIR=~/.whirr/myhadoopcluster2/
, where myhadoopcluster2 is the name of my cluster.
This is my work flow order:
<In shell 1>
./whirr launch-cluster --config ../recipes/hadoop.properties
/home/lee/.whirr/myhadoopcluster2/hadoop-proxy.sh
<In shell 2>
export HADOOP_CONF_DIR=~/.whirr/myhadoopcluster2/
hadoop fs -ls
The output in shell 1 is:
channel 2: open failed: connect failed: Connection refused
...
channel 2: open failed: connect failed: Connection refused
The output in shell 2 is:
2012-10-07 16:43:23,893 WARN conf.Configuration
(Configuration.java:<clinit>(470)) - DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
core-site.xml, mapred-site.xml and hdfs-site.xml to override properties
of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
2012-10-07 16:43:25,592 INFO ipc.Client
(Client.java:handleConnectionFailure(670)) - Retrying connect to server:
ec2-50-16-49-108.compute-1.amazonaws.com/50.16.49.108:8020. Already
tried 0 time(s).
...
2012-10-07 16:43:35,300 INFO ipc.Client
(Client.java:handleConnectionFailure(670)) - Retrying connect to server:
ec2-50-16-49-108.compute-1.amazonaws.com/50.16.49.108:8020. Already
tried 9 time(s).
ls: Failed on local exception: java.net.SocketException: Malformed reply
from SOCKS server; Host Details : local host is:
"emergent-pattern/127.0.1.1"; destination host is:
""ec2-50-16-49-108.compute-1.amazonaws.com":8020
This is my hadoop.properties file:
whirr.service-name=hadoop
whirr.cluster-name=myhadoopcluster2
whirr.cluster-user=${sys:user.name}
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,5
hadoop-datanode+hadoop-tasktracker
whirr.provider=aws-ec2
whirr.hadoop.version=0.23.3
whirr.identity=${env:AWS_ACCESS_KEY_ID}
whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
whirr.hardware-id=t1.micro
whirr.private-key-file=/home/lee/.ssh/id_rsa_whirr
whirr.public-key-file=/home/lee/.ssh/id_rsa_whirr.pub
This is my hadoop-site.xml file:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>hadoop.job.ugi</name>
<value>root,root</value>
</property>
<property>
<name>mapred.job.tracker</name>
<value>ec2-50-16-49-108.compute-1.amazonaws.com:8021</value>
</property>
<property>
<name>hadoop.socks.server</name>
<value>localhost:6666</value>
</property>
<property>
<name>fs.s3n.awsAccessKeyId</name>
<value> I DELETED THIS FOR THE MAIL LIST</value>
</property>
<property>
<name>dfs.client.use.legacy.blockreader</name>
<value>true</value>
</property>
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>I DELETED THIS FOR THE MAIL LIST</value>
</property>
<property>
<name>fs.s3.awsAccessKeyId</name>
<value>I DELETED THIS FOR THE MAIL LIST</value>
</property>
<property>
<name>hadoop.rpc.socket.factory.class.default</name>
<value>org.apache.hadoop.net.SocksSocketFactory</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://ec2-50-16-49-108.compute-1.amazonaws.com:8020/</value>
</property>
<property>
<name>fs.s3n.awsSecretAccessKey</name>
<value>I DELETED THIS FOR THE MAIL LIST</value>
</property>
</configuration>
Any advice in resolving this issue?
Thanks for any help!
