Found it, was security group setup problem ;( Andreas
Am Montag, den 10.03.2008, 16:49 +0100 schrieb Andreas Kostyrka: > Hi! > > I'm trying to setup a Hadoop 0.16.0 cluster on EC2/S3. (Manually, not > using the Hadoop AMIs) > > I've got the S3 based HDFS working, but I'm stumped when I try to get a > test job running: > > [EMAIL PROTECTED]:~/hadoop-0.16.0$ time bin/hadoop jar > contrib/streaming/hadoop-0.16.0-streaming.jar -mapper /tmp/test.sh -reducer > cat -input testlogs/* -output testlogs-output > additionalConfSpec_:null > null=@@@userJobConfProps_.get(stream.shipped.hadoopstreaming > packageJobJar: [/tmp/hadoop-hadoop/hadoop-unjar17969/] [] > /tmp/streamjob17970.jar tmpDir=null > 08/03/10 14:01:28 INFO mapred.FileInputFormat: Total input paths to process : > 152 > 08/03/10 14:02:58 INFO streaming.StreamJob: getLocalDirs(): > [/tmp/hadoop-hadoop/mapred/local] > 08/03/10 14:02:58 INFO streaming.StreamJob: Running job: job_200803101400_0001 > 08/03/10 14:02:58 INFO streaming.StreamJob: To kill this job, run: > 08/03/10 14:02:58 INFO streaming.StreamJob: > /home/hadoop/hadoop-0.16.0/bin/../bin/hadoop job > -Dmapred.job.tracker=ec2-67-202-58-97.compute-1.amazonaws.com:9001 -kill > job_200803101400_0001 > 08/03/10 14:02:58 INFO streaming.StreamJob: Tracking URL: > http://ip-10-251-75-165.ec2.internal:50030/jobdetails.jsp?jobid=job_200803101400_0001 > 08/03/10 14:02:59 INFO streaming.StreamJob: map 0% reduce 0% > > Furthermore, when I try to connect port 9001 on 10.251.75.165 via telnet from > the masterhost itself, it connects: > [EMAIL PROTECTED]:~/hadoop-0.16.0$ telnet 10.251.75.165 9001 > Trying 10.251.75.165... > Connected to 10.251.75.165. > Escape character is '^]'. > ^] > telnet> quit > Connection closed. > > When I try to do this from other VMs in my cluster, it just hangs. > (tcpdump on the masterhost shows no activity for tcp port 9001): > > [EMAIL PROTECTED]:~/hadoop-0.16.0$ telnet ip-10-251-75-165.ec2.internal 9001 > Trying 10.251.75.165... > > [EMAIL PROTECTED]:~/hadoop-0.16.0$ telnet ip-10-251-75-165.ec2.internal 22 > Trying 10.251.75.165... > Connected to ip-10-251-75-165.ec2.internal. > Escape character is '^]'. > SSH-2.0-OpenSSH_4.3p2 Debian-9 > ^] > telnet> quit > Connection closed. > > This is also shown when I connect port 50030, which shows 0 nodes ready to > process the job. > > Furthermore, the slaves show the following messages: > 2008-03-10 15:30:11,455 INFO org.apache.hadoop.ipc.RPC: Problem connecting to > server: ec2-67-202-58-97.compute-1.amazonaws.com/10.251.75.165:9001 > 2008-03-10 15:31:12,465 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: ec2-67-202-58-97.compute-1.amazonaws.com/10.251.75.165:9001. > Already tried 1 time(s). > 2008-03-10 15:32:13,475 INFO org.apache.hadoop.ipc.Client: Retrying connect > to server: ec2-67-202-58-97.compute-1.amazonaws.com/10.251.75.165:9001. > Already tried 2 time(s). > > Last but not least, here is my site conf: > <?xml version="1.0"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > <configuration> > > <property> > <name>fs.default.name</name> > <value>s3://lookhad</value> > <description>The name of the default file system. A URI whose > scheme and authority determine the FileSystem implementation. The > uri's scheme determines the config property (fs.SCHEME.impl) naming > the FileSystem implementation class. The uri's authority is used to > determine the host, port, etc. for a filesystem.</description> > </property> > > <property> > <name>fs.s3.awsAccessKeyId</name> > <value>2DFGTTFSDFDSZU5SDSD7S5202</value> > </property> > > <property> > <name>fs.s3.awsSecretAccessKey</name> > <value>RUWgsdfsd67SFDfsdflaI9Gjzfsd8789ksd2r1PtG</value> > </property> > > <property> > <name>mapred.job.tracker</name> > <value>ec2-67-202-58-97.compute-1.amazonaws.com:9001</value> > <description>The host and port that the MapReduce job tracker runs > at. If "local", then jobs are run in-process as a single map > and reduce task. > </description> > </property> > </configuration> > > The masternode listens not no localhost: > [EMAIL PROTECTED]:~/hadoop-0.16.0$ netstat -an | grep 9001 > tcp 0 0 10.251.75.165:9001 0.0.0.0:* LISTEN > > > Any ideas? My conclusions thus are: > > 1.) First, it's not a general connectivity problem, because I can connect > port 22 without any problems. > 2.) OTOH, on port 9001, inside the same group, the connectivity seems to be > limited. > 3.) All AWS docs tell me that VMs in one group have no firewalls in place. > > So what is happening here? Any ideas? > > Andreas
signature.asc
Description: Dies ist ein digital signierter Nachrichtenteil
