Everything started working after I added the following line to hadoop-env.sh
export HADOOP_SSH_OPTS="$HADOOP_SSH_OPTS -i /mnt/myVol/KeyPairName.pem " Thanks for your help. On Wed, Nov 25, 2009 at 10:25 AM, Vaibhav Puranik <[email protected]>wrote: > On master, in .ssh folder, there should be a file id_dsa.pub. This is > master's public key. > > Contents of this file should be copied to slave's authorized_keys file. > > Regards, > Vaibhav > > On Wed, Nov 25, 2009 at 9:45 AM, Something Something < > [email protected]> wrote: > > > Can you please explain what you mean by "add master's public key to > slave's > > authorized_keys"? I have a feeling I am not doing this correctly. What > I > > did is this: > > > > 1) On Master, opened ~/.ssh/authorized_key and copied lines that look > like > > this... > > > > ssh-dss > > > > > AAAAB3NzaC1kc3MAAACBAKJOife8p+6vjJ/jODSWaBgyawN7XWmasasasb0cVqhe1WOpnJQpXuJeqsaBaZYHYxj8P74m1rOIdN9lSPAz6gEjHo4ft8UM7ZVExRC6HDM7JdfdferhwRgQSVI9ruTLlssO9i1D17mBnLotsgy92CyS7bsanM25nRqhmvqkX7tuKp66nlAAAAFQCnbwPn3v+ttorRZ4OijkAKoJaVFQAAAIAGVyZGwkm5DwYvige0VYhXZns8qSvW15dwSrS1a6k41q3C91AQZylvJYDnBjNGnWwUHfyWpyruzVCOskavEmsl6FcouU5Av1zLRT1sk4llUzZxXFtv6gzRBq2+aiCcoex3O+RRJbHa89cMLy6jeZOCFahwVz6IyHcQ0b7gZjQgFQAAAIBA4vzSDy5oN27Ji3mF9ZklsSmlJJrrJxGJfOn6/BK5tyvBijLy6tEJHrH3dY8TEJUp2V9RINKwcsy5LdqyI5gJRWoM8WdNzmUoIOLAW0HIVl2MldKKqP/ctQwGI94EjykSUYVt/IAvUcjFuIrtiGrMDjsqfm7h1Gi2ZtDQ== > > r...@domu-12-31-38-00-39-c8 > > > > > > to the end of ~/.ssh/authorized_key on Slave. > > > > Something tells me there's a better way to do this. Is there? > > > > > > On Wed, Nov 25, 2009 at 9:21 AM, Vaibhav Puranik <[email protected]> > > wrote: > > > > > Actually I was wrong about adding keypair. It should be resolved by > > adding > > > master's public key to slave's authorized_keys. Amazon puts your > keypair > > in > > > authroized_keys automatically. I missed that part. > > > > > > Regards, > > > Vaibhav > > > > > > On Wed, Nov 25, 2009 at 8:38 AM, Something Something < > > > [email protected]> wrote: > > > > > > > Me again :( > > > > > > > > I decided to start fresh so that I can document the steps, but now I > > > can't > > > > even go further than I did before :( > > > > > > > > Please let me know what step I am missing: > > > > > > > > 1) Launched 2 instances on AWS console - one for Master, one for > > Slave. > > > > > > > > 2) In one command window, connected to instance 1: > > > > ssh -i MyKeyPair.pem <instance 1> > > > > (Got 'authenticity' warning.. typed 'yes') > > > > > > > > This is my "Master" window. > > > > > > > > 3) Opened another command window & connected to instance 2: > > > > ssh -i MyKeyPair.pem <instance 2> > > > > (Got 'authenticity' warning.. typed 'yes') > > > > > > > > This is my "Slave" window. > > > > > > > > 4) Opened another command window and 'scp' the key pair (from my > local > > > > machine) to instance 1 & instance 2. > > > > scp -i MyKeyPair.pem MyKeyPair.pem <master instance>:/tmp > > > > scp -i MyKeyPair.pem MyKeyPair.pem <slave instance>:/tmp > > > > > > > > 5) On BOTH (Master & Slave) added Key Pair to 'authorized_key' > > > > > > > > cat /tmp/MyKeyPair.pem >> ~/.ssh/authorized_keys > > > > > > > > 6) To setup passphraseless ssh ran these on BOTH - Master & Slave: > > > > ssh localhost (Got 'authenticiy' message. Typed 'yes'. Got > > Permission > > > > denied (public key) > > > > ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa > > > > cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys > > > > > > > > 7) In order to ssh from Master to Slave (without entering KeyPair), > > > copied > > > > the key (that starts with 'sh-dss') from Slave's 'authorized_key' > file > > to > > > > Master's 'authoried_key' file. > > > > > > > > Now, every time I try (from Master): > > > > > > > > ssh <slave's ip address> > > > > > > > > I get - "Permission denied (publickey). > > > > > > > > But, when I type: > > > > > > > > ssh -i /tmp/MyKeyPair.pem <slave's ip address> > > > > > > > > It works. > > > > > > > > I thought step #5 above should have resolved this. What am I doing > > > wrong? > > > > I believe, in order for me to start the cluster, I should be able to > > > 'ssh' > > > > to slave without using key pair, correct? > > > > > > > > > > > > Thanks for your help. > > > > > > > > > > > > On Tue, Nov 24, 2009 at 2:58 PM, Vaibhav Puranik <[email protected] > > > > > > wrote: > > > > > > > > > Install a program called Jps. > > > > > > > > > > It's a ps for java processes. execute jps, you should see the > > processes > > > > (if > > > > > they are running). > > > > > > > > > > Also make sure you check your logs. > > > > > > > > > > Regards, > > > > > Vaibhav > > > > > > > > > > On Tue, Nov 24, 2009 at 2:06 PM, Something Something < > > > > > [email protected]> wrote: > > > > > > > > > > > Awesome. Thanks, Vaibhav. Made progress. No error messages > from > > > > > > ./all-start.sh > > > > > > > > > > > > I will test to see if the cluster is deployed properly. > > > > > > > > > > > > Question: On the slave, when I do: > > > > > > > > > > > > ps -eaf | grep 'hadoop' > > > > > > > > > > > > I don't see any processes. Shouldn't I see 2 processes - one for > > > > > Datanode > > > > > > and the other for TaskTracker? > > > > > > > > > > > > Please let me know. Thanks again. > > > > > > > > > > > > > > > > > > On Tue, Nov 24, 2009 at 1:44 PM, Vaibhav Puranik < > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > > > Yes, I pretty much meant that. > > > > > > > > > > > > > > Add your master's public key to authorized_keys on slaves too. > > > > > > > > > > > > > > Regards, > > > > > > > Vaibhav > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 24, 2009 at 1:38 PM, Something Something < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Sorry.. but not sure what you mean by... "In order to achieve > > > that > > > > > you > > > > > > > need > > > > > > > > to add the keypair you are using to your > > > > > > > > auhtorized_keys file in the home dir/.ssh folder." > > > > > > > > > > > > > > > > > > > > > > > > I tried this.... > > > > > > > > > > > > > > > > cat KeyPair.pem >> /root/.ssh/authorized_keys > > > > > > > > > > > > > > > > Is that what you meant by "add the keypair to > authorized_keys"? > > > > > > > > > > > > > > > > Still can't ssh from Slave to Master. It says..... > Permission > > > > > denied > > > > > > > > (publickey). > > > > > > > > > > > > > > > > > > > > > > > > Sorry, I am a newbie when it comes to such environment > issues. > > > > > Thanks > > > > > > > for > > > > > > > > your patience. It's greatly appreciated. > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 24, 2009 at 11:16 AM, Vaibhav Puranik < > > > > > [email protected] > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > You need to be able to SSH without specifying the keypair > > from > > > a > > > > > > slave > > > > > > > to > > > > > > > > > master (I think reverse it true too, but I am not sure). > > > > > > > > > > > > > > > > > > In order to achieve that you need to add the keypair you > are > > > > using > > > > > > to > > > > > > > > your > > > > > > > > > auhtorized_keys file in the home dir/.ssh folder. > > > > > > > > > > > > > > > > > > Once you setup passphraseless ssh, it should start working. > > > > > > Furthrmore, > > > > > > > > > after you set this up, try to ssh manully (without key) and > > if > > > it > > > > > > asks > > > > > > > > you > > > > > > > > > the following message - say yes: > > > > > > > > > > > > > > > > > > The authenticity of host > > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com > > (147.127.186.243)' > > > > > can't > > > > > > > > > be established. > > > > > > > > > RSA key fingerprint is > > > > > > 4b:63:e2:23:16:6f:b6:99:de:34:f6:9b:f5:55:73:8b. > > > > > > > > > Are you sure you want to continue connecting (yes/no)? yes > > > > > > > > > Warning: Permanently added > > > > > > > > > 'ec2-147-127-186-243.compute-1.amazonaws.com > > ,147.127.186.243' > > > > > > > > > (RSA) to the list of known hosts. > > > > > > > > > > > > > > > > > > In other words, you have to make sure that the machine you > > are > > > > > trying > > > > > > > to > > > > > > > > > ssh > > > > > > > > > is in your known_hosts file. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > Vaibhav > > > > > > > > > > > > > > > > > > On Tue, Nov 24, 2009 at 10:54 AM, Something Something < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > Thanks, Vaibhav. That was helpful. I am able to scp > from > > my > > > > > local > > > > > > > > > machine > > > > > > > > > > after I opened port 22 under my security group. > > > > > > > > > > > > > > > > > > > > But my cluster is still not starting as expected. I > > mean... > > > on > > > > > my > > > > > > > > Master > > > > > > > > > > when I run: > > > > > > > > > > > > > > > > > > > > ./start-all.sh > > > > > > > > > > > > > > > > > > > > I get messages saying... > > > > > > > > > > > > > > > > > > > > 10.252.xxx.xxx: Permission denied (publickey). > > > > > > > > > > > > > > > > > > > > where 10.252.xxx.xxx is the ip address of my Slave. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I can ssh from my Slave to master if I do this: > > > > > > > > > > > > > > > > > > > > ssh -i <keypairname>.pem <Master's ip address> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any suggestions? Thanks for your help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 23, 2009 at 8:10 AM, Vaibhav Puranik < > > > > > > [email protected] > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > In order to use scp from your machine to an ec2 > instance > > > you > > > > > need > > > > > > > to > > > > > > > > > use > > > > > > > > > > > scp > > > > > > > > > > > -i and give it the path of your keypair. > > > > > > > > > > > > > > > > > > > > > > scp -i /path/my/ec2/keypair <souce> <destination> > > > > > > > > > > > > > > > > > > > > > > Secondly in order to configure an ec2 cluster for > Hadoop, > > > you > > > > > > need > > > > > > > to > > > > > > > > > > make > > > > > > > > > > > a > > > > > > > > > > > security group (let's call it hadoop) and give hadoop > > > access > > > > to > > > > > > > > hadoop. > > > > > > > > > > > > > > > > > > > > > > You can read more about security groups here - > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://somic.org/2009/09/21/security-groups-most-underappreciated-feature-of-amazon-ec2/ > > > > > > > > > > > > > > > > > > > > > > You also need to make sure that you can ssh from all > the > > > > slaves > > > > > > to > > > > > > > > > master > > > > > > > > > > > without specifying password. > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > Vaibhav Puranik > > > > > > > > > > > GumGum > > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:36 AM, Something Something < > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > Seems like I am not explaining my problem correctly. > > > > > > > > > > > > > > > > > > > > > > > > My Key Pair file is on my machine at work which is > > behind > > > a > > > > > > corp > > > > > > > > > > > firewall. > > > > > > > > > > > > As such I can't 'scp' from the ec2 instance to my > > local > > > > > > machine > > > > > > > at > > > > > > > > > > work > > > > > > > > > > > to > > > > > > > > > > > > 'get' the file. So I need a way to 'send' a file > from > > my > > > > > > machine > > > > > > > > to > > > > > > > > > > the > > > > > > > > > > > > ec2 > > > > > > > > > > > > instance. I tried using 'scp' (from my machine at > > work) > > > to > > > > > the > > > > > > > ec2 > > > > > > > > > > > machine > > > > > > > > > > > > but it says "Permission denied". Does this make > sense? > > > > > > > > > > > > > > > > > > > > > > > > May be I need to use the command line tools for EC2. > I > > > am > > > > > > > looking > > > > > > > > > into > > > > > > > > > > > > those right now, but if there's a better/easier way, > > > please > > > > > let > > > > > > > me > > > > > > > > > > know. > > > > > > > > > > > > > > > > > > > > > > > > Thanks once again for your help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:19 AM, Jeff Zhang < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > The ssh is installed on ec2 by default, otherwise > you > > > > have > > > > > no > > > > > > > way > > > > > > > > > to > > > > > > > > > > > > login > > > > > > > > > > > > > to ec2 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jeff Zhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Nov 22, 2009 at 8:09 AM, Something > Something > > < > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > As per Wikipedia ( > > > > > http://en.wikipedia.org/wiki/Secure_copy > > > > > > ) > > > > > > > > "It > > > > > > > > > > > does > > > > > > > > > > > > so > > > > > > > > > > > > > > by > > > > > > > > > > > > > > connecting to the host using SSH and there > executes > > > an > > > > > SCP > > > > > > > > server > > > > > > > > > > > > (scp)". > > > > > > > > > > > > > > > > > > > > > > > > > > > > So if SSH isn't working SCP won't work, either. > In > > > any > > > > > > case > > > > > > > I > > > > > > > > > > tried > > > > > > > > > > > to > > > > > > > > > > > > > scp > > > > > > > > > > > > > > but getting "Permission denied (public key)". > > > > > > > > > > > > > > > > > > > > > > > > > > > > Any other ideas? Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 6:12 PM, Jeff Zhang < > > > > > > > [email protected]> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > You should scp the key-pair to EC2 machine > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Jeff Zhang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Nov 21, 2009 at 4:57 PM, Something > > > Something > > > > < > > > > > > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Trying to start a Hadoop cluster on EC2. > (Yes, > > > > > > > Cloudera's > > > > > > > > > > > > > distribution > > > > > > > > > > > > > > > > works well, but trying to do this myself so I > > can > > > > > learn > > > > > > > > > what's > > > > > > > > > > > > > > happening > > > > > > > > > > > > > > > > behind the scene.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I have a Master & a Slave. When I start HDFS > > on > > > > the > > > > > > > > master, > > > > > > > > > I > > > > > > > > > > > get > > > > > > > > > > > > a > > > > > > > > > > > > > > > > message > > > > > > > > > > > > > > > > saying "10.xxx.xxx.xxx (Permission denied)" - > > > where > > > > > > > 10.xxx > > > > > > > > is > > > > > > > > > > IP > > > > > > > > > > > > > > address > > > > > > > > > > > > > > > of > > > > > > > > > > > > > > > > the slave. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Basic problem (I think) is that I can't ssh > > from > > > > the > > > > > > > master > > > > > > > > > EC2 > > > > > > > > > > > > > > instance > > > > > > > > > > > > > > > to > > > > > > > > > > > > > > > > the Slave EC2 instance. What's the best way > to > > > fix > > > > > it? > > > > > > > I > > > > > > > > > > think > > > > > > > > > > > I > > > > > > > > > > > > > need > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > "Key Pair" file on my master. I have a key > > pair > > > on > > > > > my > > > > > > > > local > > > > > > > > > > > > machine, > > > > > > > > > > > > > > but > > > > > > > > > > > > > > > > how do I transfer it to the EC2 machine? (I > > > know, > > > > I > > > > > > > know, > > > > > > > > I > > > > > > > > > > > > agree.. > > > > > > > > > > > > > I > > > > > > > > > > > > > > am > > > > > > > > > > > > > > > > dumb :) Should I FTP it? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Please help. Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
