I also use scp and remote ssh to push configuration files | start-stop-restart respective daemons in the cluster. This can simply resolution of every node in the cluster having an updated address of the new NN. But what about slave file on the NN?
If you have 20 DN/TTs in a cluster, you will have to go inside each and every one to get internal ip-address or copy external ip-addresses from EC2 web-inteface for each node and then replace the list of new nodes in your slave file. Is this how you do it? Rgds, AK-47 From: yinghua hu [mailto:[email protected]] Sent: Wednesday, November 14, 2012 12:03 PM To: [email protected] Subject: Re: shutting down instance Hi,Andy Yes. I shutdown EC2 instance every day to save some money for my company. I use shell scripts to reduce manual editing as much as possible. I just edit core-site.xml and yarn-site.xml in single node and then scp it to all the other nodes. This can be automated. You can add to /etc/ssh/ssh-config the two lines below StrictHostKeyChecking no UserKnownHostsFile /dev/null so that you will not need to type "yes" when connecting to an unknown host. So I re-edited four files core-site.xml and yarn-site.xml, slaves and master on namenode everyday. It take only a few minutes. I did not use Zookeeper to deal with this. Please let me know if you have better approach. On Wed, Nov 14, 2012 at 6:34 AM, Kartashov, Andy <[email protected]<mailto:[email protected]>> wrote: Yinghau, Last week you mentioned that you are running your cluster on EC2 that you shut them down instance(s) over the weekend. QTE Since I shutdown the EC2 instance every night, I thought that using 'master','slave1','slave2' will save typing after the full host name change with reboot. UNQTE Whenever I re-start an instance on Amazon cloud my both internal and external ip addresses for the instance change... ? p.s. Has anyone successfully configured Zookeeper to curb this matter? From: yinghua hu [mailto:[email protected]<mailto:[email protected]>] Sent: Sunday, November 11, 2012 2:06 PM To: [email protected]<mailto:[email protected]> Subject: Re: error running pi program Hi, Steve These are very helpful suggestions! Thanks! I will try using SSH proxy and see how it goes. On Sun, Nov 11, 2012 at 10:02 AM, Steve Loughran <[email protected]<mailto:[email protected]>> wrote: On 10 November 2012 18:30, yinghua hu <[email protected]<mailto:[email protected]>> wrote: Thanks Steve, Your suggestion is very helpful. But I only have terminal access to the EC2 NN. To run internet browser there will be a lot of more work. the SSH proxy sets up something that appears to be a SOCKs proxy -but which routes to the far end; you set your browser to go through this proxy (foxyproxy is a good firefox plugin here), and then your browser has access to all the machines see http://www.linuxjournal.com/content/use-ssh-create-http-proxy Since my cluster will be shut down at night and weekend. I will prefer to have an IP that does not change. How do you think of using EC2 Elastic IP? not tried that; I have used dyndns.org<http://dyndns.org> to self register an IP address that changes every time you bring up an instance, but whose browser bookmarks can remain fixed -- Regards, Yinghua NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel -- Regards, Yinghua NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
