what about DNS? if you have 2 computers (nn and dn) how nn knows dn ip? The script puts only this computer ip to /etc/hosts
On Thu, Mar 5, 2015 at 6:39 PM, max scalf <[email protected]> wrote: > Here is a easy way to go about assigning static name to your ec2 > instance. When you get the launch an EC2-instance from aws console when > you get to the point of selecting VPC, ip address screen there is a screen > that says "USER DATA"...put the below in with appropriate host name(change > CHANGE_HOST_NAME_HERE to whatever you want) and that should be able to get > you static name. > > #!/bin/bash > > HOSTNAME_TAG=CHANGE_HOST_NAME_HERE > cat > /etc/sysconfig/network << EOF > NETWORKING=yes > NETWORKING_IPV6=no > HOSTNAME=${HOSTNAME_TAG} > EOF > > IP=$(curl http://169.254.169.254/latest/meta-data/local-ipv4) > echo "${IP} ${HOSTNAME_TAG}.localhost ${HOSTNAME_TAG}" >> /etc/hosts > > echo ${HOSTNAME_TAG} > /proc/sys/kernel/hostname > service network restart > > > Also note i was able to do this on couple of spot instance for cheap > price, only thing is once you shut it down or someone outbids you, you > loose that instance but its easy/cheap to play around with.... and i have > used couple of m3.medium for my NN/SNN and couple of them for data nodes... > > On Thu, Mar 5, 2015 at 7:19 PM, Jonathan Aquilina <[email protected] > > wrote: > >> I dont know how you would do that to be honest. With EMR you have >> destinctions master core and task nodes. If you need to change >> configuration you just ssh into the EMR master node. >> >> >> >> --- >> Regards, >> Jonathan Aquilina >> Founder Eagle Eye T >> >> On 2015-03-06 02:11, Alexander Pivovarov wrote: >> >> What is the easiest way to assign names to aws ec2 computers? >> I guess computer need static hostname and dns name before it can be used >> in hadoop cluster. >> On Mar 5, 2015 4:36 PM, "Jonathan Aquilina" <[email protected]> >> wrote: >> >>> When I started with EMR it was alot of testing and trial and error. >>> HUE is already supported as something that can be installed from the AWS >>> console. What I need to know is if you need this cluster on all the time or >>> this is goign ot be what amazon call a transient cluster. Meaning you fire >>> it up run the job and tear it back down. >>> >>> >>> >>> --- >>> Regards, >>> Jonathan Aquilina >>> Founder Eagle Eye T >>> >>> On 2015-03-06 01:10, Krish Donald wrote: >>> >>> Thanks Jonathan, >>> >>> I will try to explore EMR option also. >>> Can you please let me know the configuration which you have used it? >>> Can you please recommend for me also? >>> I would like to setup Hadoop cluster using cloudera manager and then >>> would like to do below things: >>> >>> setup kerberos >>> setup federation >>> setup monitoring >>> setup hadr >>> backup and recovery >>> authorization using sentry >>> backup and recovery of individual componenets >>> performamce tuning >>> upgrade of cdh >>> upgrade of CM >>> Hue User Administration >>> Spark >>> Solr >>> >>> >>> Thanks >>> Krish >>> >>> >>> On Thu, Mar 5, 2015 at 3:57 PM, Jonathan Aquilina < >>> [email protected]> wrote: >>> >>>> krish EMR wont cost you much with all the testing and data we ran >>>> through the test systems as well as the large amont of data when everythign >>>> was read we paid about 15.00 USD. I honestly do not think that the specs >>>> there would be enough as java can be pretty ram hungry. >>>> >>>> >>>> >>>> --- >>>> Regards, >>>> Jonathan Aquilina >>>> Founder Eagle Eye T >>>> >>>> On 2015-03-06 00:41, Krish Donald wrote: >>>> >>>> Hi, >>>> >>>> I am new to AWS and would like to setup Hadoop cluster using cloudera >>>> manager for 6-7 nodes. >>>> >>>> t2.micro on AWS; Is it enough for setting up Hadoop cluster ? >>>> I would like to use free service as of now. >>>> >>>> Please advise. >>>> >>>> Thanks >>>> Krish >>>> >>>> >
