I was referring to multiple VM's on a single machine (that you have in house) for my previous comment and not EC2. FWIW, I would rather see a single heavy data node than to partition off a single box into multiple machines unless you are trying to do more on that server than just Hadoop. Obviously every person / company has their own constraints but if this box is solely for Hadoop then don't partition it otherwise you will incur a decent loss in possible map/reduce slots.
Matt -----Original Message----- From: Liam Friel [mailto:[email protected]] Sent: Monday, August 15, 2011 3:04 PM To: [email protected] Subject: Re: hadoop cluster on VM's On Mon, Aug 15, 2011 at 7:31 PM, GOEKE, MATTHEW (AG/1000) < [email protected]> wrote: > Is this just for testing purposes or are you planning on going into > production with this? If it is the latter than I would STRONGLY advise to > not give that a second thought due to how the framework handles I/O. However > if you are just trying to test out distributed daemon setup and get some ops > documentation then have at it :) > > Matt > > -----Original Message----- > From: Travis Camechis [mailto:[email protected]] > Sent: Monday, August 15, 2011 12:45 PM > To: [email protected] > Subject: hadoop cluster on VM's > > Is it recommended to install a hadoop cluster on a set of VM's that are all > connected to a SAN? > > Could you expand on that? Do you mean multiple VMs on a single server are a no-no? Or do you mean running Hadoop on something like Amazon EC2 for production is also a no-no? With some pointers to background if the latter please ... Just for my education. I have run some (test I guess you could call them) Hadoop clusters on EC2 and it was working OK. However I didn't have the equivalent pile of physical hardware lying around to do a comparison ... which I guess is why EC2 is so attractive. Ta Liam This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited. All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware". Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying this e-mail or any attachment. The information contained in this email may be subject to the export control laws and regulations of the United States, potentially including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all applicable U.S. export laws and regulations.
