Re: I have a new cluster (Xserve + 4 Mac Minis) How to Hadoop?

[Ross Boucher]

I can tell you from experience that Hadoop does run fine under OS X,  
and would second the recommendation not to  bother with Linux.  The  
process of setting up the cluster is also just as simple under OS X.

Partitioning the internal drives may not be a bad idea, so you can  
keep the DFS data on its own partition, you may also want to look  
into getting external drives for the mac minis, since the stock  
drives are 5400 RPM.  Then again, if you are not IO bound, it may not  
matter much.

I used the following link to get through basic setup tasks, it has  
corresponding multi node cluster instructions as well.

http://www.michael-noll.com/wiki/Running_Hadoop_On_Ubuntu_Linux_% 
28Single-Node_Cluster%29

I'd be happy to share anything else I've learned regarding my xserve  
cluster, so feel free to ask.

Ross Boucher
[EMAIL PROTECTED]

On Oct 11, 2007, at 9:54 PM, Colin Evans wrote:

I don't think that you need to run Linux - Hadoop should run just  
fine under OS X.  You'll need to enable ssh, and you might have to  
fiddle around with the native libraries if you want to compile  
them, but everything else should work great.



Bob Futrelle wrote:
I'm a rank beginner with clusters, but am determined to move into
them, starting with Hadoop.  I have a habuntu machine under VMware on
my MacBook Pro for starters (got it on a DVD when visiting at the
Googleplex).

Now I've just received an Apple Xserve and four Mac Minis to set  
up a cluster.

Though I know little, I did see a suggestion somewhere that I set  
up a
small Mac OS X partition on each, under VMware Fusion, for general
admin via the Apple Server, for Mac firmware updates, monitoring,  
etc.

Then I'd set up a larger partition under VMware on each of the five
machines with linux images, to do the heavy lifting.

I'm looking for any and all suggestions from whomever.  You may well
be able to skip all sorts of details and just point me to
answers/examples out there, people, search terms, etc. Then I can
start doing my homework. I'm happy to experiment, setting up stuff,
stripping it clean, setting up another approach, etc., as part of
learning what works.

Looking forward to the standard mix of confusion, frustration,
success, failure, elation, and lots of messy work.

Oh, what do I do?  Want to map out image analysis and text analysis
work to the nodes. I analyze, extract from, Biomedical research
papers.

 - Bob

Robert P. Futrelle
  Associate Professor
Biological Knowledge Laboratory
College of Computer and Information Science
Northeastern University MS WVH202
360 Huntington Ave.
Boston, MA 02115

Office: 617-373-4239
Fax:     617-373-5121
http://www.ccs.neu.edu/home/futrelle
http://www.bionlp.org
http://www.diagrams.org