On Aug 20, 2012, at 9:12 AM, Vincent Diepeveen wrote: > Look at ebay, there is cheap machines there $150 or less for 8 core > Xeons L5420, > i'm using those as well. For that $150 you get 8GB ram as well. > > Boot them over the network. > > That's 16 * $150 = $3k. > > Now there might be a difference between my and your wishes. > For me the network is important so i bought pci-e 2.0 motherboards; > > In short i bought each component separate which is more expensive > than $150. > > And those rackmounts make huge noise; i'm doing it here with 14 CM > fans that's not so noisy, > but i assume your institute has a spare room that may make people > who walk their deaf before > they're 30. > > As for you, buy 8 drives SATA from 2 TB and a $30 raid card from > ebay Those raid cards 2nd hand > are dirt cheap especially the pci-x ones and with 8 drives you > won't get a bigger bandwidth > anyway and most those online nodes have that.
online nodes => those $150k nodes have an empty pci-x slot Note pci-x is a lot slower than pci-e, yet for file server with limited amount of drives you won't get more than 700 MB/s out of it anyway and pci-x easily delivers that. > > They're 100 euro a piece here those drives. So that's 800 euro = > $1k or so. And $30 from > ebay for a total superior raid card. Put them in a ready rackmount > that allows such drives, > it's a couple of hundreds of dollars on ebay, with the same L5420's > and motherboards and 8GB > ram. So say a $500 for that rackmount with drives you can plug in. > > Put in the drives, build a raid6 partition and your fileserver , > the 17th machine, it's ready to serve you > at around a 700MB/s readspeed. > > Now i don't know the latest about genome research; the last Phd > student i helped out > there, his university used 90s software to do his research. > > That really required big crunching for months for each calculation, > at hundreds of cpu's, > yet new commercial software finished within 15 minutes each > calculation at a single core. > > That 90s software uses MPI if i recall but that'll depend upon what > sort of software your > guys want to use. > > You might want to decide next to buy the cheapest gigabit switch > you can get, in order to > boot all the nodes over the network using pci-e. correction : pxe > > It's possible those motherboards won't boot over infiniband, some > might. > > Then i'd really advice you buy a cheap 2nd hand switch infiniband, > maybe DDR, of $300 or so. > Cables $20 a piece times 17 = $140, and a bunch of 2nd hand DDR > infiniband cards and put > each one in each machine. > > So after boot over the gigabit switch, assuming the motherboards > don't boot over infiniband, > they might boot actually over infiniband in which case you don't > need the gigabit switch, > then in that case infiniband will take over and can serve you > either as 10 gigabit network > cards or for the MPI that much software in that area needs. > > So all what's left is buy a 17 infiniband cards DDR 2nd hand off > ebay. Not sure about prices, > maybe a $80 maybe $100. a piece. Let's say it's under $1600 > > Now you're done with a 16 node cluster with 17 machines from which > 1 is fileserver, > for a part of the budget you had in mind. > > Just it's noisy and loud. > > Also it's pretty low power compared to alternatives. It'll eat a > 180 watt a node or so under full load. It's 170 watt a node > here under full load (but that's with a much better psu). > > As for software to install in case you decide for infiniband, your > choices are limited as OpenFED doesn't give > much alternatives. > > Fedora Core or Scientific Linux are for free and probably your only > 2 options if you want to use free software > that are easy to get things done as you want to. > > Then install OpenFED that has the openmpi and other infiniband > stuff for free. > > Probably Debian works as well, provided you use the exact kernel > that OpenFED recommends. > Any other kernel won't work. So you have to download some older > debian then, get the exact kernel recommended > and then i guess OpenFED will install as well. > > Good Luck, > Vincent > > On Aug 20, 2012, at 6:55 AM, Duke Nguyen wrote: > >> Hi folks, >> >> First let me start that I am total novice with cluster and/or >> beowulf. I >> am familiar with unix/linux and have a few years working in a cluster >> (HPC) environment, but I never have a chance to design and admin a >> cluster. >> >> Now my new institute decides to build a (small) cluster for our next >> research focus area: genome research. The requirements are simple: >> expandable and capable of doing genome research. The budget is low, >> about $15,000, and we have decided: >> >> * cluster is a box cluster, not rack (well, mainly because our >> funding >> is low) >> * cluster OS is scientific linux with OpenMPI >> * cluster is about 16-node with a master node, expandable is a must >> >> Now next step for us is to decide hardwares and other aspects: >> >> * any recommendation for a reliable 24-port gigabit switch for the >> cluster? I heard of HP ProCurve 2824 but it is a little bit hard >> to find >> it in my country >> * should our boxes be diskless or should they have a hard disk >> inside? >> I am still not very clear the advantages if the clients has about >> 80GB >> hard disk internally except that their OS are independent and does >> not >> depend on the master node, and maybe faster data processing >> (temporay), >> but 80GB each for genome research is too small >> * hard drives/data storage: we want to have storage of about >> 10TB but >> I am not sure how to design this. Should all the hard disk be in the >> master node, or they can be on each of the node, or should it be a >> NAS? >> * any recommendation for a mainboard (gigabit network, at least >> 4 RAM >> slots) about $200-$300 good for cluster? >> >> I would love to hear any advice/suggestion from you, especially if >> you >> had built a similar cluster with similar purpose. >> >> Thank you in advance, >> >> Duke. >> >> _______________________________________________ >> Beowulf mailing list, [email protected] sponsored by Penguin >> Computing >> To change your subscription (digest mode or unsubscribe) visit >> http://www.beowulf.org/mailman/listinfo/beowulf > _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
