Re: [CentOS] Motherboards for HPC applications
On Thu, Mar 11, 2010 at 08:35:23AM +0800, Christopher Chan wrote: On Thursday, March 11, 2010 01:41 AM, Eduardo Grosclaude wrote: On Wed, Mar 10, 2010 at 2:05 AM, Christopher Chan christopher.c...@bradbury.edu.hk wrote: On the Intel side, a dual socket solution will even outperform a quad socket solution so if one is looking for Intel cpu solutions, dual socket is the only sensible choice. But that Wow, that's a pretty impressive statement, can you elaborate on that? Intel does not have a NUMA architecture solution for quad socket boards yet and so bus contention kills scaling on quad socket Intel solutions. Right now, AMD owns the quad socket market when it comes to performance. You mean, for every possible workload? Is it something you learned from direct experience, or have you read about it? If so, where? If from experience, what was the general setup, applications, etc? Anandtech did some testing last quarter where they compared what appears to be the best quad socket Intel solution against the best dual socket Intel solution and the dual socket solution ran circles around the quad in some tests and pretty much matches it in other tests. http://it.anandtech.com/IT/showdoc.aspx?i=3653p=1 According to the article, Intel will be coming out with something that will scale at the quad socket level some time this year so things are probably going to change on this front. Nehalem-EX ? I think all the big server vendors have been announcing new products based on those last week, this week, or next week. -- Pasi ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Wed, Mar 10, 2010 at 2:05 AM, Christopher Chan christopher.c...@bradbury.edu.hk wrote: On the Intel side, a dual socket solution will even outperform a quad socket solution so if one is looking for Intel cpu solutions, dual socket is the only sensible choice. But that Wow, that's a pretty impressive statement, can you elaborate on that? You mean, for every possible workload? Is it something you learned from direct experience, or have you read about it? If so, where? If from experience, what was the general setup, applications, etc? Thank you very much again for all your responses -- Eduardo Grosclaude Universidad Nacional del Comahue Neuquen, Argentina ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Thursday, March 11, 2010 01:41 AM, Eduardo Grosclaude wrote: On Wed, Mar 10, 2010 at 2:05 AM, Christopher Chan christopher.c...@bradbury.edu.hk wrote: On the Intel side, a dual socket solution will even outperform a quad socket solution so if one is looking for Intel cpu solutions, dual socket is the only sensible choice. But that Wow, that's a pretty impressive statement, can you elaborate on that? Intel does not have a NUMA architecture solution for quad socket boards yet and so bus contention kills scaling on quad socket Intel solutions. Right now, AMD owns the quad socket market when it comes to performance. You mean, for every possible workload? Is it something you learned from direct experience, or have you read about it? If so, where? If from experience, what was the general setup, applications, etc? Anandtech did some testing last quarter where they compared what appears to be the best quad socket Intel solution against the best dual socket Intel solution and the dual socket solution ran circles around the quad in some tests and pretty much matches it in other tests. http://it.anandtech.com/IT/showdoc.aspx?i=3653p=1 According to the article, Intel will be coming out with something that will scale at the quad socket level some time this year so things are probably going to change on this front. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
Christopher Chan wrote: Anandtech did some testing last quarter where they compared what appears to be the best quad socket Intel solution against the best dual socket Intel solution and the dual socket solution ran circles around the quad in some tests and pretty much matches it in other tests. thats largely because the current generation quad socket processors are still using last generation CPU cores, the current dual socket are the new I7/nehalem cores, which are inherently a lot faster. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
Eduardo Grosclaude wrote: On Tue, Mar 9, 2010 at 12:49 AM, Christopher Chan christopher.c...@bradbury.edu.hk wrote: On Tuesday, March 09, 2010 12:34 AM, Eduardo Grosclaude wrote: Hello, Can somebody recommend CentOS-OK, dual socket motherboards for compute elements? A quick look up at Intel pages suggests they are thinking of them as server boards, but then they recommend them as for SMB, I'm somewhat puzzled about it. It would be nice to know what MBs you are using, pros and cons. Thank you in advance Could you give us a bit more information on the HPC part? Is this clustering or computing? I'll be buying a single machine first, building a cluster some time later. As this second move may be delayed for an unpredictable amount of time, what I am really interested in is understanding the thought process a seasoned technician (sysadmin? clusadmin?) may follow when selecting hardware. Do you have high i/o needs? Well, perhaps this is my real problem... Don't have enough info about applications. There are several of them but I think I/O is not at premium, rather CPU computing is. If you do not have enough information on the applications, I am afraid it is going to be rather hard to make a final decision. Maybe you want to overspec on the first box, find out what those apps really do and then spec accordingly. Things to consider can include network bandwidth, disk bandwidth. 'bus' bandwidth, memory bandwidth and as John Pierce pointed out, what type of processing. Are the apps single threaded or multi threaded? Single threaded apps might call for the cpus with the highest possible frequencies while multi threaded ones not so much so but how many you can pack into whatever space you have. If cpu processing power is the sole criteria, then why limit to dual-socket boards and not go for quad-socket boards? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Tue, Mar 9, 2010 at 1:08 AM, John R Pierce pie...@hogranch.com wrote: for a high performance compute cluster, you'll probably want to use management software like Oscar, which integrates system management with MPI based distributed computing such that you can manage a cluster of 100s of servers like its a single big system I've been using Kusu with much success. Sadly, you're pretty much on your own there as the project seems unsupported or sucked dry out by Platform.com. I hope it to fully reincarnate in Red Hat's HPC proposal and that it eventually makes its way into CentOS. -- Eduardo Grosclaude Universidad Nacional del Comahue Neuquen, Argentina ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Tue, 9 Mar 2010 at 9:49pm, Chan Chung Hang Christopher wrote If cpu processing power is the sole criteria, then why limit to dual-socket boards and not go for quad-socket boards? In general, the price goes up non-linearly as you go above 2 sockets, making 2 sockets the sweet spot when it comes to price/performance. -- Joshua Baker-LePain QB3 Shared Cluster Sysadmin UCSF ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
Eduardo Grosclaude wrote: On Tue, Mar 9, 2010 at 1:08 AM, John R Pierce pie...@hogranch.com wrote: for a high performance compute cluster, you'll probably want to use management software like Oscar, which integrates system management with MPI based distributed computing such that you can manage a cluster of 100s of servers like its a single big system I've been using Kusu with much success. Sadly, you're pretty much on your own there as the project seems unsupported or sucked dry out by Platform.com. I hope it to fully reincarnate in Red Hat's HPC proposal and that it eventually makes its way into CentOS. note that Oscar 6.x can be used with Centos 5.x (or debian or suse), and it seems like Centos is their preferred platform. I setup an Oscar test cluster some time ago using some old PCs, it was surprisingly easy. you install the oscar packages on your 'master' server, this one has two connections, one to your LAN and one to your HPC cluster (which is on its own switch).Then you PXE boot your HPC nodes and they get installed with a centos+scientific kit, inculding any custom application stuff you specified. then you just run your MPI based application(s), and its automatically distributed across the nodes of the cluster, Oscar also provides monitoring (Ganglia) and other stuffs. MPI is a standard Message Passing Interface used in scientific computing, essentially you write your software such that it accepts messages telling it what to do and sends messages with the results. This works best for applications that don't need a lot of global interactions, where each unit of computation can be self contained for some reasonable period of time. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Mon, Mar 08, 2010 at 08:09:57PM -0800, nate wrote: Gordon McLellan wrote: If your application can't support GPU based processing, I think Peter's suggestion is most fitting. Load up a rack of dual socket 5520 servers from Dell or HP and then save some money by building your own shared-storage to feed the cluster. The big vendors crank out very inexpensive dual socket xeon servers, the only area they really seem to be price gouging in right now is storage. For me I have been working on spec'ing out a HPC cluster to run Hadoop on large amounts of data and fell in love with the SGI Cloud Rack C2. I managed to come up with a configuration that had roughly 600 CPU cores, 1.2TB of memory and 300 1TB SATA disks in a single rack and consumes ~16,000 watts of power with 99% efficient rack level power supplies and N+1 power redundancy, rack level cooling as well. Very cost effective as well at least for larger scale deployments, assuming you have a data center that can support such density. http://www.sgi.com/products/servers/cloudrack/cloudrackc2.html My current data center does not support such density so I came up with a configuration of 320 CPU cores, 640GB memory, and 160x1TB disks that fit in a single 24U rack, and consumes roughly 8,000 watts(208V 30A 3-phase) and weighs in at just under 1,200 pounds (everything included). Systems come fully racked, cabled ready to plug in. Systems are built with commodity components wherever possible(MB/ram/CPU/HD), only custom stuff is the enclosure, cooling, and power distribution, which is how they achieve the extreme densities and power efficiency. Wow, pretty cool system. Can you tell about the pricing? -- Pasi ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Wednesday, March 10, 2010 12:35 AM, Joshua Baker-LePain wrote: On Tue, 9 Mar 2010 at 9:49pm, Chan Chung Hang Christopher wrote If cpu processing power is the sole criteria, then why limit to dual-socket boards and not go for quad-socket boards? In general, the price goes up non-linearly as you go above 2 sockets, making 2 sockets the sweet spot when it comes to price/performance. Hmm, I see at most a 50% increase in motherboard pricing from a dual to a quad socket motherboard and that is with a difference in feature set too with the quad coming with an extra onboard LSI 8 port SAS controller. That is hardly going up non-linearly. (taking an extremely narrow angle ;-p) ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
Pasi Kärkkäinen wrote: Wow, pretty cool system. Can you tell about the pricing? I don't think I can, but it is competitive with Dell and HP as an example while the innovation put into the cloud rack is far beyond anything Dell or HP offer to mere mortals. Closest HP offers is the SL series of systems which are pretty decent, though offer roughly half the density as SGI for our particular application. http://h10010.www1.hp.com/wwpc/us/en/sm/WF02a/15351-15351-3896136.html?jumpid=re_R295_prodexp/busproducts/computing-server/proliant-sl-scalable-syspsn=servers Dell is coming out with something new soon http://www.theregister.co.uk/2010/02/03/dell_cloudedge/ I've seen them, and honestly aren't all that creative, very similar to Supermicro Twin. They are decent for CPU and memory intensive stuff, but not as good for (local) I/O intensive. They seem pretty proud about these systems though considering Supermicro has had similar stuff on the market for quite some time now there isn't much to get excited about IMO. SGI(formerly Rackable) has been pretty aggressive in patenting their designs, which is probably what lead to vendors like Supermicro building their Twin systems. http://www.sgi.com/company_info/newsroom/press_releases/rs/2007/05082007.html Dell has a custom design division which they can probably do some pretty crazy things but I'm told they have a ~1,500 server minimum to get anything from that group. nate ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
I work for Dell but I can't talk too much about the units you are referring to. The launch date is in a couple of weeks and then I can spill my guts :) I can't talk about price since, to be honest, I don't really know pricing (I'm a tech person). But let me give some general hints. The unit you are speaking about has actually been selling for a couple of years to larger customers. There are more units of this in production use right now than all of Supermicro and HP combined :) One success I can mention since it's public is Wolfram's Alpha system is powered by these units. What is new with the launch is that before you had to but them in quantities of 500-1,000. Now you can buy one of them. So if Dell was doing these 2 years ago, imagine what is coming next :) Jeff From: nate cen...@linuxpowered.net To: centos@centos.org Sent: Tue, March 9, 2010 7:37:04 PM Subject: Re: [CentOS] Motherboards for HPC applications Pasi Kärkkäinen wrote: Wow, pretty cool system. Can you tell about the pricing? I don't think I can, but it is competitive with Dell and HP as an example while the innovation put into the cloud rack is far beyond anything Dell or HP offer to mere mortals. Closest HP offers is the SL series of systems which are pretty decent, though offer roughly half the density as SGI for our particular application. http://h10010.www1.hp.com/wwpc/us/en/sm/WF02a/15351-15351-3896136.html?jumpid=re_R295_prodexp/busproducts/computing-server/proliant-sl-scalable-syspsn=servers Dell is coming out with something new soon http://www.theregister.co.uk/2010/02/03/dell_cloudedge/ I've seen them, and honestly aren't all that creative, very similar to Supermicro Twin. They are decent for CPU and memory intensive stuff, but not as good for (local) I/O intensive. They seem pretty proud about these systems though considering Supermicro has had similar stuff on the market for quite some time now there isn't much to get excited about IMO. SGI(formerly Rackable) has been pretty aggressive in patenting their designs, which is probably what lead to vendors like Supermicro building their Twin systems. http://www.sgi.com/company_info/newsroom/press_releases/rs/2007/05082007.html Dell has a custom design division which they can probably do some pretty crazy things but I'm told they have a ~1,500 server minimum to get anything from that group. nate ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
Christopher Chan wrote: Hmm, I see at most a 50% increase in motherboard pricing from a dual to a quad socket motherboard and that is with a difference in feature set too with the quad coming with an extra onboard LSI 8 port SAS controller. That is hardly going up non-linearly. (taking an extremely narrow angle ;-p did you price the quad socket CPUs ? they cost significantly more, too. the newest processor chips are usually the dual socket versions. Also, 4-socket servers may have performance issues caused by 4-way cache coherency chatter, non-uniform memory access, and so forth. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Wednesday, March 10, 2010 11:41 AM, John R Pierce wrote: Christopher Chan wrote: Hmm, I see at most a 50% increase in motherboard pricing from a dual to a quad socket motherboard and that is with a difference in feature set too with the quad coming with an extra onboard LSI 8 port SAS controller. That is hardly going up non-linearly. (taking an extremely narrow angle ;-p did you price the quad socket CPUs ? they cost significantly more, Guess why I said 'extremely narrow angle' :-D too. the newest processor chips are usually the dual socket versions. Also, 4-socket servers may have performance issues caused by 4-way cache coherency chatter, non-uniform memory access, and so forth. /me looking at AMD solutions where those issues do not exist. Performance is almost linear. On the Intel side, a dual socket solution will even outperform a quad socket solution so if one is looking for Intel cpu solutions, dual socket is the only sensible choice. But that does not mean quad socket solutions should be discounted when an AMD solution will outperform the best Intel dual-socket setup available. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Mon, Mar 8, 2010 at 11:34 AM, Eduardo Grosclaude eduardo.groscla...@gmail.com wrote: Hello, Can somebody recommend CentOS-OK, dual socket motherboards for compute elements? A quick look up at Intel pages suggests they are thinking of them as server boards, but then they recommend them as for SMB, I'm somewhat puzzled about it. It would be nice to know what MBs you are using, pros and cons. Thank you in advance Hello! We need more details... what's your budget, what processor are you looking at? SMB just means Small / Medium Business ... as opposed to a huge enterprise server that might have four or eight sockets... Name a vendor, I've probably had some sort of trouble with them... of the Big Names, Intel is probably the least troublesome. I just returned a bad Tyan board, and late last year returned two Supermicro servers that were shipped with out of date hardware (not supporting 5400 series CPU). I have an Asus board that runs Linux and Opensolaris just fine, but will not allow any version of Windows to install. I hope this is some help to you. Gordon ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Eduardo Grosclaude wrote: Hello, Can somebody recommend CentOS-OK, dual socket motherboards for compute elements? A quick look up at Intel pages suggests they are thinking of them as server boards, but then they recommend them as for SMB, I'm somewhat puzzled about it. It would be nice to know what MBs you are using, pros and cons. Thank you in advance I'm running a really old mainboard, an MSI 694D Pro (MS-6321). It is a dual socket, running two Intel PIII Cpppermine CPU's with an upper limit of 933MHz. It is classified as a server board. The board was new in 2000. The board was installed new, in 2004 (it laid around new in box for a long time), and ran RHEL3 from 2004 until I upgraded to CentOS 5.4 a couple months ago. With 2 gig RAM, it runs well for a Desktop machine. I figure I can squeeze another five years out of it, if I'm lucky. :) - -- - -wittig http://www.robertwittig.com/ http://robertwittig.net/ http://robertwittig.org/ . -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with CentOS - http://enigmail.mozdev.org/ iQEVAwUBS5Usxf9qkhAVPSgqAQJxXwf/UCwodEQHSW8yR9CrNtZvvGQK4NaeKMxk sbd907l88UGDrjKbx92lpCXzsUbPr7cJrPGwVhqJKFS1LMsYYKJWMIOvBx27ZmIF kRPNJXBXOVTEQ5e/yVWsyXGqytxlD05XcrwzcGBhwpeaO7wQl4Eykwb2NMM3UYNw C0K6crqo+TOg3wb34uJucpb6zmz/0S/efgVhTmjqOScUS2CK9skAaAxq78L3lViQ bJJ7Npq3rbrg0vLqeQ6T5BoXfKOYT56j6Q35tOhGYzTZoFEMdoXLV62hgZ0HQMxk LK8jQCOspU5olboxbRmsiJ1IECF6CDL993nWRahqOktQe0dXfdOFYQ== =2RrV -END PGP SIGNATURE- ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
Hi, 5400 series CPU). I have an Asus board that runs Linux and Opensolaris just fine, but will not allow any version of Windows to install. I want one of those :) Regards, Michel ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Mon, Mar 8, 2010 at 12:49 PM, Eduardo Grosclaude eduardo.groscla...@gmail.com wrote: I'm targeting E5520. I'll buy in Argentina, with a high stack of all sort of costs threw upon the product, so budget may not mean much to foreigners. Eduardo, Are you going to be writing your own HPC software, or using some pre-written cluster aware OS? Check around to see if Supermicro has a regional sales center for your local. They have a new line of servers setup for GPU based HPC applications... I have no need for such horsepower, but reading the specs makes me envious none the less: http://www.supermicro.com/products/system/4U/7046/SYS-7046GT-TRF.cfm?GPU=TC4 Dual 5500 series cpu (four core each) plus four nVidia Tesla GPU based supercomputing engines (included). If your application can't support GPU based processing, I think Peter's suggestion is most fitting. Load up a rack of dual socket 5520 servers from Dell or HP and then save some money by building your own shared-storage to feed the cluster. The big vendors crank out very inexpensive dual socket xeon servers, the only area they really seem to be price gouging in right now is storage. Gordon ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
On Tuesday, March 09, 2010 12:34 AM, Eduardo Grosclaude wrote: Hello, Can somebody recommend CentOS-OK, dual socket motherboards for compute elements? A quick look up at Intel pages suggests they are thinking of them as server boards, but then they recommend them as for SMB, I'm somewhat puzzled about it. It would be nice to know what MBs you are using, pros and cons. Thank you in advance Could you give us a bit more information on the HPC part? Is this clustering or computing? Do you have high i/o needs? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
Eduardo Grosclaude wrote: Hello, Can somebody recommend CentOS-OK, dual socket motherboards for compute elements? for real numeric stuff (as opposed to things like video processing that utilizes sse3), the AMD processors often outperform Intel. current AMD dual socket server processors have SIX cores each, but I dunno who's on top of the performance curve this year. The Intel I7 family, including the E5500 server chips, are screamers. what really counts in large HPC clusters is gigaflop/$$$ a number of vendors make 1U chassis designed to hold TWO compact dual processor server boards so you can get 2 nodes per U, but if you go this route, you really have to watch your cooling and power (50 or 60 of these in a rack means you'll have a REALLY high power/thermal load per rack). An example such board is http://www.intel.com/products/server/motherboards/S5500HV/S5500HV-overview.htm with the Intel E55xx series, you want to populate your memory 6 dimms at a time (assuming two CPUs), using 2gb or 4gb dimms, for max performance (each processor has 3 memory channels) for a high performance compute cluster, you'll probably want to use management software like Oscar, which integrates system management with MPI based distributed computing such that you can manage a cluster of 100s of servers like its a single big system ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Motherboards for HPC applications
Gordon McLellan wrote: If your application can't support GPU based processing, I think Peter's suggestion is most fitting. Load up a rack of dual socket 5520 servers from Dell or HP and then save some money by building your own shared-storage to feed the cluster. The big vendors crank out very inexpensive dual socket xeon servers, the only area they really seem to be price gouging in right now is storage. For me I have been working on spec'ing out a HPC cluster to run Hadoop on large amounts of data and fell in love with the SGI Cloud Rack C2. I managed to come up with a configuration that had roughly 600 CPU cores, 1.2TB of memory and 300 1TB SATA disks in a single rack and consumes ~16,000 watts of power with 99% efficient rack level power supplies and N+1 power redundancy, rack level cooling as well. Very cost effective as well at least for larger scale deployments, assuming you have a data center that can support such density. http://www.sgi.com/products/servers/cloudrack/cloudrackc2.html My current data center does not support such density so I came up with a configuration of 320 CPU cores, 640GB memory, and 160x1TB disks that fit in a single 24U rack, and consumes roughly 8,000 watts(208V 30A 3-phase) and weighs in at just under 1,200 pounds (everything included). Systems come fully racked, cabled ready to plug in. Systems are built with commodity components wherever possible(MB/ram/CPU/HD), only custom stuff is the enclosure, cooling, and power distribution, which is how they achieve the extreme densities and power efficiency. nate ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
