Edward Ned Harvey wrote: >> This is very tricky and presumes that your local disk is faster than >> your back end storage, which is not necessarily the case. A local disk >> cache can be your friend or your enemy depending on your job load and >> your architecture. If you have a big honking storage farm to serve your >> HPC cluster with lots of memory, you can serve things at nearly wire >> speed. >> > > When you say "local disk" and "backend storage," I assume you're talking > about the local sas/sata disk, and the SAN storage, right? Both of these > are a couple orders of magnitude slower than the physical ram, so, if your > SAN is faster than your sas/sata disk, why not put your swap on the SAN? > > Building a HPC farm of any reasonable size makes direct attach SAN to every node a very expensive option (prohibtively so in many cases). Currently we use NFS, though we're considering other alternatives. But, even so, I wouldn't put swap there. For our work load, anything taking ram out to disk is a huge loss in performance.
An example.. memory dimms that are just ever-so-slightly out of spec will cause some jobs to run at half performance because of the other nodes in the cluster having to wait for this node continuously to do its calculations and answer. In fact, the node looks identical to others for stream and qcdstream and qcdstreamV and qcdstreamV --sse --mem (though some fail this test), but if you run a certain MPI ping pong test on all 8 cores on the machine, the maximum response times of those tests could be 20-80% than a normal node. It's crazy stuff. hitting disk (network or flash or memory SSD) at all in any of these sorts of compute and memory bound applications for any reason is right out. _______________________________________________ Tech mailing list [email protected] http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
