Chris Linton-Ford writes: > "Enable Node Interleaving": initially disabled, when turned on the > operating system booted, but I got the following error message: > "MPO disabled because memory is interleaved"
You definitely don't want "Node Interleaving." As a BIOS option, it wins a special award for being the most misleading misfeature since plug-n-play. Opteron systems have memory attached to each CPU, but a global address space. When you access memory that's not local to your CPU, you incur a penalty going over the internal "hypertransport" between the CPUs. On OSes (such as Solaris) that are aware of NUMA hardware, that's not a problem. The OS automatically optimizes the allocation of memory based on where the task is running (or vice-versa) so that you get good locality. On OSes that are ignorant of this sort of hardware, naive testing can produce strange results. You run your test once and it runs fast. Run it again, and it's slow. Run again, it's fast again. It all depends on where Wind^Wthat OS puts your application, and it's unpredictable. This is where "Node Interleaving" comes in. Turning that misfeature on causes memory to be mapped page-by-page in a round-robin fashion among the CPUs instead of being contiguous. In other words, every 4K boundary talks to a different CPU in the system. This averages out the costs, making everything run equally poorly, and making your naive OS and benchmarks seem to be stable. The OS can't keep track of that sort of addressing mess, so it causes MPO to be turned off. You don't want this on Solaris. -- James Carlson, Solaris Networking <[EMAIL PROTECTED]> Sun Microsystems / 1 Network Drive 71.232W Vox +1 781 442 2084 MS UBUR02-212 / Burlington MA 01803-2757 42.496N Fax +1 781 442 1677 _______________________________________________ opensolaris-discuss mailing list [email protected]
