LDoms are isolated at the Hypervisor, CPU, Memory, IOMMU, and PCIe (if doing split-bus I/O Domain) level. So all the access points are isolated at the hardware level which provides a better level of isolation than you'd get with VMware or even Solaris Containers. It's not as good as Dynamic System Domains where CPU/Memory and I/O boats are electrically isolated and not shared. Where things do get complicated is with the Networking and Storage I/O which are typically shared unless you are doing split-bus or NIU configurations. In the cases of shared I/O, I always recommend the following:
1. Multi-path all your I/O -> MPXIO, LACP/IPMP 2. Use VLAN Tagging and IPQoS 3. Use dedicated SAN LUNs directly for each LDom or for its own Zpool. 4. Use half or whole cores. You want to divide along the execution units in the cores (i.e. T2/T2+ there are two that each handle 4 threads, only thing shared is the Crypto and FPU). If you do that, you can minimize a lot of the contention issues. Hopefully at some point Solaris will have the capability of limiting I/O bandwidth at the SRM, Solaris Container, and LDom levels. This can probably be done with OpenSolaris for Network I/O using VNICs, which I have to play around with in LDoms to see how it works. *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* Octave J. Orgeron Solaris Virtualization Architect and Consultant Web: http://unixconsole.blogspot.com E-Mail: [email protected] *-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-* ----- Original Message ---- From: Richard L. Hamilton <[email protected]> To: [email protected] Sent: Wed, August 18, 2010 5:04:14 AM Subject: Re: [ldoms-discuss] Can LDom guests affect each other? "Bus" isolation should be pretty good, assuming PCIe, which is actually point-to-point. But anytime different LDOMs used virtual devices (thinking particularly virtual disks) ultimately accessed via the same card or interface, I expect they'd be competing with each other for bandwidth; probably more so if they were all zvols out of a big fat zpool for example (where they'd also be competing for throughput of the underlying devices). Older models of T-series systems may have PCI-X buses, which would likely have competition across the entire bus, effectively less isolation (or so I'm guessing). AFAIK, it _is_ possible to allocate bandwidth associated with a virtual NIC. AFAIK, memory allocations are fixed. AFAIK, if one split a CPU core between LDOMs, there might be some potential for them to compete with one another (functional units of a core in use by one thread aren't available for other threads on that core at the same time); if LDOMs were assigned whole cores, I wouldn't worry about that so much. But I imagine that depending on the cache and memory architecture, some slight effect might still be possible. This is quite different from a larger system, with multiple system boards and a centerplane consisting of programmable crossbar switches, which can be divided into multiple hardware domains, fully isolated from each other (ignoring power and centerplane failures, or a system controller that spewed noise at the programmable crossbar switches). (although even then, heavy activity on LUNs of a multiport array or SAN might impact one another) I haven't seen a whitepaper on it, but haven't really looked, either. I imagine that whitepapers on LDOMs in general might get into some of this, but not to the point of setting bounds or _necessarily_ even to the point of all the configuration choices that would give the best approximation of isolation. Probably it boils down to making good configuration choices and not oversubscribing, to minimize effects. Of course, since that's all off the top of my head, and my experience with T-series systems is limited, I could be totally wrong... -- This message posted from opensolaris.org _______________________________________________ ldoms-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/ldoms-discuss _______________________________________________ ldoms-discuss mailing list [email protected] http://mail.opensolaris.org/mailman/listinfo/ldoms-discuss
