I have received several private comments expressing interest in this topic, so I'd like to generate more discussion and attempt to focus on a solution that meets most or all of the needs.
Summary of problem: ----- A zone can be configured so that its processes do not run on the CPUs in the default pset, but in a different pset. The zone can have exclusive access to those CPUs, or one or more other zones can be configured to share that pset. Zone configuration is not aware of interrupt handlers. When Solaris boots, it must assign each device's interrupt handler to a CPU. It does so without knowledge of psets or zones. The lack of awareness of integration between zones and interrupt handlers leads to situations where heavy CPU utilization in one zone can lead to performance or performance-related problems in other zones. For example, the interrupt handler for a network interface may be assigned to a CPU that is later also assigned to a zone which doesn't use that NIC. This can cause dropped network packets that are to/from zones which are not using that CPU. These problems violate the main goal of zones: workload isolation. ----- The "first order of magnitude" solution is to simply disable interrupts on zones which are assigned to non-default psets. This is often effective, but in practice requires custom scripts. Management of those scripts across a data center can be burdensome or even overwhelming. In addition, that solution may not meet the goal of workload isolation. A system could be configured with multiple zones that have separate (exclusive) NICs and CPUs. Disabling interrupts on the zones' psets will move all interrupt handling into the default pset. Solaris might assign all of the NIC interrupt handlers to one of those CPUs. Network activity generated by one zone could interfere with the ability to quickly handle network traffic associated with a different zone. Therefore, it might be desirable to configure an exclusive-IP zone so that the interrupt handler for its NIC(s) are assigned to CPUs in that zone's pset. Here are some possible solutions: 1. Add to zonecfg a property which requires that a zone's CPUs not handle interrupts. The syntax could be simple: zonecfg -z myzone set interrupts=disabled exit If the zone is configured to run in the default pset, 'verify' shoudl fail, and the zone should refuse to boot. It's not clear what should happen if the zone is booting into a shared pset that allready has zones *and* interrupt handlers. 2. Place an interrupt property in the "dedicated-cpu" feature. zonecfg -z myzone add dedicated-cpu set ncpus=4 set interrupts=disabled end exit That syntax doesn't handle zones which use "set pool=". 3. Associate an interrupt property with the "exclusive-ip" feature to allow the user to specify that all non-network interrupt handlers should be moved to the default pset, and interrupts for this zone's NIC should be handled by this zone's pset. zonecfg -z myzone set ip-type=exclusive add net set physlcal=e1000g0 set interrupts=enabled end exit Another NIC in that zone would have a separate 'interrupts' property. Its interrupts could also be handled by this zone's pset or by the default pset. Some questions: 1. Do you use "set pool=" anymore, now that the dedicated-cpu feature exists? 2. Is it sufficient to simply disable interrupts on a zone's pset? 3. Are there any other devices which (A) can be assigned exclusively to a zone (via 'set match') and generate enough interrupts to cause problems? 4. Implementing (1) or (2) should be relatively simple. Choice (3) might be significantly more effort, and might delay any of this functionality. Which is better: more granular configuration of interrupt handling or faster relief? (Either way, I wouldn't expect Sun to do this during CY2009. However, if you have sufficient interest and ability... :-) ). --JeffV On Tue, Mar 3, 2009 at 11:26 PM, Jeff Victor <jeff.j.vic...@gmail.com> wrote: > On Tue, Mar 3, 2009 at 8:39 PM, Gael <gael.marti...@gmail.com> wrote: >> >> Many thanks to Bob Netherton and Jeff for their quick help on that painful >> issue. >> The solution was to use psrset -f on the heavily used pset. >> It is fully supported and a recommended situation when CPU starvation causes >> interrupts not to be serviced in time and they get lost. Credit goes to >> Rickey Weisner for this tip. >> >> I have monitored that zone today for multiple hours without seeing any >> packet loss while it was cranking up its cpu usage... >> Jeff, following a previous mail today, as a fervent customer ;), I would >> love to see that feature directly accessible thru the zone configuration to >> avoid having to create a script and a dirty workaround to enable that >> feature on boot. Is there a RFE # out there that I can be added to thru Sun >> Support ? Got a case opened on that issue. > > Yes, the CR is 6199531 - "Device interrupts not bound to cpus > configured within a nonglobal zone" > > Please ask your contact in Sun Service to add an SR for you. > >> Will continue to monitor the situation for a few days, and if I see anything >> wrong, I will update that thread >> >> On Tue, Mar 3, 2009 at 2:19 PM, Jeff Victor <jeff.j.vic...@gmail.com> wrote: >>> >>> Hello Gael, >>> >>> On Mon, Mar 2, 2009 at 10:08 PM, Gael <gael.marti...@gmail.com> wrote: >>> > >>> > Got a zone running SAS with cpu capping enabled using a processor set as >>> > we >>> > see a few processes using quite a bit of cpu there too often. >>> >>> Is that zone assigned to a resource pool, or is it using the dedicated-cpus >>> feature? >>> >>> > When the process is running (chewing 100% of its pset), the frame nic >>> > (server is a E2900 with a ce interface) is dropping 20-30 % of its packets >>> > causing a headache. >>> >>> My first guess is that the NICs interrupts are going to a CPU that the >>> zone is using, and the CPU doesn't have enough power to run the zone's >>> workload *and* be an effective NIC interrupt handler. >>> >>> Please run the "intrstat" command as root in the global zone, to >>> determine which CPU is handling interrupts for that NIC. Also, check >>> which CPU(s) that zone can use. >>> >>> Please let us know what you learn from those. >>> >>> > Doesn't appear to be a network load issue. Not a lot happening there >>> > visibly. >>> > >>> > With Solaris 10 u4 or u6, what elegant way would you recommend to avoid >>> > that >>> > disruption caused by a single zone ? > > _______________________________________________ zones-discuss mailing list zones-discuss@opensolaris.org
