I have received several private comments expressing interest in this
topic, so I'd like to generate more discussion and attempt to focus on
a solution that meets most or all of the needs.

Summary of problem:
A zone can be configured so that its processes do not run on the CPUs
in the default pset, but in a different pset. The zone can have
exclusive access to those CPUs, or one or more other zones can be
configured to share that pset. Zone configuration is not aware of
interrupt handlers.

When Solaris boots, it must assign each device's interrupt handler to
a CPU. It does so without knowledge of psets or zones.

The lack of awareness of integration between zones and interrupt
handlers leads to situations where heavy CPU utilization in one zone
can lead to performance or performance-related problems in other
zones. For example, the interrupt handler for a network interface may
be assigned to a CPU that is later also assigned to a zone which
doesn't use that NIC. This can cause dropped network packets that are
to/from zones which are not using that CPU.

These problems violate the main goal of zones: workload isolation.

The "first order of magnitude" solution is to simply disable
interrupts on zones which are assigned to non-default psets. This is
often effective, but in practice requires custom scripts. Management
of those scripts across a data center can be burdensome or even

In addition, that solution may not meet the goal of workload
isolation. A system could be configured with multiple zones that have
separate (exclusive) NICs  and CPUs. Disabling interrupts on the
zones' psets will move all interrupt handling into the default pset.
Solaris might assign all of the NIC interrupt handlers to one of those
CPUs. Network activity generated by one zone could interfere with the
ability to quickly handle network traffic associated with a different

Therefore, it might be desirable to configure an exclusive-IP zone so
that the interrupt handler for its NIC(s) are assigned to CPUs in that
zone's pset.

Here are some possible solutions:
1. Add to zonecfg a property which requires that a zone's CPUs not
handle interrupts. The syntax could be simple:

zonecfg -z myzone
set interrupts=disabled

If the zone is configured to run in the default pset, 'verify' shoudl
fail, and the zone should refuse to boot. It's not clear what should
happen if the zone is booting into a shared pset that allready has
zones *and* interrupt handlers.

2. Place an interrupt property in the "dedicated-cpu" feature.

zonecfg -z myzone
add dedicated-cpu
 set ncpus=4
 set interrupts=disabled

That syntax doesn't handle zones which use "set pool=".

3. Associate an interrupt property with the "exclusive-ip" feature to
allow the user to specify that all non-network interrupt handlers
should be moved to the default pset, and interrupts for this zone's
NIC should be handled by this zone's pset.

zonecfg -z myzone
set ip-type=exclusive
add net
  set physlcal=e1000g0
  set interrupts=enabled

Another NIC in that zone would have a separate 'interrupts' property.
Its interrupts could also be handled by this zone's pset or by the
default pset.

Some questions:
1. Do you use "set pool=" anymore, now that the dedicated-cpu feature exists?

2. Is it sufficient to simply disable interrupts on a zone's pset?

3. Are there any other devices which (A) can be assigned exclusively
to a zone (via 'set match') and generate enough interrupts to cause

4. Implementing (1) or (2) should be relatively simple. Choice (3)
might be significantly more effort, and might delay any of this
functionality. Which is better: more granular configuration of
interrupt handling or faster relief? (Either way, I wouldn't expect
Sun to do this during CY2009. However, if you have sufficient interest
and ability... :-) ).


On Tue, Mar 3, 2009 at 11:26 PM, Jeff Victor <jeff.j.vic...@gmail.com> wrote:
> On Tue, Mar 3, 2009 at 8:39 PM, Gael <gael.marti...@gmail.com> wrote:
>> Many thanks to  Bob Netherton and Jeff for their quick help on that painful 
>> issue.
>> The solution was to use psrset -f on the heavily used pset.
>> It is fully supported and a recommended situation when CPU starvation causes
>> interrupts not to be serviced in time and they get lost.   Credit goes to 
>> Rickey Weisner for this tip.
>> I have monitored that zone today for multiple hours without seeing any
>> packet loss while it was cranking up its cpu usage...
>> Jeff, following a previous mail today, as a fervent customer ;), I would
>> love to see that feature directly accessible thru the zone configuration to
>> avoid having to create a script and a dirty workaround to enable that
>> feature on boot. Is there a RFE # out there that I can be added to thru Sun
>> Support ? Got a case opened on that issue.
> Yes, the CR is 6199531 - "Device interrupts not bound to cpus
> configured within a nonglobal zone"
> Please ask your contact in Sun Service to add an SR for you.
>> Will continue to monitor the situation for a few days, and if I see anything 
>> wrong, I will update that thread
>> On Tue, Mar 3, 2009 at 2:19 PM, Jeff Victor <jeff.j.vic...@gmail.com> wrote:
>>> Hello Gael,
>>> On Mon, Mar 2, 2009 at 10:08 PM, Gael <gael.marti...@gmail.com> wrote:
>>> >
>>> > Got a zone running SAS with cpu capping enabled using a processor set as 
>>> > we
>>> > see a few processes using quite a bit of cpu there too often.
>>> Is that zone assigned to a resource pool, or is it using the dedicated-cpus 
>>> feature?
>>> > When the process is running (chewing 100% of its pset), the frame nic
>>> > (server is a E2900 with a ce interface) is dropping 20-30 % of its packets
>>> > causing a headache.
>>> My first guess is that the NICs interrupts are going to a CPU that the
>>> zone is using, and the CPU doesn't have enough power to run the zone's
>>> workload *and* be an effective NIC interrupt handler.
>>> Please run the "intrstat" command as root in the global zone, to
>>> determine which CPU is handling interrupts for that NIC. Also, check
>>> which CPU(s) that zone can use.
>>> Please let us know what you learn from those.
>>> > Doesn't appear to be a network load issue. Not a lot happening there 
>>> > visibly.
>>> >
>>> > With Solaris 10 u4 or u6, what elegant way would you recommend to avoid 
>>> > that
>>> > disruption caused by a single zone ?
zones-discuss mailing list

Reply via email to