On Thu, Mar 5, 2009 at 1:48 PM, Steve Lawrence <stephen.lawre...@sun.com> wrote:
> On Thu, Mar 05, 2009 at 01:22:25PM -0500, Jeff Victor wrote:
>> On Thu, Mar 5, 2009 at 11:00 AM, Gael <gael.marti...@gmail.com> wrote:
>> > On Wed, Mar 4, 2009 at 9:06 AM, Jeff Victor <jeff.j.vic...@gmail.com> 
>> > wrote:
>> >>
>> >> Some questions:
>> >> 1. Do you use "set pool=" anymore, now that the dedicated-cpu feature 
>> >> exists?
>> >
>> It is now clear to me that this feature would need to support
>> disabling interrupts when a zone uses "set pool=". Currently, all pool
>> attributes are configured using the pool tools (poolcfg, pooladm) and
>> I don't see any reason to not continue. When I write this up, it will
>> fulfill that need.
> Ae you proposing that we add support for pset-interrupt disposition config
> to the pools framework?  Such as a property on a pool-pset
>        "boolean pset.interrupts = false"??

The short answer is "yes."  BobN and I came to the same conclusion
just a few hours ago... :-)

CPUs already have cpu.status which can be  on-line, no-intr (LWPs but
no interrupt handlers), or off-line (no LWPs but still able to handle
interrupts). A pset.interrupts field would allow Solaris to set
cpu.status on CPUs as they enter the pset.  Zones could then use that
so we can increase their isolation. When a CPU re-enters the default
pset, it becomes able to handle interrupts again. When needed, intrd
will give it one (or more).

> I think the right solution for "pool=" is this or similar.  It could also
> be a string value, such as:
>        "none"  no interrupts handled on cpus in the pool-pset.
>        "zone"  Device interrupts for bound zones are serviced.
>        "any"   Any device interrupts can be dispatched to the pset.

I don't see how we could do "zone" in all situations - there isn't a
1:1 mapping between zone and device (except for exclusive-IP).

 Imagine zoneA and zoneB on a pset (psetAB) with pset.interrupts=zone.
Further, zoneA and zoneC share e1000g0, but zoneB doesn't. Finally,
zoneC has its own pset. Where does the interrupt handler for e1000g0
go - psetAB or psetC?

Or are you suggesting that interrupts from one device can be
intercepted and diverted to a CPU associated with a specific pset,
based on which process the interrupt is/should be associated with?

Or am I misunderstanding the description of "zone"?

> Zonecfg could make use of these pool-pset properties to implement the
> desired behavior for "dedicated-cpu".


> The default value should be "any".  zonecfg should set "zone" for all
> dedicated-cpu zones.  zoneadm could warn if "pool=" is set, the zone has
> dedicated devices, zone the pset for that pool has not been configured to
> be "zone".

The only devices we can be sure are dedicated for the boot-session of
a zone are NICs. So this whole "segregate the interrupts per zone/pset
combo" will be limited at best. It would be nice if we could
generalize it like you say, but I don't think it's workable yet.

> legacy psets (psrset) could be extended to support this property via some new 
> flags.
> Ther other part of this is how to reconsile zonecfg and/or pools settings
> for interrupts, with device-cpu mappings that are specified via dladm.
> Currently, dladm allows the specification of a list of cpu ids.  Another
> way to approach this would be to point dladm directly at the desired pool.

Which "currently" are you on? :-)  I'm on NV94 and I don't see
anything like that in dladm(1M)

I'm beginning to think this is really a two-phase project:
* Phase 1: make it easier to disable interrupts on a zone's pset (one
configured with the pool property or dedicated-cpu resource)
* Phase 2: optimize this by enabling a zone's pset to handle
interrupts from a device which is exclusively bound to this zone.

I think that most people that need any of this only need Phase 1.
Philosophically, shifting interrupt handlers into the default pset is
consistent with the original zones principles: hardware is part of the
platform, not part of a zone. So I'm not even convinced that we should
be allowing zones' psets to selectively "attract"  interrupt handlers.

Great conversation!


>> >> 2. Is it sufficient to simply disable interrupts on a zone's pset?
>> >
>> > In our case, we do pset only when licensing requires it (aka
>> > oracle,datastage,sybase,borland apps) or when the applications behave 
>> > poorly
>> > and we keep hearing that by lack of budget/resources, the issue cannot be
>> > addressed and without direct impact on the business itself, nothing will
>> > change.
>> Gael, I realized that my question was vague. When you use a pool,
>> you're using a pset. Do you mean that you only use pools and psets
>> when licensing requires it?
>> Also, I couldn't tell how the comment responded to the question.
>> > What about creating an IO pset, and then disabling the interrupt on
>> > everything else while using it as a FSS pool or psets pools ? Very similar
>> > to ldom I would think...
>> Yes, that occurred to me, too. You can do that now, either with a pset
>> that's being used by a zone or with the default pset. But I'm not
>> convinced there's enough reason to separate an I/O pset from the
>> default pset. There's great potential for wasted CPU cycles.
>> --JeffV
> - Show quoted text -
>> _______________________________________________
>> zones-discuss mailing list
>> zones-discuss@opensolaris.org

zones-discuss mailing list

Reply via email to