On Thu, Mar 05, 2009 at 04:12:19PM -0500, Jeff Victor wrote:
> On Thu, Mar 5, 2009 at 1:48 PM, Steve Lawrence <stephen.lawre...@sun.com> 
> wrote:
> > On Thu, Mar 05, 2009 at 01:22:25PM -0500, Jeff Victor wrote:
> >> On Thu, Mar 5, 2009 at 11:00 AM, Gael <gael.marti...@gmail.com> wrote:
> >> > On Wed, Mar 4, 2009 at 9:06 AM, Jeff Victor <jeff.j.vic...@gmail.com> 
> >> > wrote:
> >> >>
> >> >> Some questions:
> >> >> 1. Do you use "set pool=" anymore, now that the dedicated-cpu feature 
> >> >> exists?
> >> >
> >> It is now clear to me that this feature would need to support
> >> disabling interrupts when a zone uses "set pool=". Currently, all pool
> >> attributes are configured using the pool tools (poolcfg, pooladm) and
> >> I don't see any reason to not continue. When I write this up, it will
> >> fulfill that need.
> >
> > Ae you proposing that we add support for pset-interrupt disposition config
> > to the pools framework?  Such as a property on a pool-pset
> >        "boolean pset.interrupts = false"??
> 
> The short answer is "yes."  BobN and I came to the same conclusion
> just a few hours ago... :-)
> 
> CPUs already have cpu.status which can be  on-line, no-intr (LWPs but
> no interrupt handlers), or off-line (no LWPs but still able to handle
> interrupts). A pset.interrupts field would allow Solaris to set
> cpu.status on CPUs as they enter the pset.  Zones could then use that
> so we can increase their isolation. When a CPU re-enters the default
> pset, it becomes able to handle interrupts again. When needed, intrd
> will give it one (or more).
> 
> 
> 
> > I think the right solution for "pool=" is this or similar.  It could also
> > be a string value, such as:
> >
> >        "none"  no interrupts handled on cpus in the pool-pset.
> >        "zone"  Device interrupts for bound zones are serviced.
> >        "any"   Any device interrupts can be dispatched to the pset.
> 
> I don't see how we could do "zone" in all situations - there isn't a
> 1:1 mapping between zone and device (except for exclusive-IP).
> 
>  Imagine zoneA and zoneB on a pset (psetAB) with pset.interrupts=zone.
> Further, zoneA and zoneC share e1000g0, but zoneB doesn't. Finally,
> zoneC has its own pset. Where does the interrupt handler for e1000g0
> go - psetAB or psetC?

I was thinking in the exclusive case.  For shared stack zones, the devices
would all be bound to the global zone's (aka default) pset.

> 
> Or are you suggesting that interrupts from one device can be
> intercepted and diverted to a CPU associated with a specific pset,
> based on which process the interrupt is/should be associated with?

No, although I'm not sure what is configurable for vnics.  It may be possible
for shared stack zones using a exclusive vnic (not exclusive stack) to have
some of the vnic workload bound to it's pset.

> 
> Or am I misunderstanding the description of "zone"?
> 
> 
> > Zonecfg could make use of these pool-pset properties to implement the
> > desired behavior for "dedicated-cpu".
> 
> Exactly.
> 
> > The default value should be "any".  zonecfg should set "zone" for all
> > dedicated-cpu zones.  zoneadm could warn if "pool=" is set, the zone has
> > dedicated devices, zone the pset for that pool has not been configured to
> > be "zone".
> 
> The only devices we can be sure are dedicated for the boot-session of
> a zone are NICs. So this whole "segregate the interrupts per zone/pset
> combo" will be limited at best. It would be nice if we could
> generalize it like you say, but I don't think it's workable yet.

Agreed.  This is really just for network devices at this point.

> 
> > legacy psets (psrset) could be extended to support this property via some 
> > new flags.
> >
> > Ther other part of this is how to reconsile zonecfg and/or pools settings
> > for interrupts, with device-cpu mappings that are specified via dladm.
> > Currently, dladm allows the specification of a list of cpu ids.  Another
> > way to approach this would be to point dladm directly at the desired pool.
> 
> Which "currently" are you on? :-)  I'm on NV94 and I don't see
> anything like that in dladm(1M)

Crossbow when into 105.

http://blogs.sun.com/nitin/entry/resource_allocation_for_network_processing

> 
> I'm beginning to think this is really a two-phase project:
> * Phase 1: make it easier to disable interrupts on a zone's pset (one
> configured with the pool property or dedicated-cpu resource)
> * Phase 2: optimize this by enabling a zone's pset to handle
> interrupts from a device which is exclusively bound to this zone.

As long as phase one is compatable with phase two, meaning that this
case such as this one are properly defined:
        1. pool mypool has property "interrupts=disabled".
        2. Zone has pool=mypool
        3. Zone property stating to bind network interrupts to pool.

One solution would be to alow this config, and bind the net interrupts to
mypool anyway.  Another would be to only allow auto-net-binding in zonecfg
when using dedicated-cpu.

> 
> I think that most people that need any of this only need Phase 1.

Agreed.

> Philosophically, shifting interrupt handlers into the default pset is
> consistent with the original zones principles: hardware is part of the
> platform, not part of a zone. So I'm not even convinced that we should
> be allowing zones' psets to selectively "attract"  interrupt handlers.

I agree.  It should be directed by a zonecfg property, not a pool property.
zonecfg should be used to state when to bind the interrupts to the target
pool.  Something senseable just needs to happen when that pool is defined
as no-intr (from phase 1).

-Steve

> 
> 
> Great conversation!
> 
> --JeffV
> 
> >> >> 2. Is it sufficient to simply disable interrupts on a zone's pset?
> >> >
> >> > In our case, we do pset only when licensing requires it (aka
> >> > oracle,datastage,sybase,borland apps) or when the applications behave 
> >> > poorly
> >> > and we keep hearing that by lack of budget/resources, the issue cannot be
> >> > addressed and without direct impact on the business itself, nothing will
> >> > change.
> >>
> >> Gael, I realized that my question was vague. When you use a pool,
> >> you're using a pset. Do you mean that you only use pools and psets
> >> when licensing requires it?
> >>
> >> Also, I couldn't tell how the comment responded to the question.
> >>
> >> > What about creating an IO pset, and then disabling the interrupt on
> >> > everything else while using it as a FSS pool or psets pools ? Very 
> >> > similar
> >> > to ldom I would think...
> >>
> >> Yes, that occurred to me, too. You can do that now, either with a pset
> >> that's being used by a zone or with the default pset. But I'm not
> >> convinced there's enough reason to separate an I/O pset from the
> >> default pset. There's great potential for wasted CPU cycles.
> >>
> >>
> >> --JeffV
> > - Show quoted text -
> >> _______________________________________________
> >> zones-discuss mailing list
> >> zones-discuss@opensolaris.org
> >
> 
> 
> 
> -- 
> --JeffV
_______________________________________________
zones-discuss mailing list
zones-discuss@opensolaris.org

Reply via email to