Re: [zones-discuss] Zone in a pset with high load generating high packet loss at the frame level
1. Do you use set pool= anymore, now that the dedicated-cpu feature exists? Until Oracle develops a more rational licensing scheme you should expect this feature to be in use. I may have many Oracle instances, each in a separate zone, using the same pool. The sampling on this discussion list may not give you a good idea of its use. Might pose this question on your blog as well ? That said, this requires manual configuration of the pool. I don't think it would be asking too much for customers using this feature to also set up a boot time service (SMF or RC) to disable interrupts on all CPUs in the pool. If needed (may not always be needed). 2. Is it sufficient to simply disable interrupts on a zone's pset? I like your idea of turning off interrupts for dynamic resource pools under zoneadm/rcapd control, and leaving it a configurable item. I would also think that when CPUs are removed from the pool that interrupts should be turned back on unless given to a another pool with interrupts=disabled. I would hate for several zone reboots to turn off interrupts to all CPUs :-( Bob ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Zone in a pset with high load generating high packet loss at the frame level
On Wed, Mar 4, 2009 at 9:06 AM, Jeff Victor jeff.j.vic...@gmail.com wrote: Some questions: 1. Do you use set pool= anymore, now that the dedicated-cpu feature exists? We got over one hundred physical frames running zones here, covering nearly all versions of Solaris 10, we are currently sticking to set pool until we can get the whole environment upgraded. Before that, cannot afford to have the whole team of admins handling zones differently depending on the OS version. Headache... 2. Is it sufficient to simply disable interrupts on a zone's pset? In our case, we do pset only when licensing requires it (aka oracle,datastage,sybase,borland apps) or when the applications behave poorly and we keep hearing that by lack of budget/resources, the issue cannot be addressed and without direct impact on the business itself, nothing will change. What about creating an IO pset, and then disabling the interrupt on everything else while using it as a FSS pool or psets pools ? Very similar to ldom I would think... Regards -- Gael Martinez ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Zone in a pset with high load generating high packet loss at the frame level
Thanks for the great feedback Gael. Comments below. On Thu, Mar 5, 2009 at 11:00 AM, Gael gael.marti...@gmail.com wrote: On Wed, Mar 4, 2009 at 9:06 AM, Jeff Victor jeff.j.vic...@gmail.com wrote: Some questions: 1. Do you use set pool= anymore, now that the dedicated-cpu feature exists? We got over one hundred physical frames running zones here, covering nearly all versions of Solaris 10, we are currently sticking to set pool until we can get the whole environment upgraded. Before that, cannot afford to have the whole team of admins handling zones differently depending on the OS version. Headache... It is now clear to me that this feature would need to support disabling interrupts when a zone uses set pool=. Currently, all pool attributes are configured using the pool tools (poolcfg, pooladm) and I don't see any reason to not continue. When I write this up, it will fulfill that need. 2. Is it sufficient to simply disable interrupts on a zone's pset? In our case, we do pset only when licensing requires it (aka oracle,datastage,sybase,borland apps) or when the applications behave poorly and we keep hearing that by lack of budget/resources, the issue cannot be addressed and without direct impact on the business itself, nothing will change. Gael, I realized that my question was vague. When you use a pool, you're using a pset. Do you mean that you only use pools and psets when licensing requires it? Also, I couldn't tell how the comment responded to the question. What about creating an IO pset, and then disabling the interrupt on everything else while using it as a FSS pool or psets pools ? Very similar to ldom I would think... Yes, that occurred to me, too. You can do that now, either with a pset that's being used by a zone or with the default pset. But I'm not convinced there's enough reason to separate an I/O pset from the default pset. There's great potential for wasted CPU cycles. --JeffV ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Zone in a pset with high load generating high packet loss at the frame level
On Thu, Mar 05, 2009 at 01:22:25PM -0500, Jeff Victor wrote: Thanks for the great feedback Gael. Comments below. On Thu, Mar 5, 2009 at 11:00 AM, Gael gael.marti...@gmail.com wrote: On Wed, Mar 4, 2009 at 9:06 AM, Jeff Victor jeff.j.vic...@gmail.com wrote: Some questions: 1. Do you use set pool= anymore, now that the dedicated-cpu feature exists? We got over one hundred physical frames running zones here, covering nearly all versions of Solaris 10, we are currently sticking to set pool until we can get the whole environment upgraded. Before that, cannot afford to have the whole team of admins handling zones differently depending on the OS version. Headache... It is now clear to me that this feature would need to support disabling interrupts when a zone uses set pool=. Currently, all pool attributes are configured using the pool tools (poolcfg, pooladm) and I don't see any reason to not continue. When I write this up, it will fulfill that need. Ae you proposing that we add support for pset-interrupt disposition config to the pools framework? Such as a property on a pool-pset boolean pset.interrupts = false?? I think the right solution for pool= is this or similar. It could also be a string value, such as: none no interrupts handled on cpus in the pool-pset. zone Device interrupts for bound zones are serviced. any Any device interrupts can be dispatched to the pset. Zonecfg could make use of these pool-pset properties to implement the desired behavior for dedicated-cpu. The default value should be any. zonecfg should set zone for all dedicated-cpu zones. zoneadm could warn if pool= is set, the zone has dedicated devices, zone the pset for that pool has not been configured to be zone. legacy psets (psrset) could be extended to support this property via some new flags. Ther other part of this is how to reconsile zonecfg and/or pools settings for interrupts, with device-cpu mappings that are specified via dladm. Currently, dladm allows the specification of a list of cpu ids. Another way to approach this would be to point dladm directly at the desired pool. -Steve 2. Is it sufficient to simply disable interrupts on a zone's pset? In our case, we do pset only when licensing requires it (aka oracle,datastage,sybase,borland apps) or when the applications behave poorly and we keep hearing that by lack of budget/resources, the issue cannot be addressed and without direct impact on the business itself, nothing will change. Gael, I realized that my question was vague. When you use a pool, you're using a pset. Do you mean that you only use pools and psets when licensing requires it? Also, I couldn't tell how the comment responded to the question. What about creating an IO pset, and then disabling the interrupt on everything else while using it as a FSS pool or psets pools ? Very similar to ldom I would think... Yes, that occurred to me, too. You can do that now, either with a pset that's being used by a zone or with the default pset. But I'm not convinced there's enough reason to separate an I/O pset from the default pset. There's great potential for wasted CPU cycles. --JeffV ___ zones-discuss mailing list zones-discuss@opensolaris.org ___ zones-discuss mailing list zones-discuss@opensolaris.org
Re: [zones-discuss] Zone in a pset with high load generating high packet loss at the frame level
On Thu, Mar 5, 2009 at 1:48 PM, Steve Lawrence stephen.lawre...@sun.com wrote: On Thu, Mar 05, 2009 at 01:22:25PM -0500, Jeff Victor wrote: On Thu, Mar 5, 2009 at 11:00 AM, Gael gael.marti...@gmail.com wrote: On Wed, Mar 4, 2009 at 9:06 AM, Jeff Victor jeff.j.vic...@gmail.com wrote: Some questions: 1. Do you use set pool= anymore, now that the dedicated-cpu feature exists? It is now clear to me that this feature would need to support disabling interrupts when a zone uses set pool=. Currently, all pool attributes are configured using the pool tools (poolcfg, pooladm) and I don't see any reason to not continue. When I write this up, it will fulfill that need. Ae you proposing that we add support for pset-interrupt disposition config to the pools framework? Such as a property on a pool-pset boolean pset.interrupts = false?? The short answer is yes. BobN and I came to the same conclusion just a few hours ago... :-) CPUs already have cpu.status which can be on-line, no-intr (LWPs but no interrupt handlers), or off-line (no LWPs but still able to handle interrupts). A pset.interrupts field would allow Solaris to set cpu.status on CPUs as they enter the pset. Zones could then use that so we can increase their isolation. When a CPU re-enters the default pset, it becomes able to handle interrupts again. When needed, intrd will give it one (or more). I think the right solution for pool= is this or similar. It could also be a string value, such as: none no interrupts handled on cpus in the pool-pset. zone Device interrupts for bound zones are serviced. any Any device interrupts can be dispatched to the pset. I don't see how we could do zone in all situations - there isn't a 1:1 mapping between zone and device (except for exclusive-IP). Imagine zoneA and zoneB on a pset (psetAB) with pset.interrupts=zone. Further, zoneA and zoneC share e1000g0, but zoneB doesn't. Finally, zoneC has its own pset. Where does the interrupt handler for e1000g0 go - psetAB or psetC? Or are you suggesting that interrupts from one device can be intercepted and diverted to a CPU associated with a specific pset, based on which process the interrupt is/should be associated with? Or am I misunderstanding the description of zone? Zonecfg could make use of these pool-pset properties to implement the desired behavior for dedicated-cpu. Exactly. The default value should be any. zonecfg should set zone for all dedicated-cpu zones. zoneadm could warn if pool= is set, the zone has dedicated devices, zone the pset for that pool has not been configured to be zone. The only devices we can be sure are dedicated for the boot-session of a zone are NICs. So this whole segregate the interrupts per zone/pset combo will be limited at best. It would be nice if we could generalize it like you say, but I don't think it's workable yet. legacy psets (psrset) could be extended to support this property via some new flags. Ther other part of this is how to reconsile zonecfg and/or pools settings for interrupts, with device-cpu mappings that are specified via dladm. Currently, dladm allows the specification of a list of cpu ids. Another way to approach this would be to point dladm directly at the desired pool. Which currently are you on? :-) I'm on NV94 and I don't see anything like that in dladm(1M) I'm beginning to think this is really a two-phase project: * Phase 1: make it easier to disable interrupts on a zone's pset (one configured with the pool property or dedicated-cpu resource) * Phase 2: optimize this by enabling a zone's pset to handle interrupts from a device which is exclusively bound to this zone. I think that most people that need any of this only need Phase 1. Philosophically, shifting interrupt handlers into the default pset is consistent with the original zones principles: hardware is part of the platform, not part of a zone. So I'm not even convinced that we should be allowing zones' psets to selectively attract interrupt handlers. Great conversation! --JeffV 2. Is it sufficient to simply disable interrupts on a zone's pset? In our case, we do pset only when licensing requires it (aka oracle,datastage,sybase,borland apps) or when the applications behave poorly and we keep hearing that by lack of budget/resources, the issue cannot be addressed and without direct impact on the business itself, nothing will change. Gael, I realized that my question was vague. When you use a pool, you're using a pset. Do you mean that you only use pools and psets when licensing requires it? Also, I couldn't tell how the comment responded to the question. What about creating an IO pset, and then disabling the interrupt on everything else while using it as a FSS pool or psets pools ? Very similar to ldom I would think... Yes, that occurred to me, too. You can do that now, either with a pset that's being used by a zone or with the
Re: [zones-discuss] Zone in a pset with high load generating high packet loss at the frame level
On Thu, Mar 05, 2009 at 04:12:19PM -0500, Jeff Victor wrote: On Thu, Mar 5, 2009 at 1:48 PM, Steve Lawrence stephen.lawre...@sun.com wrote: On Thu, Mar 05, 2009 at 01:22:25PM -0500, Jeff Victor wrote: On Thu, Mar 5, 2009 at 11:00 AM, Gael gael.marti...@gmail.com wrote: On Wed, Mar 4, 2009 at 9:06 AM, Jeff Victor jeff.j.vic...@gmail.com wrote: Some questions: 1. Do you use set pool= anymore, now that the dedicated-cpu feature exists? It is now clear to me that this feature would need to support disabling interrupts when a zone uses set pool=. Currently, all pool attributes are configured using the pool tools (poolcfg, pooladm) and I don't see any reason to not continue. When I write this up, it will fulfill that need. Ae you proposing that we add support for pset-interrupt disposition config to the pools framework? Such as a property on a pool-pset boolean pset.interrupts = false?? The short answer is yes. BobN and I came to the same conclusion just a few hours ago... :-) CPUs already have cpu.status which can be on-line, no-intr (LWPs but no interrupt handlers), or off-line (no LWPs but still able to handle interrupts). A pset.interrupts field would allow Solaris to set cpu.status on CPUs as they enter the pset. Zones could then use that so we can increase their isolation. When a CPU re-enters the default pset, it becomes able to handle interrupts again. When needed, intrd will give it one (or more). I think the right solution for pool= is this or similar. It could also be a string value, such as: none no interrupts handled on cpus in the pool-pset. zone Device interrupts for bound zones are serviced. any Any device interrupts can be dispatched to the pset. I don't see how we could do zone in all situations - there isn't a 1:1 mapping between zone and device (except for exclusive-IP). Imagine zoneA and zoneB on a pset (psetAB) with pset.interrupts=zone. Further, zoneA and zoneC share e1000g0, but zoneB doesn't. Finally, zoneC has its own pset. Where does the interrupt handler for e1000g0 go - psetAB or psetC? I was thinking in the exclusive case. For shared stack zones, the devices would all be bound to the global zone's (aka default) pset. Or are you suggesting that interrupts from one device can be intercepted and diverted to a CPU associated with a specific pset, based on which process the interrupt is/should be associated with? No, although I'm not sure what is configurable for vnics. It may be possible for shared stack zones using a exclusive vnic (not exclusive stack) to have some of the vnic workload bound to it's pset. Or am I misunderstanding the description of zone? Zonecfg could make use of these pool-pset properties to implement the desired behavior for dedicated-cpu. Exactly. The default value should be any. zonecfg should set zone for all dedicated-cpu zones. zoneadm could warn if pool= is set, the zone has dedicated devices, zone the pset for that pool has not been configured to be zone. The only devices we can be sure are dedicated for the boot-session of a zone are NICs. So this whole segregate the interrupts per zone/pset combo will be limited at best. It would be nice if we could generalize it like you say, but I don't think it's workable yet. Agreed. This is really just for network devices at this point. legacy psets (psrset) could be extended to support this property via some new flags. Ther other part of this is how to reconsile zonecfg and/or pools settings for interrupts, with device-cpu mappings that are specified via dladm. Currently, dladm allows the specification of a list of cpu ids. Another way to approach this would be to point dladm directly at the desired pool. Which currently are you on? :-) I'm on NV94 and I don't see anything like that in dladm(1M) Crossbow when into 105. http://blogs.sun.com/nitin/entry/resource_allocation_for_network_processing I'm beginning to think this is really a two-phase project: * Phase 1: make it easier to disable interrupts on a zone's pset (one configured with the pool property or dedicated-cpu resource) * Phase 2: optimize this by enabling a zone's pset to handle interrupts from a device which is exclusively bound to this zone. As long as phase one is compatable with phase two, meaning that this case such as this one are properly defined: 1. pool mypool has property interrupts=disabled. 2. Zone has pool=mypool 3. Zone property stating to bind network interrupts to pool. One solution would be to alow this config, and bind the net interrupts to mypool anyway. Another would be to only allow auto-net-binding in zonecfg when using dedicated-cpu. I think that most people that need any of this only need Phase 1. Agreed. Philosophically, shifting interrupt handlers into the default pset is consistent with the
