Re: SLAB cache reaper on isolated cpus
Christoph Lameter wrote: > On Tue, 20 Feb 2007, Max Krasnyansky wrote: > >> Suppose I need to isolate a CPU. We already support at the scheduler and >> irq levels (irq affinity). But I want to go a bit further and avoid >> doing kernel work on isolated cpus as much as possible. For example I >> would not want to schedule work queues and stuff on them. Currently >> there are just a few users of the schedule_delayed_work_on(). cpufreq >> (don't care for isolation purposes), oprofile (same here) and slab. For >> the slab it'd be nice to run the reaper on some other CPU. But you're >> saying that locking depends on CPU pinning. Is there any other option >> besides disabling cache reap ? Is there a way for example to constraint >> the slabs on CPU X to not exceed N megs ? > > There is no way to constrain the amount of slab work. In order to make the > above work we would have to disable the per cpu caches for a certain cpu. > Then there would be no need to run the cache reaper at all. > > To some extend such functionality already exists. F.e. kmalloc_node() > already bypasses the per cpu caches (most of the time). kmalloc_node will > have to take a spinlock on a shared cacheline on each invocation. kmalloc > does only touch per cpu data during regular operations. Thus kmalloc() is > much > faster than kmalloc_node() and the cachelines for kmalloc() can be kept in > the per cpu cache. > > If we could disable all per cpu caches for certain cpus then you could > make this work. All slab OS interference would be off the processor. Hmm. That's an idea. I'll play with it later today or tomorrow. Thanks Max - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: SLAB cache reaper on isolated cpus
On Tue, 20 Feb 2007, Max Krasnyansky wrote: > Suppose I need to isolate a CPU. We already support at the scheduler and > irq levels (irq affinity). But I want to go a bit further and avoid > doing kernel work on isolated cpus as much as possible. For example I > would not want to schedule work queues and stuff on them. Currently > there are just a few users of the schedule_delayed_work_on(). cpufreq > (don't care for isolation purposes), oprofile (same here) and slab. For > the slab it'd be nice to run the reaper on some other CPU. But you're > saying that locking depends on CPU pinning. Is there any other option > besides disabling cache reap ? Is there a way for example to constraint > the slabs on CPU X to not exceed N megs ? There is no way to constrain the amount of slab work. In order to make the above work we would have to disable the per cpu caches for a certain cpu. Then there would be no need to run the cache reaper at all. To some extend such functionality already exists. F.e. kmalloc_node() already bypasses the per cpu caches (most of the time). kmalloc_node will have to take a spinlock on a shared cacheline on each invocation. kmalloc does only touch per cpu data during regular operations. Thus kmalloc() is much faster than kmalloc_node() and the cachelines for kmalloc() can be kept in the per cpu cache. If we could disable all per cpu caches for certain cpus then you could make this work. All slab OS interference would be off the processor. - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
SLAB cache reaper on isolated cpus
Christoph Lameter wrote: On Tue, 20 Feb 2007, Max Krasnyansky wrote: Ok. Sounds like disabling cache_reaper is a better option for now. Like you said it's unlikely that slabs will grow much if that cpu is not heavily used by the kernel. Running for prolonged times without cache_reaper is no good. What we are talking about here is to disable the cache_reaper during cpu shutdown. The slab cpu shutdown will clean the per cpu caches anyways so we really do not need the slab_reaper running during cpu shutdown. Ok. Let me restart the thread so that we're not confusing two issues :). I'm not talking about CPU shutdown or CPU hotplug in general. My proposal seemed related to the CPU shutdown issue that you guys were discussing, but it turns out it's not. Suppose I need to isolate a CPU. We already support at the scheduler and irq levels (irq affinity). But I want to go a bit further and avoid doing kernel work on isolated cpus as much as possible. For example I would not want to schedule work queues and stuff on them. Currently there are just a few users of the schedule_delayed_work_on(). cpufreq (don't care for isolation purposes), oprofile (same here) and slab. For the slab it'd be nice to run the reaper on some other CPU. But you're saying that locking depends on CPU pinning. Is there any other option besides disabling cache reap ? Is there a way for example to constraint the slabs on CPU X to not exceed N megs ? Max - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
SLAB cache reaper on isolated cpus
Christoph Lameter wrote: On Tue, 20 Feb 2007, Max Krasnyansky wrote: Ok. Sounds like disabling cache_reaper is a better option for now. Like you said it's unlikely that slabs will grow much if that cpu is not heavily used by the kernel. Running for prolonged times without cache_reaper is no good. What we are talking about here is to disable the cache_reaper during cpu shutdown. The slab cpu shutdown will clean the per cpu caches anyways so we really do not need the slab_reaper running during cpu shutdown. Ok. Let me restart the thread so that we're not confusing two issues :). I'm not talking about CPU shutdown or CPU hotplug in general. My proposal seemed related to the CPU shutdown issue that you guys were discussing, but it turns out it's not. Suppose I need to isolate a CPU. We already support at the scheduler and irq levels (irq affinity). But I want to go a bit further and avoid doing kernel work on isolated cpus as much as possible. For example I would not want to schedule work queues and stuff on them. Currently there are just a few users of the schedule_delayed_work_on(). cpufreq (don't care for isolation purposes), oprofile (same here) and slab. For the slab it'd be nice to run the reaper on some other CPU. But you're saying that locking depends on CPU pinning. Is there any other option besides disabling cache reap ? Is there a way for example to constraint the slabs on CPU X to not exceed N megs ? Max - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: SLAB cache reaper on isolated cpus
On Tue, 20 Feb 2007, Max Krasnyansky wrote: Suppose I need to isolate a CPU. We already support at the scheduler and irq levels (irq affinity). But I want to go a bit further and avoid doing kernel work on isolated cpus as much as possible. For example I would not want to schedule work queues and stuff on them. Currently there are just a few users of the schedule_delayed_work_on(). cpufreq (don't care for isolation purposes), oprofile (same here) and slab. For the slab it'd be nice to run the reaper on some other CPU. But you're saying that locking depends on CPU pinning. Is there any other option besides disabling cache reap ? Is there a way for example to constraint the slabs on CPU X to not exceed N megs ? There is no way to constrain the amount of slab work. In order to make the above work we would have to disable the per cpu caches for a certain cpu. Then there would be no need to run the cache reaper at all. To some extend such functionality already exists. F.e. kmalloc_node() already bypasses the per cpu caches (most of the time). kmalloc_node will have to take a spinlock on a shared cacheline on each invocation. kmalloc does only touch per cpu data during regular operations. Thus kmalloc() is much faster than kmalloc_node() and the cachelines for kmalloc() can be kept in the per cpu cache. If we could disable all per cpu caches for certain cpus then you could make this work. All slab OS interference would be off the processor. - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: SLAB cache reaper on isolated cpus
Christoph Lameter wrote: On Tue, 20 Feb 2007, Max Krasnyansky wrote: Suppose I need to isolate a CPU. We already support at the scheduler and irq levels (irq affinity). But I want to go a bit further and avoid doing kernel work on isolated cpus as much as possible. For example I would not want to schedule work queues and stuff on them. Currently there are just a few users of the schedule_delayed_work_on(). cpufreq (don't care for isolation purposes), oprofile (same here) and slab. For the slab it'd be nice to run the reaper on some other CPU. But you're saying that locking depends on CPU pinning. Is there any other option besides disabling cache reap ? Is there a way for example to constraint the slabs on CPU X to not exceed N megs ? There is no way to constrain the amount of slab work. In order to make the above work we would have to disable the per cpu caches for a certain cpu. Then there would be no need to run the cache reaper at all. To some extend such functionality already exists. F.e. kmalloc_node() already bypasses the per cpu caches (most of the time). kmalloc_node will have to take a spinlock on a shared cacheline on each invocation. kmalloc does only touch per cpu data during regular operations. Thus kmalloc() is much faster than kmalloc_node() and the cachelines for kmalloc() can be kept in the per cpu cache. If we could disable all per cpu caches for certain cpus then you could make this work. All slab OS interference would be off the processor. Hmm. That's an idea. I'll play with it later today or tomorrow. Thanks Max - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
