Re: [patch 00/19] VM pageout scalability improvements

2008-01-11 Thread Rik van Riel
On Fri, 11 Jan 2008 16:11:15 +0530 Balbir Singh <[EMAIL PROTECTED]> wrote: > I've just started the patch series, the compile fails for me on a > powerpc box. global_lru_pages() is defined under CONFIG_PM, but used > else where in mm/page-writeback.c. None of the global_lru_pages() > parameters

Re: [patch 00/19] VM pageout scalability improvements

2008-01-11 Thread Balbir Singh
* Rik van Riel <[EMAIL PROTECTED]> [2008-01-08 15:59:39]: > Changelog: > - merge memcontroller split LRU code into the main split LRU patch, > since it is not functionally different (it was split up only to help > people who had seen the last version of the patch series review it) Hi, Rik,

Re: [patch 00/19] VM pageout scalability improvements

2008-01-11 Thread Balbir Singh
* Rik van Riel <[EMAIL PROTECTED]> [2008-01-08 15:59:39]: > On large memory systems, the VM can spend way too much time scanning > through pages that it cannot (or should not) evict from memory. Not > only does it use up CPU time, but it also provokes lock contention > and can leave large systems

Re: [patch 00/19] VM pageout scalability improvements

2008-01-11 Thread Balbir Singh
* Rik van Riel [EMAIL PROTECTED] [2008-01-08 15:59:39]: On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems under

Re: [patch 00/19] VM pageout scalability improvements

2008-01-11 Thread Balbir Singh
* Rik van Riel [EMAIL PROTECTED] [2008-01-08 15:59:39]: Changelog: - merge memcontroller split LRU code into the main split LRU patch, since it is not functionally different (it was split up only to help people who had seen the last version of the patch series review it) Hi, Rik, I see

Re: [patch 00/19] VM pageout scalability improvements

2008-01-11 Thread Rik van Riel
On Fri, 11 Jan 2008 16:11:15 +0530 Balbir Singh [EMAIL PROTECTED] wrote: I've just started the patch series, the compile fails for me on a powerpc box. global_lru_pages() is defined under CONFIG_PM, but used else where in mm/page-writeback.c. None of the global_lru_pages() parameters depend

Re: [patch 00/19] VM pageout scalability improvements

2008-01-10 Thread Mike Snitzer
On Jan 10, 2008 10:41 AM, Rik van Riel <[EMAIL PROTECTED]> wrote: > > On Wed, 9 Jan 2008 23:39:02 -0500 > "Mike Snitzer" <[EMAIL PROTECTED]> wrote: > > > How much trouble am I asking for if I were to try to get your patchset > > to fly on a fairly recent "stable" kernel (e.g. 2.6.22.15)? If > >

Re: [patch 00/19] VM pageout scalability improvements

2008-01-10 Thread Rik van Riel
On Wed, 9 Jan 2008 23:39:02 -0500 "Mike Snitzer" <[EMAIL PROTECTED]> wrote: > How much trouble am I asking for if I were to try to get your patchset > to fly on a fairly recent "stable" kernel (e.g. 2.6.22.15)? If > workable, is such an effort before it's time relative to your TODO? Quite a bit

Re: [patch 00/19] VM pageout scalability improvements

2008-01-10 Thread Rik van Riel
On Wed, 9 Jan 2008 23:39:02 -0500 Mike Snitzer [EMAIL PROTECTED] wrote: How much trouble am I asking for if I were to try to get your patchset to fly on a fairly recent stable kernel (e.g. 2.6.22.15)? If workable, is such an effort before it's time relative to your TODO? Quite a bit :) The

Re: [patch 00/19] VM pageout scalability improvements

2008-01-10 Thread Mike Snitzer
On Jan 10, 2008 10:41 AM, Rik van Riel [EMAIL PROTECTED] wrote: On Wed, 9 Jan 2008 23:39:02 -0500 Mike Snitzer [EMAIL PROTECTED] wrote: How much trouble am I asking for if I were to try to get your patchset to fly on a fairly recent stable kernel (e.g. 2.6.22.15)? If workable, is such

Re: [patch 00/19] VM pageout scalability improvements

2008-01-09 Thread Mike Snitzer
On Jan 8, 2008 3:59 PM, Rik van Riel <[EMAIL PROTECTED]> wrote: > On large memory systems, the VM can spend way too much time scanning > through pages that it cannot (or should not) evict from memory. Not > only does it use up CPU time, but it also provokes lock contention > and can leave large

Re: [patch 00/19] VM pageout scalability improvements

2008-01-09 Thread Mike Snitzer
On Jan 8, 2008 3:59 PM, Rik van Riel [EMAIL PROTECTED] wrote: On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems

[patch 00/19] VM pageout scalability improvements

2008-01-08 Thread Rik van Riel
On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems under memory presure in a catatonic state. Against 2.6.24-rc6-mm1

[patch 00/19] VM pageout scalability improvements

2008-01-08 Thread Rik van Riel
On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems under memory presure in a catatonic state. Against 2.6.24-rc6-mm1

Re: [patch 00/19] VM pageout scalability improvements

2008-01-07 Thread Rik van Riel
On Mon, 7 Jan 2008 11:07:54 -0800 (PST) Christoph Lameter <[EMAIL PROTECTED]> wrote: > On Fri, 4 Jan 2008, Lee Schermerhorn wrote: > > > We see this on both NUMA and non-NUMA. x86_64 and ia64. The basic > > criteria to reproduce is to be able to run thousands [or low 10s of > > thousands] of

Re: [patch 00/19] VM pageout scalability improvements

2008-01-07 Thread Christoph Lameter
On Fri, 4 Jan 2008, Lee Schermerhorn wrote: > We see this on both NUMA and non-NUMA. x86_64 and ia64. The basic > criteria to reproduce is to be able to run thousands [or low 10s of > thousands] of tasks, continually increasing the number until the system > just goes into reclaim. Instead of

Re: [patch 00/19] VM pageout scalability improvements

2008-01-07 Thread Rik van Riel
On Mon, 7 Jan 2008 19:06:10 +0900 KAMEZAWA Hiroyuki <[EMAIL PROTECTED]> wrote: > On Thu, 3 Jan 2008 12:00:00 -0500 > Rik van Riel <[EMAIL PROTECTED]> wrote: > > If there is no swap space, my VM code will not bother scanning > > any anon pages. This has the same effect as moving the pages > > to

Re: [patch 00/19] VM pageout scalability improvements

2008-01-07 Thread KAMEZAWA Hiroyuki
On Thu, 3 Jan 2008 12:00:00 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote: > On Thu, 03 Jan 2008 11:52:08 -0500 > Lee Schermerhorn <[EMAIL PROTECTED]> wrote: > > > Also, I should point out that the full noreclaim series includes a > > couple of other patches NOT posted here by Rik: > > > > 1)

Re: [patch 00/19] VM pageout scalability improvements

2008-01-07 Thread Rik van Riel
On Mon, 7 Jan 2008 19:06:10 +0900 KAMEZAWA Hiroyuki [EMAIL PROTECTED] wrote: On Thu, 3 Jan 2008 12:00:00 -0500 Rik van Riel [EMAIL PROTECTED] wrote: If there is no swap space, my VM code will not bother scanning any anon pages. This has the same effect as moving the pages to the

Re: [patch 00/19] VM pageout scalability improvements

2008-01-07 Thread Christoph Lameter
On Fri, 4 Jan 2008, Lee Schermerhorn wrote: We see this on both NUMA and non-NUMA. x86_64 and ia64. The basic criteria to reproduce is to be able to run thousands [or low 10s of thousands] of tasks, continually increasing the number until the system just goes into reclaim. Instead of

Re: [patch 00/19] VM pageout scalability improvements

2008-01-07 Thread Rik van Riel
On Mon, 7 Jan 2008 11:07:54 -0800 (PST) Christoph Lameter [EMAIL PROTECTED] wrote: On Fri, 4 Jan 2008, Lee Schermerhorn wrote: We see this on both NUMA and non-NUMA. x86_64 and ia64. The basic criteria to reproduce is to be able to run thousands [or low 10s of thousands] of tasks,

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Larry Woodman
Rik van Riel wrote: On Fri, 04 Jan 2008 17:34:00 +0100 Andi Kleen <[EMAIL PROTECTED]> wrote: Lee Schermerhorn <[EMAIL PROTECTED]> writes: We can easily [he says, glibly] reproduce the hang on the anon_vma lock Is that a NUMA platform? On non x86? Perhaps you just need queued

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Lee Schermerhorn
On Fri, 2008-01-04 at 17:34 +0100, Andi Kleen wrote: > Lee Schermerhorn <[EMAIL PROTECTED]> writes: > > > We can easily [he says, glibly] reproduce the hang on the anon_vma lock > > Is that a NUMA platform? On non x86? Perhaps you just need queued spinlocks? We see this on both NUMA and

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Rik van Riel
On Fri, 04 Jan 2008 17:34:00 +0100 Andi Kleen <[EMAIL PROTECTED]> wrote: > Lee Schermerhorn <[EMAIL PROTECTED]> writes: > > > We can easily [he says, glibly] reproduce the hang on the anon_vma lock > > Is that a NUMA platform? On non x86? Perhaps you just need queued spinlocks? I really think

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Andi Kleen
Lee Schermerhorn <[EMAIL PROTECTED]> writes: > We can easily [he says, glibly] reproduce the hang on the anon_vma lock Is that a NUMA platform? On non x86? Perhaps you just need queued spinlocks? -Andi -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Lee Schermerhorn
On Thu, 2008-01-03 at 17:00 -0500, Rik van Riel wrote: > On Thu, 03 Jan 2008 12:13:32 -0500 > Lee Schermerhorn <[EMAIL PROTECTED]> wrote: > > > Yes, but the problem, when it occurs, is very awkward. The system just > > hangs for hours/days spinning on the reverse mapping locks--in both > >

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Andi Kleen
Lee Schermerhorn [EMAIL PROTECTED] writes: We can easily [he says, glibly] reproduce the hang on the anon_vma lock Is that a NUMA platform? On non x86? Perhaps you just need queued spinlocks? -Andi -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Lee Schermerhorn
On Thu, 2008-01-03 at 17:00 -0500, Rik van Riel wrote: On Thu, 03 Jan 2008 12:13:32 -0500 Lee Schermerhorn [EMAIL PROTECTED] wrote: Yes, but the problem, when it occurs, is very awkward. The system just hangs for hours/days spinning on the reverse mapping locks--in both

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Lee Schermerhorn
On Fri, 2008-01-04 at 17:34 +0100, Andi Kleen wrote: Lee Schermerhorn [EMAIL PROTECTED] writes: We can easily [he says, glibly] reproduce the hang on the anon_vma lock Is that a NUMA platform? On non x86? Perhaps you just need queued spinlocks? We see this on both NUMA and non-NUMA.

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Rik van Riel
On Fri, 04 Jan 2008 17:34:00 +0100 Andi Kleen [EMAIL PROTECTED] wrote: Lee Schermerhorn [EMAIL PROTECTED] writes: We can easily [he says, glibly] reproduce the hang on the anon_vma lock Is that a NUMA platform? On non x86? Perhaps you just need queued spinlocks? I really think that the

Re: [patch 00/19] VM pageout scalability improvements

2008-01-04 Thread Larry Woodman
Rik van Riel wrote: On Fri, 04 Jan 2008 17:34:00 +0100 Andi Kleen [EMAIL PROTECTED] wrote: Lee Schermerhorn [EMAIL PROTECTED] writes: We can easily [he says, glibly] reproduce the hang on the anon_vma lock Is that a NUMA platform? On non x86? Perhaps you just need queued

Re: [patch 00/19] VM pageout scalability improvements

2008-01-03 Thread Rik van Riel
On Thu, 03 Jan 2008 12:13:32 -0500 Lee Schermerhorn <[EMAIL PROTECTED]> wrote: > Yes, but the problem, when it occurs, is very awkward. The system just > hangs for hours/days spinning on the reverse mapping locks--in both > page_referenced() and try_to_unmap(). No pages get reclaimed and NO OOM

Re: [patch 00/19] VM pageout scalability improvements

2008-01-03 Thread Lee Schermerhorn
On Thu, 2008-01-03 at 12:00 -0500, Rik van Riel wrote: > On Thu, 03 Jan 2008 11:52:08 -0500 > Lee Schermerhorn <[EMAIL PROTECTED]> wrote: > > > Also, I should point out that the full noreclaim series includes a > > couple of other patches NOT posted here by Rik: > > > > 1) treat swap backed

Re: [patch 00/19] VM pageout scalability improvements

2008-01-03 Thread Rik van Riel
On Thu, 03 Jan 2008 11:52:08 -0500 Lee Schermerhorn <[EMAIL PROTECTED]> wrote: > Also, I should point out that the full noreclaim series includes a > couple of other patches NOT posted here by Rik: > > 1) treat swap backed pages as nonreclaimable when no swap space is > available. This

Re: [patch 00/19] VM pageout scalability improvements

2008-01-03 Thread Lee Schermerhorn
On Wed, 2008-01-02 at 17:41 -0500, [EMAIL PROTECTED] wrote: > On large memory systems, the VM can spend way too much time scanning > through pages that it cannot (or should not) evict from memory. Not > only does it use up CPU time, but it also provokes lock contention > and can leave large

Re: [patch 00/19] VM pageout scalability improvements

2008-01-03 Thread Lee Schermerhorn
On Wed, 2008-01-02 at 17:41 -0500, [EMAIL PROTECTED] wrote: On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems

Re: [patch 00/19] VM pageout scalability improvements

2008-01-03 Thread Lee Schermerhorn
On Thu, 2008-01-03 at 12:00 -0500, Rik van Riel wrote: On Thu, 03 Jan 2008 11:52:08 -0500 Lee Schermerhorn [EMAIL PROTECTED] wrote: Also, I should point out that the full noreclaim series includes a couple of other patches NOT posted here by Rik: 1) treat swap backed pages as

Re: [patch 00/19] VM pageout scalability improvements

2008-01-03 Thread Rik van Riel
On Thu, 03 Jan 2008 11:52:08 -0500 Lee Schermerhorn [EMAIL PROTECTED] wrote: Also, I should point out that the full noreclaim series includes a couple of other patches NOT posted here by Rik: 1) treat swap backed pages as nonreclaimable when no swap space is available. This addresses a

Re: [patch 00/19] VM pageout scalability improvements

2008-01-03 Thread Rik van Riel
On Thu, 03 Jan 2008 12:13:32 -0500 Lee Schermerhorn [EMAIL PROTECTED] wrote: Yes, but the problem, when it occurs, is very awkward. The system just hangs for hours/days spinning on the reverse mapping locks--in both page_referenced() and try_to_unmap(). No pages get reclaimed and NO OOM

[patch 00/19] VM pageout scalability improvements

2008-01-02 Thread linux-kernel
On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems under memory presure in a catatonic state. Against 2.6.24-rc6-mm1

[patch 00/19] VM pageout scalability improvements

2008-01-02 Thread linux-kernel
On large memory systems, the VM can spend way too much time scanning through pages that it cannot (or should not) evict from memory. Not only does it use up CPU time, but it also provokes lock contention and can leave large systems under memory presure in a catatonic state. Against 2.6.24-rc6-mm1