Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Paul Mackerras wrote: Rik van Riel writes: I guess we'll need to call tlb_remove_tlb_entry() inside the MADV_FREE code to keep powerpc happy. I don't see why; once ptep_test_and_clear_young has returned, the entry in the hash table has already been removed. OK, so this one won't be

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Paul Mackerras
Rik van Riel writes: > I guess we'll need to call tlb_remove_tlb_entry() inside the > MADV_FREE code to keep powerpc happy. I don't see why; once ptep_test_and_clear_young has returned, the entry in the hash table has already been removed. Adding the tlb_remove_tlb_entry call certainly won't do

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Andrew Morton
On Mon, 23 Apr 2007 22:53:49 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: > I don't see why we need the attached, but in case you find > a good reason, here's my signed-off-by line for Andrew :) Andew is in a defensive crouch trying to work his way through all the bugs he's been sent. After

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Nick Piggin wrote: What the tlb flush used to be able to assume is that the page has been removed from the pagetables when they are put in the tlb flush batch. I think this is still the case, to a degree. There should be no harm in removing the TLB entries after the page table has been

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Rik van Riel wrote: This should fix the MADV_FREE code for PPC's hashed tlb. Signed-off-by: Rik van Riel <[EMAIL PROTECTED]> --- Nick Piggin wrote: Nick Piggin wrote: 3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
This should fix the MADV_FREE code for PPC's hashed tlb. Signed-off-by: Rik van Riel <[EMAIL PROTECTED]> --- Nick Piggin wrote: Nick Piggin wrote: 3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka undefined behaviour

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Rik van Riel wrote: Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system. Signed-off-by: Rik van Riel <[EMAIL PROTECTED]> --- Nick Piggin wrote: 3) because of this, we can treat any such accesses as happening

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Rik van Riel wrote: First some ebizzy runs... This is interesting. Ginormous speedups in ebizzy[1] on my quad core test system. The following numbers are the average of 10 runs, since ebizzy shows some variability. You can see a big influence from the tlb batching and from Nick's madv_sem

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system. Signed-off-by: Rik van Riel <[EMAIL PROTECTED]> --- Nick Piggin wrote: 3) because of this, we can treat any such accesses as happening simultaneously with the

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Jakub Jelinek
On Mon, Apr 23, 2007 at 08:21:37PM +1000, Nick Piggin wrote: > I guess it is a good idea to batch these things. But can you > do that on all architectures? What happens if your tlb flush > happens after another thread already accesses it again, or > after it subsequently gets removed from the

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Rik van Riel wrote: Nick Piggin wrote: It looks like the tlb flushes (and IPIs) from zap_pte_range() could have been the problem. They're gone now. I guess it is a good idea to batch these things. But can you do that on all architectures? What happens if your tlb flush happens after

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Nick Piggin wrote: It looks like the tlb flushes (and IPIs) from zap_pte_range() could have been the problem. They're gone now. I guess it is a good idea to batch these things. But can you do that on all architectures? What happens if your tlb flush happens after another thread already

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Rik van Riel wrote: Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system. Signed-off-by: Rik van Riel <[EMAIL PROTECTED]> --- Rik van Riel wrote: I've added a 5th column, with just your mmap_sem patch and without my

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Nick Piggin wrote: I haven't tested your MADV_FREE patch yet. Good. It turned out that one behaved a bit strange without tlb batching anyway. I'm now running ebizzy across the whole set of kernels I tested before, and will post the results in a bit. -- Politics is the struggle between

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Nick Piggin wrote: Rik van Riel wrote: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails. Thanks! (I edited slightly so it doesn't wrap)

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system. Signed-off-by: Rik van Riel <[EMAIL PROTECTED]> --- Rik van Riel wrote: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system. Signed-off-by: Rik van Riel [EMAIL PROTECTED] --- Rik van Riel wrote: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Nick Piggin wrote: Rik van Riel wrote: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails. Thanks! (I edited slightly so it doesn't wrap)

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Nick Piggin wrote: I haven't tested your MADV_FREE patch yet. Good. It turned out that one behaved a bit strange without tlb batching anyway. I'm now running ebizzy across the whole set of kernels I tested before, and will post the results in a bit. -- Politics is the struggle between

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Rik van Riel wrote: Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system. Signed-off-by: Rik van Riel [EMAIL PROTECTED] --- Rik van Riel wrote: I've added a 5th column, with just your mmap_sem patch and without my

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Nick Piggin wrote: It looks like the tlb flushes (and IPIs) from zap_pte_range() could have been the problem. They're gone now. I guess it is a good idea to batch these things. But can you do that on all architectures? What happens if your tlb flush happens after another thread already

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Rik van Riel wrote: Nick Piggin wrote: It looks like the tlb flushes (and IPIs) from zap_pte_range() could have been the problem. They're gone now. I guess it is a good idea to batch these things. But can you do that on all architectures? What happens if your tlb flush happens after

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Jakub Jelinek
On Mon, Apr 23, 2007 at 08:21:37PM +1000, Nick Piggin wrote: I guess it is a good idea to batch these things. But can you do that on all architectures? What happens if your tlb flush happens after another thread already accesses it again, or after it subsequently gets removed from the address

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system. Signed-off-by: Rik van Riel [EMAIL PROTECTED] --- Nick Piggin wrote: 3) because of this, we can treat any such accesses as happening simultaneously with the

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Rik van Riel wrote: First some ebizzy runs... This is interesting. Ginormous speedups in ebizzy[1] on my quad core test system. The following numbers are the average of 10 runs, since ebizzy shows some variability. You can see a big influence from the tlb batching and from Nick's madv_sem

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Rik van Riel wrote: Use TLB batching for MADV_FREE. Adds another 10-15% extra performance to the MySQL sysbench results on my quad core system. Signed-off-by: Rik van Riel [EMAIL PROTECTED] --- Nick Piggin wrote: 3) because of this, we can treat any such accesses as happening

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
This should fix the MADV_FREE code for PPC's hashed tlb. Signed-off-by: Rik van Riel [EMAIL PROTECTED] --- Nick Piggin wrote: Nick Piggin wrote: 3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka undefined behaviour

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Nick Piggin
Rik van Riel wrote: This should fix the MADV_FREE code for PPC's hashed tlb. Signed-off-by: Rik van Riel [EMAIL PROTECTED] --- Nick Piggin wrote: Nick Piggin wrote: 3) because of this, we can treat any such accesses as happening simultaneously with the MADV_FREE and as illegal, aka

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Nick Piggin wrote: What the tlb flush used to be able to assume is that the page has been removed from the pagetables when they are put in the tlb flush batch. I think this is still the case, to a degree. There should be no harm in removing the TLB entries after the page table has been

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Andrew Morton
On Mon, 23 Apr 2007 22:53:49 -0400 Rik van Riel [EMAIL PROTECTED] wrote: I don't see why we need the attached, but in case you find a good reason, here's my signed-off-by line for Andrew :) Andew is in a defensive crouch trying to work his way through all the bugs he's been sent. After I've

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Paul Mackerras
Rik van Riel writes: I guess we'll need to call tlb_remove_tlb_entry() inside the MADV_FREE code to keep powerpc happy. I don't see why; once ptep_test_and_clear_young has returned, the entry in the hash table has already been removed. Adding the tlb_remove_tlb_entry call certainly won't do

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-23 Thread Rik van Riel
Paul Mackerras wrote: Rik van Riel writes: I guess we'll need to call tlb_remove_tlb_entry() inside the MADV_FREE code to keep powerpc happy. I don't see why; once ptep_test_and_clear_young has returned, the entry in the hash table has already been removed. OK, so this one won't be

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Nick Piggin
Jakub Jelinek wrote: On Fri, Apr 20, 2007 at 07:52:44PM -0400, Rik van Riel wrote: It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU! We _definately_ want both patches, there's a huge benefit

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Rik van Riel
Nick Piggin wrote: So where is the down_write coming from in this workload, I wonder? Heap management? What syscalls? Trying to answer this question, I straced the mysql threads that showed up in top when running a single threaded sysbench workload. There were no mmap, munmap, brk, mprotect

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Nick Piggin
Rik van Riel wrote: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails. Thanks! (I edited slightly so it doesn't wrap) vanilla new

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Rik van Riel
Rik van Riel wrote: Nick Piggin wrote: Rik van Riel wrote: Nick Piggin wrote: Rik van Riel wrote: Here are the transactions/seconds for each combination: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Rik van Riel
Nick Piggin wrote: Rik van Riel wrote: Nick Piggin wrote: Rik van Riel wrote: Here are the transactions/seconds for each combination: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Nick Piggin
Rik van Riel wrote: Nick Piggin wrote: Rik van Riel wrote: Here are the transactions/seconds for each combination: vanilla new glibc madv_free kernel madv_free + mmap_sem threads 1 610 609 596545 2103211361196

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Ulrich Drepper
On 4/22/07, Christoph Hellwig <[EMAIL PROTECTED]> wrote: Why isn't MADV_FREE defined to 5 for linux? It's our first free madv value? Also the behaviour should better match the one in solaris or BSD, the last thing we need is slightly different behaviour from operating systems supporting this

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Christoph Hellwig
On Sun, Apr 22, 2007 at 01:18:10AM -0700, Andrew Morton wrote: > On Tue, 17 Apr 2007 03:15:51 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: > > > Make it possible for applications to have the kernel free memory > > lazily. This reduces a repeated free/malloc cycle from freeing > > pages and

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Andrew Morton
On Tue, 17 Apr 2007 03:15:51 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: > Make it possible for applications to have the kernel free memory > lazily. This reduces a repeated free/malloc cycle from freeing > pages and allocating them, to just marking them freeable. If the > application wants

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Rik van Riel
Nick Piggin wrote: Rik van Riel wrote: Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: Andrew Morton wrote: I've also merged Nick's "mm: madvise avoid exclusive mmap_sem". - Nick's patch also will help this problem. It could be that your

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Rik van Riel
Nick Piggin wrote: Rik van Riel wrote: Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Andrew Morton wrote: I've also merged Nick's mm: madvise avoid exclusive mmap_sem. - Nick's patch also will help this problem. It could be that your

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Andrew Morton
On Tue, 17 Apr 2007 03:15:51 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Make it possible for applications to have the kernel free memory lazily. This reduces a repeated free/malloc cycle from freeing pages and allocating them, to just marking them freeable. If the application wants to

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Christoph Hellwig
On Sun, Apr 22, 2007 at 01:18:10AM -0700, Andrew Morton wrote: On Tue, 17 Apr 2007 03:15:51 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Make it possible for applications to have the kernel free memory lazily. This reduces a repeated free/malloc cycle from freeing pages and allocating

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Ulrich Drepper
On 4/22/07, Christoph Hellwig [EMAIL PROTECTED] wrote: Why isn't MADV_FREE defined to 5 for linux? It's our first free madv value? Also the behaviour should better match the one in solaris or BSD, the last thing we need is slightly different behaviour from operating systems supporting this for

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Nick Piggin
Rik van Riel wrote: Nick Piggin wrote: Rik van Riel wrote: Here are the transactions/seconds for each combination: vanilla new glibc madv_free kernel madv_free + mmap_sem threads 1 610 609 596545 2103211361196

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Rik van Riel
Nick Piggin wrote: Rik van Riel wrote: Nick Piggin wrote: Rik van Riel wrote: Here are the transactions/seconds for each combination: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Rik van Riel
Rik van Riel wrote: Nick Piggin wrote: Rik van Riel wrote: Nick Piggin wrote: Rik van Riel wrote: Here are the transactions/seconds for each combination: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Nick Piggin
Rik van Riel wrote: I've added a 5th column, with just your mmap_sem patch and without my madv_free patch. It is run with the glibc patch, which should make it fall back to MADV_DONTNEED after the first MADV_FREE call fails. Thanks! (I edited slightly so it doesn't wrap) vanilla new

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Rik van Riel
Nick Piggin wrote: So where is the down_write coming from in this workload, I wonder? Heap management? What syscalls? Trying to answer this question, I straced the mysql threads that showed up in top when running a single threaded sysbench workload. There were no mmap, munmap, brk, mprotect

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-22 Thread Nick Piggin
Jakub Jelinek wrote: On Fri, Apr 20, 2007 at 07:52:44PM -0400, Rik van Riel wrote: It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU! We _definately_ want both patches, there's a huge benefit

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Nick Piggin
Nick Piggin wrote: Rik van Riel wrote: Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: Andrew Morton wrote: I've also merged Nick's "mm: madvise avoid exclusive mmap_sem". - Nick's patch also will help this problem. It could be that your

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Nick Piggin
Rik van Riel wrote: Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: Andrew Morton wrote: I've also merged Nick's "mm: madvise avoid exclusive mmap_sem". - Nick's patch also will help this problem. It could be that your patch no longer

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Rik van Riel
Hugh Dickins wrote: On Fri, 20 Apr 2007, Rik van Riel wrote: Andrew Morton wrote: I do go on about that. But we're adding page flags at about one per year, and when we run out we're screwed - we'll need to grow the pageframe. If you want, I can take a look at folding this into the

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-21 Thread Ulrich Drepper
On 4/21/07, Hugh Dickins <[EMAIL PROTECTED]> wrote: But the Linux MADV_DONTNEED does throw away data from a PROT_WRITE,MAP_PRIVATE mapping (or brk or stack) - those changes are discarded, and a subsequent access will revert to zeroes or the underlying mapped file. Been like that since before

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-21 Thread Hugh Dickins
On Fri, 20 Apr 2007, Ulrich Drepper wrote: > > Just for reference: the MADV_CURRENT behavior is to throw away data in > the range. Not exactly. The Linux MADV_DONTNEED never throws away data from a PROT_WRITE,MAP_SHARED mapping (or shm) - it propagates the dirty bit, the page will eventually

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Hugh Dickins
On Fri, 20 Apr 2007, Rik van Riel wrote: > Andrew Morton wrote: > > > I do go on about that. But we're adding page flags at about one per > > year, and when we run out we're screwed - we'll need to grow the > > pageframe. > > If you want, I can take a look at folding this into the >

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Jakub Jelinek
On Fri, Apr 20, 2007 at 07:52:44PM -0400, Rik van Riel wrote: > It turns out that Nick's patch does not improve peak > performance much, but it does prevent the decline when > running with 16 threads on my quad core CPU! > > We _definately_ want both patches, there's a huge benefit > in having

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Jakub Jelinek
On Fri, Apr 20, 2007 at 07:52:44PM -0400, Rik van Riel wrote: It turns out that Nick's patch does not improve peak performance much, but it does prevent the decline when running with 16 threads on my quad core CPU! We _definately_ want both patches, there's a huge benefit in having them

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Hugh Dickins
On Fri, 20 Apr 2007, Rik van Riel wrote: Andrew Morton wrote: I do go on about that. But we're adding page flags at about one per year, and when we run out we're screwed - we'll need to grow the pageframe. If you want, I can take a look at folding this into the -mapping

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-21 Thread Hugh Dickins
On Fri, 20 Apr 2007, Ulrich Drepper wrote: Just for reference: the MADV_CURRENT behavior is to throw away data in the range. Not exactly. The Linux MADV_DONTNEED never throws away data from a PROT_WRITE,MAP_SHARED mapping (or shm) - it propagates the dirty bit, the page will eventually get

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-21 Thread Ulrich Drepper
On 4/21/07, Hugh Dickins [EMAIL PROTECTED] wrote: But the Linux MADV_DONTNEED does throw away data from a PROT_WRITE,MAP_PRIVATE mapping (or brk or stack) - those changes are discarded, and a subsequent access will revert to zeroes or the underlying mapped file. Been like that since before

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Rik van Riel
Hugh Dickins wrote: On Fri, 20 Apr 2007, Rik van Riel wrote: Andrew Morton wrote: I do go on about that. But we're adding page flags at about one per year, and when we run out we're screwed - we'll need to grow the pageframe. If you want, I can take a look at folding this into the

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Nick Piggin
Rik van Riel wrote: Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Andrew Morton wrote: I've also merged Nick's mm: madvise avoid exclusive mmap_sem. - Nick's patch also will help this problem. It could be that your patch no longer

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-21 Thread Nick Piggin
Nick Piggin wrote: Rik van Riel wrote: Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Andrew Morton wrote: I've also merged Nick's mm: madvise avoid exclusive mmap_sem. - Nick's patch also will help this problem. It could be that your

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Rik van Riel
Eric Dumazet wrote: Rik van Riel a écrit : Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: Andrew Morton wrote: I've also merged Nick's "mm: madvise avoid exclusive mmap_sem". - Nick's patch also will help this problem. It could be that

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Eric Dumazet
Rik van Riel a écrit : Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: Andrew Morton wrote: I've also merged Nick's "mm: madvise avoid exclusive mmap_sem". - Nick's patch also will help this problem. It could be that your patch no longer

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Rik van Riel
Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: Andrew Morton wrote: I've also merged Nick's "mm: madvise avoid exclusive mmap_sem". - Nick's patch also will help this problem. It could be that your patch no longer offers a 2x speedup when

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Andrew Morton
On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: > Andrew Morton wrote: > > > I've also merged Nick's "mm: madvise avoid exclusive mmap_sem". > > > > - Nick's patch also will help this problem. It could be that your patch > > no longer offers a 2x speedup when

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Rik van Riel
Andrew Morton wrote: I've also merged Nick's "mm: madvise avoid exclusive mmap_sem". - Nick's patch also will help this problem. It could be that your patch no longer offers a 2x speedup when combined with Nick's patch. It could well be that the combination of the two is even better, but

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-20 Thread Ulrich Drepper
On 4/20/07, Andrew Morton <[EMAIL PROTECTED]> wrote: OK, we need to flesh this out a lot please. People often get confused about what our MADV_DONTNEED behaviour is. Well, there's not really much to flesh out. The current MADV_DONTNEED is useful in some situations. The behavior cannot be

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-20 Thread Andrew Morton
On Thu, 19 Apr 2007 17:15:28 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: > Restore MADV_DONTNEED to its original Linux behaviour. This is still > not the same behaviour as POSIX, but applications may be depending on > the Linux behaviour already. Besides, glibc catches POSIX_MADV_DONTNEED >

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Andrew Morton
On Tue, 17 Apr 2007 03:15:51 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: > Make it possible for applications to have the kernel free memory > lazily. This reduces a repeated free/malloc cycle from freeing > pages and allocating them, to just marking them freeable. If the > application wants

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Andrew Morton
On Tue, 17 Apr 2007 03:15:51 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Make it possible for applications to have the kernel free memory lazily. This reduces a repeated free/malloc cycle from freeing pages and allocating them, to just marking them freeable. If the application wants to

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-20 Thread Andrew Morton
On Thu, 19 Apr 2007 17:15:28 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Restore MADV_DONTNEED to its original Linux behaviour. This is still not the same behaviour as POSIX, but applications may be depending on the Linux behaviour already. Besides, glibc catches POSIX_MADV_DONTNEED and

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-20 Thread Ulrich Drepper
On 4/20/07, Andrew Morton [EMAIL PROTECTED] wrote: OK, we need to flesh this out a lot please. People often get confused about what our MADV_DONTNEED behaviour is. Well, there's not really much to flesh out. The current MADV_DONTNEED is useful in some situations. The behavior cannot be

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Rik van Riel
Andrew Morton wrote: I've also merged Nick's mm: madvise avoid exclusive mmap_sem. - Nick's patch also will help this problem. It could be that your patch no longer offers a 2x speedup when combined with Nick's patch. It could well be that the combination of the two is even better, but

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Andrew Morton
On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Andrew Morton wrote: I've also merged Nick's mm: madvise avoid exclusive mmap_sem. - Nick's patch also will help this problem. It could be that your patch no longer offers a 2x speedup when combined with

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Rik van Riel
Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Andrew Morton wrote: I've also merged Nick's mm: madvise avoid exclusive mmap_sem. - Nick's patch also will help this problem. It could be that your patch no longer offers a 2x speedup when

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Eric Dumazet
Rik van Riel a écrit : Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Andrew Morton wrote: I've also merged Nick's mm: madvise avoid exclusive mmap_sem. - Nick's patch also will help this problem. It could be that your patch no longer

Re: [PATCH] lazy freeing of memory through MADV_FREE

2007-04-20 Thread Rik van Riel
Eric Dumazet wrote: Rik van Riel a écrit : Andrew Morton wrote: On Fri, 20 Apr 2007 17:38:06 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Andrew Morton wrote: I've also merged Nick's mm: madvise avoid exclusive mmap_sem. - Nick's patch also will help this problem. It could be that your

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-19 Thread Rik van Riel
Restore MADV_DONTNEED to its original Linux behaviour. This is still not the same behaviour as POSIX, but applications may be depending on the Linux behaviour already. Besides, glibc catches POSIX_MADV_DONTNEED and makes sure nothing is done... Signed-off-by: Rik van Riel <[EMAIL PROTECTED]>

Re: [PATCH] lazy freeing of memory through MADV_FREE 2/2

2007-04-19 Thread Rik van Riel
Restore MADV_DONTNEED to its original Linux behaviour. This is still not the same behaviour as POSIX, but applications may be depending on the Linux behaviour already. Besides, glibc catches POSIX_MADV_DONTNEED and makes sure nothing is done... Signed-off-by: Rik van Riel [EMAIL PROTECTED] ---

[PATCH] lazy freeing of memory through MADV_FREE

2007-04-17 Thread Rik van Riel
Make it possible for applications to have the kernel free memory lazily. This reduces a repeated free/malloc cycle from freeing pages and allocating them, to just marking them freeable. If the application wants to reuse them before the kernel needs the memory, not even a page fault will happen.

[PATCH] lazy freeing of memory through MADV_FREE

2007-04-17 Thread Rik van Riel
Make it possible for applications to have the kernel free memory lazily. This reduces a repeated free/malloc cycle from freeing pages and allocating them, to just marking them freeable. If the application wants to reuse them before the kernel needs the memory, not even a page fault will happen.