Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-08-06 Thread Paolo Ciarrocchi
On 8/6/07, Nick Piggin <[EMAIL PROTECTED]> wrote: [...] > > this completely ignores the use case where the > > swapping was exactly the > > right thing to do, but memory has been freed up from > > a program exiting so > > that you couldnow fill that empty ram with data that > > was swapped out. >

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-08-06 Thread Nick Piggin
--- [EMAIL PROTECTED] wrote: > On Mon, 6 Aug 2007, Nick Piggin wrote: > > > [EMAIL PROTECTED] wrote: > >> On Sun, 29 Jul 2007, Rene Herman wrote: > >> > >> > On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: > >> > > >> > > I agree that tinkering with the core VM code > should not be done

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-08-06 Thread Nick Piggin
--- [EMAIL PROTECTED] wrote: On Mon, 6 Aug 2007, Nick Piggin wrote: [EMAIL PROTECTED] wrote: On Sun, 29 Jul 2007, Rene Herman wrote: On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: I agree that tinkering with the core VM code should not be done lightly, but

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-08-06 Thread Paolo Ciarrocchi
On 8/6/07, Nick Piggin [EMAIL PROTECTED] wrote: [...] this completely ignores the use case where the swapping was exactly the right thing to do, but memory has been freed up from a program exiting so that you couldnow fill that empty ram with data that was swapped out. Yeah. However,

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-08-05 Thread david
On Mon, 6 Aug 2007, Nick Piggin wrote: [EMAIL PROTECTED] wrote: On Sun, 29 Jul 2007, Rene Herman wrote: > On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: > > > I agree that tinkering with the core VM code should not be done > > lightly, > > but this has been put through the proper

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-08-05 Thread Nick Piggin
[EMAIL PROTECTED] wrote: On Sun, 29 Jul 2007, Rene Herman wrote: On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: I agree that tinkering with the core VM code should not be done lightly, but this has been put through the proper process and is stalled with no hints on how to move forward.

Re: [ck] Re: -mm merge plans for 2.6.23

2007-08-05 Thread Nick Piggin
Matthew Hawkins wrote: On 7/25/07, Nick Piggin <[EMAIL PROTECTED]> wrote: I guess /proc/meminfo, /proc/zoneinfo, /proc/vmstat, /proc/slabinfo before and after the updatedb run with the latest kernel would be a first step. top and vmstat output during the run wouldn't hurt either. Hi Nick,

Re: [ck] Re: -mm merge plans for 2.6.23

2007-08-05 Thread Nick Piggin
Matthew Hawkins wrote: On 7/25/07, Nick Piggin [EMAIL PROTECTED] wrote: I guess /proc/meminfo, /proc/zoneinfo, /proc/vmstat, /proc/slabinfo before and after the updatedb run with the latest kernel would be a first step. top and vmstat output during the run wouldn't hurt either. Hi Nick,

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-08-05 Thread Nick Piggin
[EMAIL PROTECTED] wrote: On Sun, 29 Jul 2007, Rene Herman wrote: On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: I agree that tinkering with the core VM code should not be done lightly, but this has been put through the proper process and is stalled with no hints on how to move forward.

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-08-05 Thread david
On Mon, 6 Aug 2007, Nick Piggin wrote: [EMAIL PROTECTED] wrote: On Sun, 29 Jul 2007, Rene Herman wrote: On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: I agree that tinkering with the core VM code should not be done lightly, but this has been put through the proper process

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-08-04 Thread Pavel Machek
Hi! > > That would just save reading the directories. Not sure > > it helps that much. Much better would be actually if it didn't stat the > > individual files (and force their dentries/inodes in). I bet it does that > > to > > find out if they are directories or not. But in a modern system it

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-08-04 Thread Pavel Machek
Hi! That would just save reading the directories. Not sure it helps that much. Much better would be actually if it didn't stat the individual files (and force their dentries/inodes in). I bet it does that to find out if they are directories or not. But in a modern system it could

Re: [ck] Re: -mm merge plans for 2.6.23

2007-07-31 Thread Matthew Hawkins
On 7/25/07, Nick Piggin <[EMAIL PROTECTED]> wrote: > I guess /proc/meminfo, /proc/zoneinfo, /proc/vmstat, /proc/slabinfo > before and after the updatedb run with the latest kernel would be a > first step. top and vmstat output during the run wouldn't hurt either. Hi Nick, I've attached two files

Re: [ck] Re: -mm merge plans for 2.6.23

2007-07-31 Thread Matthew Hawkins
On 7/25/07, Nick Piggin [EMAIL PROTECTED] wrote: I guess /proc/meminfo, /proc/zoneinfo, /proc/vmstat, /proc/slabinfo before and after the updatedb run with the latest kernel would be a first step. top and vmstat output during the run wouldn't hurt either. Hi Nick, I've attached two files with

Re: [ck] Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-30 Thread Helge Hafting
Matthew Hawkins wrote: updatedb by itself doesn't really bug me, its just that on occasion its still running at 7am You should start it earlier then - assuming it doesn't already start at the earliest opportunity? Helge Hafting - To unsubscribe from this list: send the line "unsubscribe

Re: [ck] Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-30 Thread Helge Hafting
Matthew Hawkins wrote: updatedb by itself doesn't really bug me, its just that on occasion its still running at 7am You should start it earlier then - assuming it doesn't already start at the earliest opportunity? Helge Hafting - To unsubscribe from this list: send the line unsubscribe

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread david
On Sun, 29 Jul 2007, Rene Herman wrote: On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: I agree that tinkering with the core VM code should not be done lightly, but this has been put through the proper process and is stalled with no hints on how to move forward. It has not. Concerns

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Daniel Hazelton
On Sunday 29 July 2007 16:00:22 Ray Lee wrote: > On 7/29/07, Paul Jackson <[EMAIL PROTECTED]> wrote: > > If the problem is reading stuff back in from swap at the *same time* > > that the application is reading stuff from some user file system, and if > > that user file system is on the same drive

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Paul Jackson <[EMAIL PROTECTED]> wrote: > Ray wrote: > > Ah, so in a normal scenario where a working-set is getting faulted > > back in, we have the swap storage as well as the file-backed stuff > > that needs to be read as well. So even if swap is organized perfectly, > > we're still

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Paul Jackson
Ray wrote: > Ah, so in a normal scenario where a working-set is getting faulted > back in, we have the swap storage as well as the file-backed stuff > that needs to be read as well. So even if swap is organized perfectly, > we're still seeking. Damn. Perhaps this applies in some cases ...

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Paul Jackson <[EMAIL PROTECTED]> wrote: > If the problem is reading stuff back in from swap at the *same time* > that the application is reading stuff from some user file system, and if > that user file system is on the same drive as the swap partition > (typical on laptops), then

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Paul Jackson
Ray wrote: > a log structured scheme, where the writeout happens to sequential spaces > on the drive instead of scattered about. If the problem is reading stuff back in from swap quickly when needed, then this likely helps, by reducing the seeks needed. If the problem is reading stuff back in

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 07:52 PM, Ray Lee wrote: Well, that doesn't match my systems. My laptop has 400MB in swap: Which in your case is slightly more than 1/3 of available swap space. Quite a lot for a desktop indeed. And if it's more than a few percent fragmented, please fix current swapout

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman <[EMAIL PROTECTED]> wrote: > On 07/29/2007 07:19 PM, Ray Lee wrote: > For me, it is generally the case yes. We are still discussing this in the > context of desktop machines and their problems with being slow as things > have been swapped out and generally I expect a

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Alan Cox
> > Is that generally the case on your systems? Every linux system I've > > run, regardless of RAM, has always pushed things out to swap. > > For me, it is generally the case yes. We are still discussing this in the > context of desktop machines and their problems with being slow as things >

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 07:19 PM, Ray Lee wrote: The program is not a real-world issue and if you do not consider it a useful boundary condition either (okay I guess), how would log structured swap help if I just assume I have plenty of free swap to begin with? Is that generally the case on your

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman <[EMAIL PROTECTED]> wrote: > On 07/29/2007 06:04 PM, Ray Lee wrote: > >> I am very aware of the costs of seeks (on current magnetic media). > > > > Then perhaps you can just take it on faith -- log structured layouts > > are designed to help minimize seeks, read and write.

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 06:04 PM, Ray Lee wrote: I am very aware of the costs of seeks (on current magnetic media). Then perhaps you can just take it on faith -- log structured layouts are designed to help minimize seeks, read and write. I am particularly bad at faith. Let's take that stupid program

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman <[EMAIL PROTECTED]> wrote: > On 07/29/2007 05:20 PM, Ray Lee wrote: > This seems to be now fixing the different problem of swap-space filling up. > I'm quite willing to for now assume I've got plenty free. I was trying to point out that currently, as an example, memory

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 05:20 PM, Ray Lee wrote: I understand what log structure is generally, but how does it help swapin? Look at the swap out case first. Right now, when swapping out the kernel places whatever it can wherever it can inside the swap space. The closer you are to filling your swap

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman <[EMAIL PROTECTED]> wrote: > On 07/29/2007 04:58 PM, Ray Lee wrote: > > On 7/29/07, Rene Herman <[EMAIL PROTECTED]> wrote: > >> Right over my head. Why does log-structure help anything? > > > > Log structured disk layouts allow for better placement of writeout, so > > that

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 04:58 PM, Ray Lee wrote: On 7/29/07, Rene Herman <[EMAIL PROTECTED]> wrote: On 07/29/2007 03:12 PM, Alan Cox wrote: More radically if anyone wants to do real researchy type work - how about log structured swap with a cleaner ? Right over my head. Why does log-structure

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman <[EMAIL PROTECTED]> wrote: > On 07/29/2007 03:12 PM, Alan Cox wrote: > > More radically if anyone wants to do real researchy type work - how about > > log structured swap with a cleaner ? > > Right over my head. Why does log-structure help anything? Log structured disk

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 03:12 PM, Alan Cox wrote: What are the tradeoffs here? What wants small chunks? Also, as far as I'm aware Linux does not do things like up the granularity when it notices it's swapping in heavily? That sounds sort of promising... Small chunks means you get better efficiency of

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: And now you do it again :-) There is no conclusion -- just the inescapable observation that swap-prefetch was (or may have been) masking the problem of GNU locate being a program that noone in their right mind should be using. isn't your

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Alan Cox
> Contrived thing and all, but what it does do is show exactly how bad seeking > all over swap-space is. If you push it out before hitting enter, the time it > takes easily grows past 10 minutes (with my 768M) versus sub-second (!) when > it's all in to start with. Think in "operations/second"

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Paul Jackson
Andi wrote: > GNU sort uses a merge sort with temporary files on disk. Not sure > how much it keeps in memory during that, but it's probably less > than 150MB. If I'm reading the source code for GNU sort correctly, then the following snippet of shell code displays how much memory it uses for its

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread david
On Sun, 29 Jul 2007, Rene Herman wrote: On 07/28/2007 11:00 PM, [EMAIL PROTECTED] wrote: > many -mm users use it anyway? He himself said he's not convinced of > usefulness having not seen it help for him (and notice that most > developers are also users), turned it off due to it annoying

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/28/2007 11:00 PM, [EMAIL PROTECTED] wrote: many -mm users use it anyway? He himself said he's not convinced of usefulness having not seen it help for him (and notice that most developers are also users), turned it off due to it annoying him at some point and hasn't seen a serious

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/28/2007 01:21 PM, Alan Cox wrote: It is. Prefetched pages can be dropped on the floor without additional I/O. Which is essentially free for most cases. In addition your disk access may well have been in idle time (and should be for this sort of stuff) Yes. The swap-prefetch patch

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/28/2007 01:21 PM, Alan Cox wrote: It is. Prefetched pages can be dropped on the floor without additional I/O. Which is essentially free for most cases. In addition your disk access may well have been in idle time (and should be for this sort of stuff) Yes. The swap-prefetch patch

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/28/2007 11:00 PM, [EMAIL PROTECTED] wrote: many -mm users use it anyway? He himself said he's not convinced of usefulness having not seen it help for him (and notice that most developers are also users), turned it off due to it annoying him at some point and hasn't seen a serious

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread david
On Sun, 29 Jul 2007, Rene Herman wrote: On 07/28/2007 11:00 PM, [EMAIL PROTECTED] wrote: many -mm users use it anyway? He himself said he's not convinced of usefulness having not seen it help for him (and notice that most developers are also users), turned it off due to it annoying him

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Paul Jackson
Andi wrote: GNU sort uses a merge sort with temporary files on disk. Not sure how much it keeps in memory during that, but it's probably less than 150MB. If I'm reading the source code for GNU sort correctly, then the following snippet of shell code displays how much memory it uses for its

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Alan Cox
Contrived thing and all, but what it does do is show exactly how bad seeking all over swap-space is. If you push it out before hitting enter, the time it takes easily grows past 10 minutes (with my 768M) versus sub-second (!) when it's all in to start with. Think in operations/second and

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: And now you do it again :-) There is no conclusion -- just the inescapable observation that swap-prefetch was (or may have been) masking the problem of GNU locate being a program that noone in their right mind should be using. isn't your

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 03:12 PM, Alan Cox wrote: What are the tradeoffs here? What wants small chunks? Also, as far as I'm aware Linux does not do things like up the granularity when it notices it's swapping in heavily? That sounds sort of promising... Small chunks means you get better efficiency of

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman [EMAIL PROTECTED] wrote: On 07/29/2007 03:12 PM, Alan Cox wrote: More radically if anyone wants to do real researchy type work - how about log structured swap with a cleaner ? Right over my head. Why does log-structure help anything? Log structured disk layouts

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman [EMAIL PROTECTED] wrote: On 07/29/2007 04:58 PM, Ray Lee wrote: On 7/29/07, Rene Herman [EMAIL PROTECTED] wrote: Right over my head. Why does log-structure help anything? Log structured disk layouts allow for better placement of writeout, so that you cn eliminate

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 04:58 PM, Ray Lee wrote: On 7/29/07, Rene Herman [EMAIL PROTECTED] wrote: On 07/29/2007 03:12 PM, Alan Cox wrote: More radically if anyone wants to do real researchy type work - how about log structured swap with a cleaner ? Right over my head. Why does log-structure help

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 05:20 PM, Ray Lee wrote: I understand what log structure is generally, but how does it help swapin? Look at the swap out case first. Right now, when swapping out the kernel places whatever it can wherever it can inside the swap space. The closer you are to filling your swap

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman [EMAIL PROTECTED] wrote: On 07/29/2007 05:20 PM, Ray Lee wrote: This seems to be now fixing the different problem of swap-space filling up. I'm quite willing to for now assume I've got plenty free. I was trying to point out that currently, as an example, memory that is

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 06:04 PM, Ray Lee wrote: I am very aware of the costs of seeks (on current magnetic media). Then perhaps you can just take it on faith -- log structured layouts are designed to help minimize seeks, read and write. I am particularly bad at faith. Let's take that stupid program

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman [EMAIL PROTECTED] wrote: On 07/29/2007 06:04 PM, Ray Lee wrote: I am very aware of the costs of seeks (on current magnetic media). Then perhaps you can just take it on faith -- log structured layouts are designed to help minimize seeks, read and write. I am

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 07:19 PM, Ray Lee wrote: The program is not a real-world issue and if you do not consider it a useful boundary condition either (okay I guess), how would log structured swap help if I just assume I have plenty of free swap to begin with? Is that generally the case on your

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Alan Cox
Is that generally the case on your systems? Every linux system I've run, regardless of RAM, has always pushed things out to swap. For me, it is generally the case yes. We are still discussing this in the context of desktop machines and their problems with being slow as things have been

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Rene Herman [EMAIL PROTECTED] wrote: On 07/29/2007 07:19 PM, Ray Lee wrote: For me, it is generally the case yes. We are still discussing this in the context of desktop machines and their problems with being slow as things have been swapped out and generally I expect a desktop to

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Rene Herman
On 07/29/2007 07:52 PM, Ray Lee wrote: Shrug Well, that doesn't match my systems. My laptop has 400MB in swap: Which in your case is slightly more than 1/3 of available swap space. Quite a lot for a desktop indeed. And if it's more than a few percent fragmented, please fix current swapout

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Paul Jackson
Ray wrote: a log structured scheme, where the writeout happens to sequential spaces on the drive instead of scattered about. If the problem is reading stuff back in from swap quickly when needed, then this likely helps, by reducing the seeks needed. If the problem is reading stuff back in from

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Paul Jackson [EMAIL PROTECTED] wrote: If the problem is reading stuff back in from swap at the *same time* that the application is reading stuff from some user file system, and if that user file system is on the same drive as the swap partition (typical on laptops), then

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Paul Jackson
Ray wrote: Ah, so in a normal scenario where a working-set is getting faulted back in, we have the swap storage as well as the file-backed stuff that needs to be read as well. So even if swap is organized perfectly, we're still seeking. Damn. Perhaps this applies in some cases ... perhaps.

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Ray Lee
On 7/29/07, Paul Jackson [EMAIL PROTECTED] wrote: Ray wrote: Ah, so in a normal scenario where a working-set is getting faulted back in, we have the swap storage as well as the file-backed stuff that needs to be read as well. So even if swap is organized perfectly, we're still seeking.

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread Daniel Hazelton
On Sunday 29 July 2007 16:00:22 Ray Lee wrote: On 7/29/07, Paul Jackson [EMAIL PROTECTED] wrote: If the problem is reading stuff back in from swap at the *same time* that the application is reading stuff from some user file system, and if that user file system is on the same drive as the

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-29 Thread david
On Sun, 29 Jul 2007, Rene Herman wrote: On 07/29/2007 01:41 PM, [EMAIL PROTECTED] wrote: I agree that tinkering with the core VM code should not be done lightly, but this has been put through the proper process and is stalled with no hints on how to move forward. It has not. Concerns

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Andrew Morton
On Sat, 28 Jul 2007 21:33:59 -0400 Rik van Riel <[EMAIL PROTECTED]> wrote: > Andrew Morton wrote: > > > What I think is killing us here is the blockdev pagecache: the pagecache > > which backs those directory entries and inodes. These pages get read > > multiple times because they hold multiple

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rik van Riel
Andrew Morton wrote: What I think is killing us here is the blockdev pagecache: the pagecache which backs those directory entries and inodes. These pages get read multiple times because they hold multiple directory entries and multiple inodes. These multiple touches will put those pages onto

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Daniel Hazelton
On Saturday 28 July 2007 17:06:50 [EMAIL PROTECTED] wrote: > On Sat, 28 Jul 2007, Daniel Hazelton wrote: > > On Saturday 28 July 2007 04:55:58 [EMAIL PROTECTED] wrote: > >> On Sat, 28 Jul 2007, Rene Herman wrote: > >>> On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: > On Fri, 27 Jul 2007,

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread david
On Sat, 28 Jul 2007, Daniel Hazelton wrote: On Saturday 28 July 2007 04:55:58 [EMAIL PROTECTED] wrote: On Sat, 28 Jul 2007, Rene Herman wrote: On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: On Fri, 27 Jul 2007, Rene Herman wrote: On 07/27/2007 07:45 PM, Daniel Hazelton wrote: nobody

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread david
On Sat, 28 Jul 2007, Alan Cox wrote: It is. Prefetched pages can be dropped on the floor without additional I/O. Which is essentially free for most cases. In addition your disk access may well have been in idle time (and should be for this sort of stuff) and if it was in the same chunk as

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread david
On Sat, 28 Jul 2007, Rene Herman wrote: On 07/28/2007 10:55 AM, [EMAIL PROTECTED] wrote: it looks to me like unless the code was really bad (and after 23 months in -mm it doesn't sound like it is) Not to sound pretentious or anything but I assume that Andrew has a fairly good overview of

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Ray Lee
On 7/28/07, Alan Cox <[EMAIL PROTECTED]> wrote: > Actual physical disk ops are precious resource and anything that mostly > reduces the number will be a win - not to stay swap prefetch is the right > answer but accidentally or otherwise there are good reasons it may happen > to help. > > Bigger

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Daniel Hazelton
On Saturday 28 July 2007 04:55:58 [EMAIL PROTECTED] wrote: > On Sat, 28 Jul 2007, Rene Herman wrote: > > On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: > >> On Fri, 27 Jul 2007, Rene Herman wrote: > >> > On 07/27/2007 07:45 PM, Daniel Hazelton wrote: > >> > > Questions about it: > >> > >

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Daniel Hazelton
On Saturday 28 July 2007 03:48:13 Mike Galbraith wrote: > On Fri, 2007-07-27 at 18:51 -0400, Daniel Hazelton wrote: > > Now, once more, I'm going to ask: What is so terribly wrong with swap > > prefetch? Why does it seem that everyone against it says "Its treating a > > symptom, so it can't go

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Alan Cox
> It is. Prefetched pages can be dropped on the floor without additional I/O. Which is essentially free for most cases. In addition your disk access may well have been in idle time (and should be for this sort of stuff) and if it was in the same chunk as something nearby was effectively free

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rene Herman
On 07/28/2007 10:55 AM, [EMAIL PROTECTED] wrote: in at some situations swap prefetch can help becouse something that used memory freed it so there is free memory that could be filled with data (which is something that Linux does agressivly in most other situations) in some other situations

Re: -mm merge plans for 2.6.23

2007-07-28 Thread Stefan Richter
Daniel Cheng wrote: > but merging maps2 have higher risk which should be done in a development > branch (er... 2.7, but we don't have it now). This is off-topic and has been discussed to death, but: Rather than one stable branch and one development branch, we have a few stable branches and a lot

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread david
On Sat, 28 Jul 2007, Rene Herman wrote: On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: On Fri, 27 Jul 2007, Rene Herman wrote: > On 07/27/2007 07:45 PM, Daniel Hazelton wrote: > > > Questions about it: > > Q) Does swap-prefetch help with this? > > A) [From all reports I've seen

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rene Herman
On 07/28/2007 09:35 AM, Rene Herman wrote: By the way -- I'm unable to make my slocate grow substantial here but I'll try what GNU locate does. If it's really as bad as I hear then regardless of anything else it should really be either fixed or dumped... Yes. GNU locate is broken and nobody

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Mike Galbraith
On Fri, 2007-07-27 at 18:51 -0400, Daniel Hazelton wrote: > Now, once more, I'm going to ask: What is so terribly wrong with swap > prefetch? Why does it seem that everyone against it says "Its treating a > symptom, so it can't go in"? And once again, I personally have nothing against

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rene Herman
On 07/28/2007 01:15 AM, Björn Steinbrink wrote: On 2007.07.27 20:16:32 +0200, Rene Herman wrote: Here's swap-prefetch's author saying the same: http://lkml.org/lkml/2007/2/9/112 | It can't help the updatedb scenario. Updatedb leaves the ram full and | swap prefetch wants to cost as little

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rene Herman
On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: On Fri, 27 Jul 2007, Rene Herman wrote: On 07/27/2007 07:45 PM, Daniel Hazelton wrote: Questions about it: Q) Does swap-prefetch help with this? A) [From all reports I've seen (*)] Yes, it does. No it does not. If updatedb filled

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rene Herman
On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: On Fri, 27 Jul 2007, Rene Herman wrote: On 07/27/2007 07:45 PM, Daniel Hazelton wrote: Questions about it: Q) Does swap-prefetch help with this? A) [From all reports I've seen (*)] Yes, it does. No it does not. If updatedb filled

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rene Herman
On 07/28/2007 01:15 AM, Björn Steinbrink wrote: On 2007.07.27 20:16:32 +0200, Rene Herman wrote: Here's swap-prefetch's author saying the same: http://lkml.org/lkml/2007/2/9/112 | It can't help the updatedb scenario. Updatedb leaves the ram full and | swap prefetch wants to cost as little

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Mike Galbraith
On Fri, 2007-07-27 at 18:51 -0400, Daniel Hazelton wrote: Now, once more, I'm going to ask: What is so terribly wrong with swap prefetch? Why does it seem that everyone against it says Its treating a symptom, so it can't go in? And once again, I personally have nothing against

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rene Herman
On 07/28/2007 09:35 AM, Rene Herman wrote: By the way -- I'm unable to make my slocate grow substantial here but I'll try what GNU locate does. If it's really as bad as I hear then regardless of anything else it should really be either fixed or dumped... Yes. GNU locate is broken and nobody

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread david
On Sat, 28 Jul 2007, Rene Herman wrote: On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: On Fri, 27 Jul 2007, Rene Herman wrote: On 07/27/2007 07:45 PM, Daniel Hazelton wrote: Questions about it: Q) Does swap-prefetch help with this? A) [From all reports I've seen (*)]

Re: -mm merge plans for 2.6.23

2007-07-28 Thread Stefan Richter
Daniel Cheng wrote: but merging maps2 have higher risk which should be done in a development branch (er... 2.7, but we don't have it now). This is off-topic and has been discussed to death, but: Rather than one stable branch and one development branch, we have a few stable branches and a lot

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rene Herman
On 07/28/2007 10:55 AM, [EMAIL PROTECTED] wrote: in at some situations swap prefetch can help becouse something that used memory freed it so there is free memory that could be filled with data (which is something that Linux does agressivly in most other situations) in some other situations

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Alan Cox
It is. Prefetched pages can be dropped on the floor without additional I/O. Which is essentially free for most cases. In addition your disk access may well have been in idle time (and should be for this sort of stuff) and if it was in the same chunk as something nearby was effectively free

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Daniel Hazelton
On Saturday 28 July 2007 03:48:13 Mike Galbraith wrote: On Fri, 2007-07-27 at 18:51 -0400, Daniel Hazelton wrote: Now, once more, I'm going to ask: What is so terribly wrong with swap prefetch? Why does it seem that everyone against it says Its treating a symptom, so it can't go in? And

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Daniel Hazelton
On Saturday 28 July 2007 04:55:58 [EMAIL PROTECTED] wrote: On Sat, 28 Jul 2007, Rene Herman wrote: On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: On Fri, 27 Jul 2007, Rene Herman wrote: On 07/27/2007 07:45 PM, Daniel Hazelton wrote: Questions about it: Q) Does swap-prefetch

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Ray Lee
On 7/28/07, Alan Cox [EMAIL PROTECTED] wrote: Actual physical disk ops are precious resource and anything that mostly reduces the number will be a win - not to stay swap prefetch is the right answer but accidentally or otherwise there are good reasons it may happen to help. Bigger more

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread david
On Sat, 28 Jul 2007, Rene Herman wrote: On 07/28/2007 10:55 AM, [EMAIL PROTECTED] wrote: it looks to me like unless the code was really bad (and after 23 months in -mm it doesn't sound like it is) Not to sound pretentious or anything but I assume that Andrew has a fairly good overview of

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread david
On Sat, 28 Jul 2007, Alan Cox wrote: It is. Prefetched pages can be dropped on the floor without additional I/O. Which is essentially free for most cases. In addition your disk access may well have been in idle time (and should be for this sort of stuff) and if it was in the same chunk as

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread david
On Sat, 28 Jul 2007, Daniel Hazelton wrote: On Saturday 28 July 2007 04:55:58 [EMAIL PROTECTED] wrote: On Sat, 28 Jul 2007, Rene Herman wrote: On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: On Fri, 27 Jul 2007, Rene Herman wrote: On 07/27/2007 07:45 PM, Daniel Hazelton wrote: nobody

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Daniel Hazelton
On Saturday 28 July 2007 17:06:50 [EMAIL PROTECTED] wrote: On Sat, 28 Jul 2007, Daniel Hazelton wrote: On Saturday 28 July 2007 04:55:58 [EMAIL PROTECTED] wrote: On Sat, 28 Jul 2007, Rene Herman wrote: On 07/27/2007 09:43 PM, [EMAIL PROTECTED] wrote: On Fri, 27 Jul 2007, Rene Herman

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Rik van Riel
Andrew Morton wrote: What I think is killing us here is the blockdev pagecache: the pagecache which backs those directory entries and inodes. These pages get read multiple times because they hold multiple directory entries and multiple inodes. These multiple touches will put those pages onto

Re: RFT: updatedb morning after problem [was: Re: -mm merge plans for 2.6.23]

2007-07-28 Thread Andrew Morton
On Sat, 28 Jul 2007 21:33:59 -0400 Rik van Riel [EMAIL PROTECTED] wrote: Andrew Morton wrote: What I think is killing us here is the blockdev pagecache: the pagecache which backs those directory entries and inodes. These pages get read multiple times because they hold multiple directory

Re: RFT: updatedb "morning after" problem [was: Re: -mm merge plans for 2.6.23]

2007-07-27 Thread Rene Herman
On 07/27/2007 10:28 PM, Daniel Hazelton wrote: Check the attitude at the door then re-read what I actually said: Attitude? You wanted attitude dear boy? Updatedb or another process that uses the FS heavily runs on a users 256MB P3-800 (when it is idle) and the VFS caches grow, causing

Re: -mm merge plans for 2.6.23

2007-07-27 Thread Daniel Cheng
Andrew Morton wrote: [...] > > And userspace can do a much better implementation of this > how-to-handle-large-load-shifts problem, because it is really quite > complex. The system needs to be monitored to determine what is the "usual" [...] > All this would end up needing runtime

  1   2   3   4   5   6   7   8   >