> On Thu, 08 Mar 2007 13:29:02 +0530 Vaidyanathan Srinivasan <[EMAIL
> PROTECTED]> wrote:
> > That all sounds reasonably doable. It'd be pretty complex to do it
> > in-kernel but we could do it there too. Problem is if course that the
> > above strategy is explicitly optimised for the backup pro
Andrew Morton wrote:
> On Sun, 4 Mar 2007 00:01:55 +0100 bert hubert <[EMAIL PROTECTED]> wrote:
>
>> On Sat, Mar 03, 2007 at 02:26:09PM -0800, Andrew Morton wrote:
> It is *not* a global instruction. It uses setenv, so the user's policy
> affects only the target process and its forked c
On Wed, 07 Mar 2007 11:39:02 + Pádraig Brady <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
> > On Tue, 06 Mar 2007 12:10:49 +
> > P__draig Brady <[EMAIL PROTECTED]> wrote:
> >> Perhaps one could possibly just evict pages with _mapcount==0 ?
> >
> > That is the present fadvise(FADV_DON
Andrew Morton wrote:
> On Tue, 06 Mar 2007 12:10:49 +
> P__draig Brady <[EMAIL PROTECTED]> wrote:
>> Perhaps one could possibly just evict pages with _mapcount==0 ?
>
> That is the present fadvise(FADV_DONTNEED) behaviour.
Ah right. It doesn't invalidate page_mapped() pages.
If that means it
Andrew Morton wrote:
On Tue, 06 Mar 2007 12:10:49 +
P__draig Brady <[EMAIL PROTECTED]> wrote:
Andrew Morton wrote:
If I'm the target
audience for that API then it's broken as I'd mess it up,
or would take too long to get it right.
Can't we just fix the posix_fadvise() implementation to
o
On Tue, 06 Mar 2007 12:10:49 +
P__draig Brady <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
> > Yes. Let's flesh it out the backup program policy some more:
> >
> > - Unconditionally invalidate output files
> >
> > - on entry to read(), probe pagecache, record which pages in the range a
Andrew Morton wrote:
> Yes. Let's flesh it out the backup program policy some more:
>
> - Unconditionally invalidate output files
>
> - on entry to read(), probe pagecache, record which pages in the range are
> present
>
> - on entry to next read(), shoot down those pages from the previous rea
On Mon, 05 Mar 2007 11:02:43 + Pádraig Brady <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
> > I've uploaded to http://userweb.kernel.org/~akpm/pagecache-management/ a
> > little tool which permits the management of the pagecache usage of
> > arbitrary applications. Effectively it prevent
Andrew Morton wrote:
> I've uploaded to http://userweb.kernel.org/~akpm/pagecache-management/ a
> little tool which permits the management of the pagecache usage of
> arbitrary applications. Effectively it prevents the targetted application
> from using any pagecache at all.
Cool, Kinda like noca
Andrew Morton wrote:
On Sat, 03 Mar 2007 20:56:27 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
Andrew Morton wrote:
Doing a refault thing would help a bit, but stops working at a certain point.
At what point does it stop working?
We need to store that this-page-got-reclaimed info somewhere
On Sun, 2007-03-04 at 04:07 -0800, Andrew Morton wrote:
> On Sat, 03 Mar 2007 20:56:27 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
>
> > Andrew Morton wrote:
> >
> > >>> Doing a refault thing would help a bit, but stops working at a certain
> > >>> point.
> > >> At what point does it stop work
On Sat, 03 Mar 2007 20:56:27 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
>
> >>> Doing a refault thing would help a bit, but stops working at a certain
> >>> point.
> >> At what point does it stop working?
> >
> > We need to store that this-page-got-reclaimed info somew
On Sat, 3 Mar 2007 21:35:59 -0500 "Lee Revell" <[EMAIL PROTECTED]> wrote:
> On 3/3/07, Andrew Morton <[EMAIL PROTECTED]> wrote:
> > But yes, updatedb's pagecache usage will be mainly metadata, and this tool
> > doesn't address metadata pagecache, although it could do so.
> >
>
> With no kernel ch
On 3/3/07, Andrew Morton <[EMAIL PROTECTED]> wrote:
But yes, updatedb's pagecache usage will be mainly metadata, and this tool
doesn't address metadata pagecache, although it could do so.
With no kernel changes? How? I can't find an equivalent API to
posix_fadvise() for metadata.
Lee
-
To u
Andrew Morton wrote:
Doing a refault thing would help a bit, but stops working at a certain point.
At what point does it stop working?
We need to store that this-page-got-reclaimed info somewhere. I don't know
how space-efficient that is. Did anyone ever do an implementation?
One 32 bit w
On Sat, 03 Mar 2007 20:23:07 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
> > On Sat, 03 Mar 2007 19:01:01 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
>
> >> The use-once policy we have in the kernel should work
> >> perfectly fine for backups. All we need to do is
> >>
Eric St-Laurent wrote:
While I think that more user space applications should use fadvise() to
avoid polluting the page cache with unneeded data, I still think the
kernel should be more fair in regard to page cache management.
Personally, I've experienced some sluggish performance after copyi
On Sat, 3 Mar 2007 20:16:09 -0500 "Lee Revell" <[EMAIL PROTECTED]> wrote:
> On 3/3/07, Andrew Morton <[EMAIL PROTECTED]> wrote:
> > The tool uses an LD_PRELOAD hack to intercept glibc's read(), pread(),
> > write(), pwrite(), close() and dup2() functions. pagecache control is done
> > via posix_f
Andrew Morton wrote:
On Sat, 03 Mar 2007 19:01:01 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
The use-once policy we have in the kernel should work
perfectly fine for backups. All we need to do is
actually honor the accessed bit on active page cache
pages, instead of flushing them onto the
On Sat, 03 Mar 2007 17:02:31 -0800 Ray Lee <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
> >
>
> Would there be any other users of it than updatedb?
updatedb is the notorious one.
Alas, one can envisage sane workloads which really really really really
want to cache millions of dentries and
On 3/3/07, Andrew Morton <[EMAIL PROTECTED]> wrote:
The tool uses an LD_PRELOAD hack to intercept glibc's read(), pread(),
write(), pwrite(), close() and dup2() functions. pagecache control is done
via posix_fadvise() and sync_file_range().
How could this have any effect on the updatedb probl
On Sat, 03 Mar 2007 19:14:59 -0500 Eric St-Laurent <[EMAIL PROTECTED]> wrote:
> On Sat, 2007-03-03 at 12:29 -0800, Andrew Morton wrote:
>
>
> > There is much more which could be done to make this code smarter, but I
> > think the lesson here is that we can produce a far, far better result doing
On Sat, 2007-03-03 at 12:29 -0800, Andrew Morton wrote:
> There is much more which could be done to make this code smarter, but I
> think the lesson here is that we can produce a far, far better result doing
> this work in userspace than we could ever hope to do with an in-kernel
> implementation
On Sat, 03 Mar 2007 19:01:01 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
> > On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> >
> >> backup program
> >
> > A suitable policy for a backup program would probably be to invalidate any
> > output f
Andrew Morton wrote:
>
Would there be any other users of it than updatedb? I'm not coming up
with much, but given that I'm not always clever, that doesn't mean much.
A hypothetical on-demand file virus scanner is
going to hit already cached or about-to-be-cached entries by definition.
Perhaps s
Andrew Morton wrote:
On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
backup program
A suitable policy for a backup program would probably be to invalidate any
output file(s) and to invalidate those pages of the input files which were
not in cache when the backup pro
On Sun, 4 Mar 2007 00:01:55 +0100 bert hubert <[EMAIL PROTECTED]> wrote:
> On Sat, Mar 03, 2007 at 02:26:09PM -0800, Andrew Morton wrote:
> > > > It is *not* a global instruction. It uses setenv, so the user's policy
> > > > affects only the target process and its forked children.
> > >
> > > ..
On Sat Mar 03, 2007 at 02:26:09PM -0800, Andrew Morton wrote:
> On Sat, 03 Mar 2007 17:19:00 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
>
> > > It is *not* a global instruction. It uses setenv, so the user's policy
> > > affects only the target process and its forked children.
> >
> > ... and
On Sat, 3 Mar 2007 14:58:48 -0800 "Ray Lee" <[EMAIL PROTECTED]> wrote:
> On 3/3/07, Andrew Morton <[EMAIL PROTECTED]> wrote:
> > It is to address the "waah, backups fill my memory with pagecache" and the
> > "waah, updatedb swapped everything out" and the "waah, copying a DVD
> > gobbled all my me
On Sat, Mar 03, 2007 at 02:26:09PM -0800, Andrew Morton wrote:
> > > It is *not* a global instruction. It uses setenv, so the user's policy
> > > affects only the target process and its forked children.
> >
> > ... and all other processes accessing the same file(s)!
> >
> > Your library and the
On 3/3/07, Andrew Morton <[EMAIL PROTECTED]> wrote:
It is to address the "waah, backups fill my memory with pagecache" and the
"waah, updatedb swapped everything out" and the "waah, copying a DVD
gobbled all my memory" problems.
Is the updatedb problem really due to pagecache?
When running
On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> backup program
A suitable policy for a backup program would probably be to invalidate any
output file(s) and to invalidate those pages of the input files which were
not in cache when the backup program first opened those
On Sat, 03 Mar 2007 17:28:35 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
> > On Sat, 03 Mar 2007 17:19:00 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> >
> >>> It is *not* a global instruction. It uses setenv, so the user's policy
> >>> affects only the target process
On Sat, 03 Mar 2007 17:25:30 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
>
> > Well, backup programs are a unique case. Let's say instead that the user
> > has just generated a 600MB ISO image.
> >
> > The kernel *just doesn't know* whether the user will next try to rea
Andrew Morton wrote:
On Sat, 03 Mar 2007 17:19:00 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
It is *not* a global instruction. It uses setenv, so the user's policy
affects only the target process and its forked children.
... and all other processes accessing the same file(s)!
Your library
On Sat, 03 Mar 2007 17:19:00 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> > It is *not* a global instruction. It uses setenv, so the user's policy
> > affects only the target process and its forked children.
>
> ... and all other processes accessing the same file(s)!
>
> Your library and the
Andrew Morton wrote:
Well, backup programs are a unique case. Let's say instead that the user
has just generated a 600MB ISO image.
The kernel *just doesn't know* whether the user will next try to read the
kernel tree or will next try to read that ISO image.
That, Rik, is my point, and is the
Andrew Morton wrote:
On Sat, 3 Mar 2007 22:41:09 +0100 bert hubert <[EMAIL PROTECTED]> wrote:
How can you make global policy decisions based on the intent
of one program?
By not doing so.
yup.
Andrew's program is fine in principle, except that the
linux kernel treats the communication of a
On Sat, 3 Mar 2007 22:41:09 +0100 bert hubert <[EMAIL PROTECTED]> wrote:
> > How can you make global policy decisions based on the intent
> > of one program?
>
> By not doing so.
yup.
> Andrew's program is fine in principle, except that the
> linux kernel treats the communication of a program's
On Sat, 03 Mar 2007 16:30:56 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
> > On Sat, 03 Mar 2007 15:40:42 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
>
> >> I am sick and tired of the "this is hard, let userspace do it" attitude.
> >
> > Anything you try to do in-kerne
On Sat, Mar 03, 2007 at 04:30:56PM -0500, Rik van Riel wrote:
> The user has been accessing the kernel tree over and over
> again, for hours on end (compile testing a patch). Along
> comes a backup program, that tells you to evict the whole
> thing from the cache.
This is arguably due to a linux
Andrew Morton wrote:
On Sat, 03 Mar 2007 15:40:42 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
I am sick and tired of the "this is hard, let userspace do it" attitude.
Anything you try to do in-kernel will catastrophically screw up some
workloads. You don't have a chance of getting this ri
On Sat, 03 Mar 2007 15:40:42 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote:
> Andrew Morton wrote:
>
> > It is to address the "waah, backups fill my memory with pagecache" and the
> > "waah, updatedb swapped everything out" and the "waah, copying a DVD
> > gobbled all my memory" problems.
>
> By
Andrew Morton wrote:
It is to address the "waah, backups fill my memory with pagecache" and the
"waah, updatedb swapped everything out" and the "waah, copying a DVD
gobbled all my memory" problems.
By removing pressure from the page cache, you'll only allow updatedb
to grow the inode and dentr
I've uploaded to http://userweb.kernel.org/~akpm/pagecache-management/ a
little tool which permits the management of the pagecache usage of
arbitrary applications. Effectively it prevents the targetted application
from using any pagecache at all.
It is to address the "waah, backups fill my memor
45 matches
Mail list logo
