Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-30 Thread Seigo Tanimura
On Mon, 28 Oct 2002 00:54:57 -0800 (PST), Matthew Dillon [EMAIL PROTECTED] said: dillon I can demonstrate the issue with a simple test. Create a large file dillon with dd, larger then physical memory: dillon dd if=/dev/zero of=test bs=1m count=4096# create a 4G file. dillon

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-30 Thread Matthew Dillon
Yes, this makes a lot of sense to me. You are exercising the system in a way that breaks the LRU algorithm. The buffer cache, without your patch, is carefully tuned to deal with this case... that is why vm_page_dontneed() exists and why the vm_object code calls it. This

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-28 Thread Matthew Dillon
Hmm. Well, the real problem is not going to be the struct bio but will instead be the filesystem support. Filesystems expect KVA mapped data from the buffer cache, and they use pointers to the data all over the place. The buffer cache is very efficient, at least as long

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-28 Thread Matthew Dillon
:I was going to comment on fragmentation issues, but that seems to have :been very well covered. I would like to point out that removing the :buffer_map not only contributes to kernel map fragmentation, but also :contention for the kernel map. It might also prevent us from removing :giant from

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-27 Thread Seigo Tanimura
On Wed, 23 Oct 2002 16:51:44 -0400 (EDT), Jeff Roberson [EMAIL PROTECTED] said: jroberson I do, however, like the page unwiring idea. As long as it's not too jroberson expensive. I have been somewhat disappointed that the buffer cache's jroberson buffers are hands off for the vm. I'm

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-27 Thread Seigo Tanimura
On Thu, 24 Oct 2002 15:05:30 +1000 (EST), Bruce Evans [EMAIL PROTECTED] said: bde Almost exactly what we have. It turns out to be not very good, at least bde in its current implementation, since remapping is too expensive. Things bde work OK to the extent that remapping is not required, but

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-27 Thread Bruce Evans
On Mon, 28 Oct 2002, Seigo Tanimura wrote: On Thu, 24 Oct 2002 15:05:30 +1000 (EST), Bruce Evans [EMAIL PROTECTED] said: bde Almost exactly what we have. It turns out to be not very good, at least bde in its current implementation, since remapping is too expensive. Things bde work OK to

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-23 Thread Jeff Roberson
On Wed, 23 Oct 2002, Seigo Tanimura wrote: On Wed, 23 Oct 2002 16:44:06 +1000 (EST), Bruce Evans [EMAIL PROTECTED] said: Incidentally, Solaris 7 on sun4u reserves a space of 256MB in the KVM according to Solaris Internals. On i386 (x86), the size is only 4MB. Not sure whether they use

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-23 Thread Julian Elischer
On Wed, 23 Oct 2002, Jeff Roberson wrote: I do, however, like the page unwiring idea. As long as it's not too expensive. I have been somewhat disappointed that the buffer cache's buffers are hands off for the vm. I'm confused about your approach though. I think that the rewire

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-23 Thread Poul-Henning Kamp
In message [EMAIL PROTECTED], Ju lian Elischer writes: Bill Jolitz had a plan for 386BSD where all the buffers were nearly always unmapped from KVM. He was going to have a number of slots available for mapping them which would be used in a lifo order This entire area needs to be rethought. And

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-23 Thread Bruce Evans
On Wed, 23 Oct 2002, Julian Elischer wrote: Bill Jolitz had a plan for 386BSD where all the buffers were nearly always unmapped from KVM. He was going to have a number of slots available for mapping them which would be used in a lifo order The number of slots was going to be somehow tunable

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-23 Thread Bruce Evans
On Tue, 22 Oct 2002, Seigo Tanimura wrote: Introduction: The I/O buffer of the kernel are currently allocated in buffer_map sized statically upon boot, and never grows. This limits the scale of I/O performance on a host with large physical memory. We used to tune NBUF to cope with that

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-23 Thread Seigo Tanimura
On Wed, 23 Oct 2002 16:44:06 +1000 (EST), Bruce Evans [EMAIL PROTECTED] said: bde I should be the last to defend the current design and implementation of bde the buffer cache, since I think it gets almost everything wrong (the bde implementation is OK, but has vast complications to work around

Dynamic growth of the buffer and buffer page reclaim

2002-10-22 Thread Seigo Tanimura
Introduction: The I/O buffer of the kernel are currently allocated in buffer_map sized statically upon boot, and never grows. This limits the scale of I/O performance on a host with large physical memory. We used to tune NBUF to cope with that problem. This workaround, however, results in a

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-22 Thread Poul-Henning Kamp
In message [EMAIL PROTECTED], Seigo Tanimur a writes: The patch at: http://people.FreeBSD.org/~tanimura/patches/dynamicbuf.diff.gz Comments and flames are welcome. Thanks a lot. This looks very very interesting! -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 [EMAIL PROTECTED]

Re: Dynamic growth of the buffer and buffer page reclaim

2002-10-22 Thread Julian Elischer
On Tue, 22 Oct 2002, Seigo Tanimura wrote: Introduction: [...] The patch at: http://people.FreeBSD.org/~tanimura/patches/dynamicbuf.diff.gz Cool.. -jbaselinew/ my patch realusersys realusersys 1 1608.21 1387.94 125.96 1577.88