Re: DVD blockdevice buffers

2001-06-01 Thread Pavel Machek
Hi! > I can easily give more examples - just ask. BTW, the fact that this stuff > is so fragmented is not a bug - we want it evenly spread over disk, just > to have the ability to allocate a block/inode not too far from the piece > of bitmap we'll need to modify. BTW is this still true? This

Re: DVD blockdevice buffers

2001-06-01 Thread Pavel Machek
Hi! I can easily give more examples - just ask. BTW, the fact that this stuff is so fragmented is not a bug - we want it evenly spread over disk, just to have the ability to allocate a block/inode not too far from the piece of bitmap we'll need to modify. BTW is this still true? This

Re: DVD blockdevice buffers

2001-05-25 Thread Linus Torvalds
On 25 May 2001, Eric W. Biederman wrote: > > I obviously picked a bad name, and a bad place to start. > int data_uptodate(struct page *page, unsigned offset, unsigned len) > > This is really an extension to PG_uptodate, not readpage. Ugh. The above is just horrible. It doesn't fix any

Re: DVD blockdevice buffers

2001-05-25 Thread Eric W. Biederman
Linus Torvalds <[EMAIL PROTECTED]> writes: > On 25 May 2001, Eric W. Biederman wrote: > > > > For the small random read case we could use a > > mapping->a_ops->readpartialpage > > No, if so I'd prefer to just change "readpage()" to take the same kinds of > arguments commit_page() does,

Re: blkdev-pagecache-2 [was Re: DVD blockdevice buffers]

2001-05-25 Thread Andrea Arcangeli
On Fri, May 25, 2001 at 10:12:51PM +0200, Andrea Arcangeli wrote: > >ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.4.5pre6/blkdev-pagecache-2 ^ 4 sorry - To unsubscribe from this list: send the line

blkdev-pagecache-2 [was Re: DVD blockdevice buffers]

2001-05-25 Thread Andrea Arcangeli
On Thu, May 24, 2001 at 12:32:20AM +0200, Andrea Arcangeli wrote: > userspace. I will try to work on the blkdev patch tomorrow to bring it > in an usable state. It seems in an usable state right but it is still very early beta, I need to recheck the whole thing, I will do that tomorrow, for now

Re: DVD blockdevice buffers

2001-05-25 Thread Stephen C. Tweedie
Hi, On Fri, May 25, 2001 at 02:24:52PM -0400, Alexander Viro wrote: > If you are OK with adding two extra arguments to ->readpage() I could > submit a patch replacing that with plain and simple page cache by tomorrow. > It should not be a problem to port, but I want to get some sleep before >

Re: DVD blockdevice buffers

2001-05-25 Thread Stephen C. Tweedie
Hi, On Fri, May 25, 2001 at 09:09:37AM -0600, Eric W. Biederman wrote: > The case we don't get quite right are partial reads that hit cached > data, on a page that doesn't have PG_Uptodate set. We don't actually > need to do the I/O on the surrounding page to satisfy the read > request. But

Re: DVD blockdevice buffers

2001-05-25 Thread Alexander Viro
On Fri, 25 May 2001, Linus Torvalds wrote: > For example, I suspect that the metadata bitmaps in particular cache so > well that the fact that we need to do several seeks over them every once > in a while is a non-issue: we might be happier having the bitmaps in > memory (and having simpler

Re: DVD blockdevice buffers

2001-05-25 Thread Linus Torvalds
On Fri, 25 May 2001, Alexander Viro wrote: > > OK, here's a real-world scenario: inode table on 1Kb ext2 (or 4Kb on > Alpha, etc.) consists of compact pieces - one per cylinder group. > > There is a natural mapping from inodes to offsets in that beast. > However, these pieces can trivially be

Re: DVD blockdevice buffers

2001-05-25 Thread Alexander Viro
On Fri, 25 May 2001, Linus Torvalds wrote: > If somebody can show that the above is worth it and worth implementing (ie > the Al Viro kind of "I have a real-life schenario where I'd like to use > it"), and implements it (should be a fairly trivial exercise), then I'll > happily accept new

Re: DVD blockdevice buffers

2001-05-25 Thread Linus Torvalds
On 25 May 2001, Eric W. Biederman wrote: > > For the small random read case we could use a > mapping->a_ops->readpartialpage No, if so I'd prefer to just change "readpage()" to take the same kinds of arguments commit_page() does, namely the beginning and end of the read area. Filesystems

Re: DVD blockdevice buffers

2001-05-25 Thread Eric W. Biederman
"Stephen C. Tweedie" <[EMAIL PROTECTED]> writes: > Hi, > > On Wed, May 23, 2001 at 01:01:56PM -0700, Linus Torvalds wrote: > > > On Wed, 23 May 2001, Stephen C. Tweedie wrote: > > > > that the filesystems already do. And you can do it a lot _better_ than the > > > > > current

Re: DVD blockdevice buffers

2001-05-25 Thread Eric W. Biederman
Stephen C. Tweedie [EMAIL PROTECTED] writes: Hi, On Wed, May 23, 2001 at 01:01:56PM -0700, Linus Torvalds wrote: On Wed, 23 May 2001, Stephen C. Tweedie wrote: that the filesystems already do. And you can do it a lot _better_ than the current buffer-cache-based approach. Done

Re: DVD blockdevice buffers

2001-05-25 Thread Linus Torvalds
On 25 May 2001, Eric W. Biederman wrote: For the small random read case we could use a mapping-a_ops-readpartialpage No, if so I'd prefer to just change readpage() to take the same kinds of arguments commit_page() does, namely the beginning and end of the read area. Filesystems could

Re: DVD blockdevice buffers

2001-05-25 Thread Alexander Viro
On Fri, 25 May 2001, Linus Torvalds wrote: If somebody can show that the above is worth it and worth implementing (ie the Al Viro kind of I have a real-life schenario where I'd like to use it), and implements it (should be a fairly trivial exercise), then I'll happily accept new semantics

Re: DVD blockdevice buffers

2001-05-25 Thread Linus Torvalds
On Fri, 25 May 2001, Alexander Viro wrote: OK, here's a real-world scenario: inode table on 1Kb ext2 (or 4Kb on Alpha, etc.) consists of compact pieces - one per cylinder group. There is a natural mapping from inodes to offsets in that beast. However, these pieces can trivially be not

Re: DVD blockdevice buffers

2001-05-25 Thread Alexander Viro
On Fri, 25 May 2001, Linus Torvalds wrote: For example, I suspect that the metadata bitmaps in particular cache so well that the fact that we need to do several seeks over them every once in a while is a non-issue: we might be happier having the bitmaps in memory (and having simpler code),

Re: DVD blockdevice buffers

2001-05-25 Thread Stephen C. Tweedie
Hi, On Fri, May 25, 2001 at 09:09:37AM -0600, Eric W. Biederman wrote: The case we don't get quite right are partial reads that hit cached data, on a page that doesn't have PG_Uptodate set. We don't actually need to do the I/O on the surrounding page to satisfy the read request. But we do

Re: DVD blockdevice buffers

2001-05-25 Thread Stephen C. Tweedie
Hi, On Fri, May 25, 2001 at 02:24:52PM -0400, Alexander Viro wrote: If you are OK with adding two extra arguments to -readpage() I could submit a patch replacing that with plain and simple page cache by tomorrow. It should not be a problem to port, but I want to get some sleep before

blkdev-pagecache-2 [was Re: DVD blockdevice buffers]

2001-05-25 Thread Andrea Arcangeli
On Thu, May 24, 2001 at 12:32:20AM +0200, Andrea Arcangeli wrote: userspace. I will try to work on the blkdev patch tomorrow to bring it in an usable state. It seems in an usable state right but it is still very early beta, I need to recheck the whole thing, I will do that tomorrow, for now it

Re: blkdev-pagecache-2 [was Re: DVD blockdevice buffers]

2001-05-25 Thread Andrea Arcangeli
On Fri, May 25, 2001 at 10:12:51PM +0200, Andrea Arcangeli wrote: ftp://ftp.kernel.org/pub/linux/kernel/people/andrea/patches/v2.2/2.4.5pre6/blkdev-pagecache-2 ^ 4 sorry - To unsubscribe from this list: send the line

Re: DVD blockdevice buffers

2001-05-25 Thread Eric W. Biederman
Linus Torvalds [EMAIL PROTECTED] writes: On 25 May 2001, Eric W. Biederman wrote: For the small random read case we could use a mapping-a_ops-readpartialpage No, if so I'd prefer to just change readpage() to take the same kinds of arguments commit_page() does, namely the beginning

Re: DVD blockdevice buffers

2001-05-25 Thread Linus Torvalds
On 25 May 2001, Eric W. Biederman wrote: I obviously picked a bad name, and a bad place to start. int data_uptodate(struct page *page, unsigned offset, unsigned len) This is really an extension to PG_uptodate, not readpage. Ugh. The above is just horrible. It doesn't fix any problems,

Re: DVD blockdevice buffers

2001-05-24 Thread Stephen C. Tweedie
Hi, On Wed, May 23, 2001 at 01:01:56PM -0700, Linus Torvalds wrote: > On Wed, 23 May 2001, Stephen C. Tweedie wrote: > > > that the filesystems already do. And you can do it a lot _better_ than the > > > current buffer-cache-based approach. Done right, you can actually do all > > > IO in

Re: DVD blockdevice buffers

2001-05-23 Thread Andrea Arcangeli
On Wed, May 23, 2001 at 04:40:14PM -0400, Jeff Garzik wrote: > Linus Torvalds wrote: > > Now, it may be that the preliminary patches from Andrea do not work this > > way. I didn't look at them too closely, and I assume that Andrea basically > > made the block-size be the same as the page size.

Re: DVD blockdevice buffers

2001-05-23 Thread Andrea Arcangeli
On Wed, May 23, 2001 at 06:13:13PM -0400, Alexander Viro wrote: > Uh-oh... After you solved what? The superblock is pinned by the kernel in buffercache while you fsck a ro mounted ext2, so I must somehow uptodate this superblock in the buffercache before collecting away the pagecache containing

Re: DVD blockdevice buffers

2001-05-23 Thread Andrea Arcangeli
On Wed, May 23, 2001 at 01:01:56PM -0700, Linus Torvalds wrote: > [..] I assume that Andrea basically > made the block-size be the same as the page size. That's how I would have exactly (softblocksize is 4k fixed, regardless of the page cache size to avoid confusing device drivers). > done it

Re: DVD blockdevice buffers

2001-05-23 Thread Alexander Viro
On Thu, 24 May 2001, Andrea Arcangeli wrote: > prefer to think about it after I solved the coherency issues between > pinned buffer cache and filesystem, so after the showstoppers are solved Uh-oh... After you solved what? - To unsubscribe from this list: send the line "unsubscribe

Re: DVD blockdevice buffers

2001-05-23 Thread Jeff Garzik
Linus Torvalds wrote: > Now, it may be that the preliminary patches from Andrea do not work this > way. I didn't look at them too closely, and I assume that Andrea basically > made the block-size be the same as the page size. That's how I would have > done it (and then waited for people to find

Re: DVD blockdevice buffers

2001-05-23 Thread Linus Torvalds
On Wed, 23 May 2001, Stephen C. Tweedie wrote: > > that the filesystems already do. And you can do it a lot _better_ than the > > current buffer-cache-based approach. Done right, you can actually do all > > IO in page-sized chunks, BUT fall down on sector-sized things for the > > cases where

Re: DVD blockdevice buffers

2001-05-23 Thread Stephen C. Tweedie
Hi, On Wed, May 23, 2001 at 11:12:00AM -0700, Linus Torvalds wrote: > > On Wed, 23 May 2001, Stephen C. Tweedie wrote: > No, you can actually do all the "prepare_write()"/"commit_write()" stuff > that the filesystems already do. And you can do it a lot _better_ than the > current

Re: DVD blockdevice buffers

2001-05-23 Thread Linus Torvalds
On Wed, 23 May 2001, Stephen C. Tweedie wrote: > > Right. I'd like to see buffered IO able to work well --- apart from > the VM issues, it's the easiest way to allow the application to take > advantage of readahead. However, there's one sticking point we > encountered, which is applications

Re: DVD blockdevice buffers

2001-05-23 Thread Stephen C. Tweedie
Hi, On Sat, May 19, 2001 at 07:36:07PM -0700, Linus Torvalds wrote: > Right now we don't try to aggressively drop streaming pages, but it's > possible. Using raw devices is a silly work-around that should not be > needed, and this load shows a real problem in current Linux (one soon to > be

Re: DVD blockdevice buffers

2001-05-23 Thread Stephen C. Tweedie
Hi, On Sat, May 19, 2001 at 07:36:07PM -0700, Linus Torvalds wrote: Right now we don't try to aggressively drop streaming pages, but it's possible. Using raw devices is a silly work-around that should not be needed, and this load shows a real problem in current Linux (one soon to be fixed,

Re: DVD blockdevice buffers

2001-05-23 Thread Linus Torvalds
On Wed, 23 May 2001, Stephen C. Tweedie wrote: Right. I'd like to see buffered IO able to work well --- apart from the VM issues, it's the easiest way to allow the application to take advantage of readahead. However, there's one sticking point we encountered, which is applications which

Re: DVD blockdevice buffers

2001-05-23 Thread Stephen C. Tweedie
Hi, On Wed, May 23, 2001 at 11:12:00AM -0700, Linus Torvalds wrote: On Wed, 23 May 2001, Stephen C. Tweedie wrote: No, you can actually do all the prepare_write()/commit_write() stuff that the filesystems already do. And you can do it a lot _better_ than the current buffer-cache-based

Re: DVD blockdevice buffers

2001-05-23 Thread Linus Torvalds
On Wed, 23 May 2001, Stephen C. Tweedie wrote: that the filesystems already do. And you can do it a lot _better_ than the current buffer-cache-based approach. Done right, you can actually do all IO in page-sized chunks, BUT fall down on sector-sized things for the cases where you want

Re: DVD blockdevice buffers

2001-05-23 Thread Jeff Garzik
Linus Torvalds wrote: Now, it may be that the preliminary patches from Andrea do not work this way. I didn't look at them too closely, and I assume that Andrea basically made the block-size be the same as the page size. That's how I would have done it (and then waited for people to find real

Re: DVD blockdevice buffers

2001-05-23 Thread Alexander Viro
On Thu, 24 May 2001, Andrea Arcangeli wrote: prefer to think about it after I solved the coherency issues between pinned buffer cache and filesystem, so after the showstoppers are solved Uh-oh... After you solved what? - To unsubscribe from this list: send the line unsubscribe linux-kernel

Re: DVD blockdevice buffers

2001-05-23 Thread Andrea Arcangeli
On Wed, May 23, 2001 at 01:01:56PM -0700, Linus Torvalds wrote: [..] I assume that Andrea basically made the block-size be the same as the page size. That's how I would have exactly (softblocksize is 4k fixed, regardless of the page cache size to avoid confusing device drivers). done it (and

Re: DVD blockdevice buffers

2001-05-23 Thread Andrea Arcangeli
On Wed, May 23, 2001 at 06:13:13PM -0400, Alexander Viro wrote: Uh-oh... After you solved what? The superblock is pinned by the kernel in buffercache while you fsck a ro mounted ext2, so I must somehow uptodate this superblock in the buffercache before collecting away the pagecache containing

Re: DVD blockdevice buffers

2001-05-23 Thread Andrea Arcangeli
On Wed, May 23, 2001 at 04:40:14PM -0400, Jeff Garzik wrote: Linus Torvalds wrote: Now, it may be that the preliminary patches from Andrea do not work this way. I didn't look at them too closely, and I assume that Andrea basically made the block-size be the same as the page size. That's

Re: DVD blockdevice buffers

2001-05-21 Thread Jens Axboe
On Mon, May 21 2001, Adam Schrotenboer wrote: > On Sun, 20 May 2001, Jens Axboe wrote: > > > On Sat, May 19 2001, Adam Schrotenboer wrote: > > > /dev/raw* Where? I can't find it in my .config (grep RAW .config). I am > > > using 2.4.4-ac11 and playing w/ 2.4.5-pre3. > > > > It's automagically

Re: DVD blockdevice buffers

2001-05-21 Thread Adam Schrotenboer
On Sun, 20 May 2001, Jens Axboe wrote: > On Sat, May 19 2001, Adam Schrotenboer wrote: > > /dev/raw* Where? I can't find it in my .config (grep RAW .config). I am > > using 2.4.4-ac11 and playing w/ 2.4.5-pre3. > > It's automagically included, no config options necessary >

Re: DVD blockdevice buffers

2001-05-21 Thread Adam Schrotenboer
On Sun, 20 May 2001, Jens Axboe wrote: On Sat, May 19 2001, Adam Schrotenboer wrote: /dev/raw* Where? I can't find it in my .config (grep RAW .config). I am using 2.4.4-ac11 and playing w/ 2.4.5-pre3. It's automagically included, no config options necessary (drivers/char/raw.c) Then

Re: DVD blockdevice buffers

2001-05-21 Thread Jens Axboe
On Mon, May 21 2001, Adam Schrotenboer wrote: On Sun, 20 May 2001, Jens Axboe wrote: On Sat, May 19 2001, Adam Schrotenboer wrote: /dev/raw* Where? I can't find it in my .config (grep RAW .config). I am using 2.4.4-ac11 and playing w/ 2.4.5-pre3. It's automagically included, no

Re: DVD blockdevice buffers

2001-05-19 Thread Linus Torvalds
In article <[EMAIL PROTECTED]>, Jens Axboe <[EMAIL PROTECTED]> wrote: >> >> As a result the system performance goes down. I'm still able to use >> my applications, but es every single piece of unused memory is swapped >> out, and swapping in costs a certain amount of time. > >That's why

Re: DVD blockdevice buffers

2001-05-19 Thread Adam Schrotenboer
On Sun, 20 May 2001, Jens Axboe wrote: > On Sat, May 19 2001, Adam Schrotenboer wrote: > > /dev/raw* Where? I can't find it in my .config (grep RAW .config). I am > > using 2.4.4-ac11 and playing w/ 2.4.5-pre3. > > It's automagically included, no config options necessary >

Re: DVD blockdevice buffers

2001-05-19 Thread Jens Axboe
On Sat, May 19 2001, Adam Schrotenboer wrote: > /dev/raw* Where? I can't find it in my .config (grep RAW .config). I am > using 2.4.4-ac11 and playing w/ 2.4.5-pre3. It's automagically included, no config options necessary (drivers/char/raw.c) -- Jens Axboe - To unsubscribe from this list:

Re: DVD blockdevice buffers

2001-05-19 Thread Adam Schrotenboer
/dev/raw* Where? I can't find it in my .config (grep RAW .config). I am using 2.4.4-ac11 and playing w/ 2.4.5-pre3. TIA Adam Schrotenboer - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] More majordomo info at

Re: DVD blockdevice buffers

2001-05-19 Thread Adam Schrotenboer
/dev/raw* Where? I can't find it in my .config (grep RAW .config). I am using 2.4.4-ac11 and playing w/ 2.4.5-pre3. TIA Adam Schrotenboer - To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to [EMAIL PROTECTED] More majordomo info at

Re: DVD blockdevice buffers

2001-05-19 Thread Jens Axboe
On Sat, May 19 2001, Adam Schrotenboer wrote: /dev/raw* Where? I can't find it in my .config (grep RAW .config). I am using 2.4.4-ac11 and playing w/ 2.4.5-pre3. It's automagically included, no config options necessary (drivers/char/raw.c) -- Jens Axboe - To unsubscribe from this list:

Re: DVD blockdevice buffers

2001-05-19 Thread Adam Schrotenboer
On Sun, 20 May 2001, Jens Axboe wrote: On Sat, May 19 2001, Adam Schrotenboer wrote: /dev/raw* Where? I can't find it in my .config (grep RAW .config). I am using 2.4.4-ac11 and playing w/ 2.4.5-pre3. It's automagically included, no config options necessary (drivers/char/raw.c) then

Re: DVD blockdevice buffers

2001-05-19 Thread Linus Torvalds
In article [EMAIL PROTECTED], Jens Axboe [EMAIL PROTECTED] wrote: As a result the system performance goes down. I'm still able to use my applications, but es every single piece of unused memory is swapped out, and swapping in costs a certain amount of time. That's why streaming media

Re: DVD blockdevice buffers

2001-05-18 Thread Eduard Hasenleithner
On Fri, May 18, 2001 at 09:25:31PM +0200, Jens Axboe wrote: > On Fri, May 18 2001, Eduard Hasenleithner wrote: > > I have a problem with the buffering mechanism of my blockdevice, > > namely a ide_scsi DVD-ROM drive. After inserting a DVD and reading > > data linearly from the DVD, an excessive

Re: DVD blockdevice buffers

2001-05-18 Thread Jens Axboe
On Fri, May 18 2001, Eduard Hasenleithner wrote: > I have a problem with the buffering mechanism of my blockdevice, > namely a ide_scsi DVD-ROM drive. After inserting a DVD and reading > data linearly from the DVD, an excessive amount of buffer memory gets > allocated. > > This can easily be

Re: DVD blockdevice buffers

2001-05-18 Thread Jens Axboe
On Fri, May 18 2001, Eduard Hasenleithner wrote: I have a problem with the buffering mechanism of my blockdevice, namely a ide_scsi DVD-ROM drive. After inserting a DVD and reading data linearly from the DVD, an excessive amount of buffer memory gets allocated. This can easily be

Re: DVD blockdevice buffers

2001-05-18 Thread Eduard Hasenleithner
On Fri, May 18, 2001 at 09:25:31PM +0200, Jens Axboe wrote: On Fri, May 18 2001, Eduard Hasenleithner wrote: I have a problem with the buffering mechanism of my blockdevice, namely a ide_scsi DVD-ROM drive. After inserting a DVD and reading data linearly from the DVD, an excessive amount