Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-10 Thread Daniel Phillips
On Monday 10 December 2007 13:31, Jonathan Corbet wrote: > Hey, Daniel, > > I'm just getting around to looking at this. One thing jumped out at me: > > + if (bio->bi_throttle) { > > + struct request_queue *q = bio->bi_queue; > > + bio->bi_throttle = 0; /* or detect multiple e

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-10 Thread Pekka Enberg
Hi, On Dec 10, 2007 11:31 PM, Jonathan Corbet <[EMAIL PROTECTED]> wrote: > I'm just getting around to looking at this. One thing jumped out at me: > > > + if (bio->bi_throttle) { > > + struct request_queue *q = bio->bi_queue; > > + bio->bi_throttle = 0; /* or detect mu

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-10 Thread Jonathan Corbet
Hey, Daniel, I'm just getting around to looking at this. One thing jumped out at me: > + if (bio->bi_throttle) { > + struct request_queue *q = bio->bi_queue; > + bio->bi_throttle = 0; /* or detect multiple endio and err? */ > + atomic_add(bio->bi_throttle,

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-10 Thread Daniel Phillips
On Monday 10 December 2007 02:47, Jens Axboe wrote: > ...the warning on request_queue_t... There you go, Jens, service with a smile. Regards, Daniel --- 2.6.24-rc3-mm.clean/block/ll_rw_blk.c 2007-12-04 14:45:25.0 -0800 +++ 2.6.24-rc3-mm/block/ll_rw_blk.c 2007-12-10 03:27:42.0 -08

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-10 Thread Jens Axboe
On Wed, Dec 05 2007, Daniel Phillips wrote: > --- 2.6.24-rc3-mm.clean/block/ll_rw_blk.c 2007-12-04 14:45:25.0 > -0800 > +++ 2.6.24-rc3-mm/block/ll_rw_blk.c 2007-12-04 14:01:18.0 -0800 > @@ -3210,7 +3210,7 @@ static inline int bio_check_eod(struct b > */ > static inline voi

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-10 Thread Daniel Phillips
Hi Andrew, Unfortunately, I agreed with your suggestion too hastily. Not only would it be complex to implement, It does not work. It took me several days to put my finger on exactly why. Here it is in a nutshell: resources may be consumed _after_ the gatekeeper runs the "go, no go" throttl

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Daniel Phillips
On Thursday 06 December 2007 16:29, Andrew Morton wrote: > On Thu, 6 Dec 2007 16:04:41 -0800 > > Daniel Phillips <[EMAIL PROTECTED]> wrote: > > The runner up key idea is that we will gain a notion of "block > > device stack" (or block stack for short, so that we may implement > > block stackers) wh

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Andrew Morton
On Thu, 6 Dec 2007 16:04:41 -0800 Daniel Phillips <[EMAIL PROTECTED]> wrote: > The runner up key idea is that we will gain a notion of "block device > stack" (or block stack for short, so that we may implement block > stackers) which for the time being will simply be Device Mapper's > notion of

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Daniel Phillips
On Thursday 06 December 2007 13:53, Bill Davidsen wrote: > Daniel Phillips wrote: > The problem is that you (a) may or may not know just how bad a worst > case can be, and (b) may block unnecessarily by being pessimistic. True, but after a quick introspect I realized that that issue (it's really

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Bill Davidsen
Daniel Phillips wrote: On Wednesday 05 December 2007 17:24, Andrew Morton wrote: On Wed, 5 Dec 2007 16:03:01 -0800 Daniel Phillips <[EMAIL PROTECTED]> wrote: ...a block device these days may not be just a single device, but may be a stack of devices connected together by a generic mechanism su

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Daniel Phillips
On Thursday 06 December 2007 12:27, Andrew Morton wrote: > On Thu, 6 Dec 2007 12:04:14 -0800 > > Daniel Phillips <[EMAIL PROTECTED]> wrote: > > Any idea > > how to extend the accounting idea to all tasks involved in a > > particular block device stack? > > SMOP, I'd have thought. Agreed, which I r

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Andrew Morton
On Thu, 6 Dec 2007 12:04:14 -0800 Daniel Phillips <[EMAIL PROTECTED]> wrote: > Any idea > how to extend the accounting idea to all tasks involved in a particular > block device stack? SMOP, I'd have thought. As long as each piece of code which handles data for this stack knows that it's handli

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Daniel Phillips
On Thursday 06 December 2007 03:55, Andrew Morton wrote: > Consider an example. > > - We a-priori decide to limit a particular stack's peak memory usage > to 1MB Ah, this is the piece I was missing. > - We empirically discover that the maximum amount of memory which is > allocated by that stack

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Rik van Riel
On Thu, 6 Dec 2007 09:34:32 -0800 Andrew Morton <[EMAIL PROTECTED]> wrote: > That's all rather handwavy and misses a lot of details and might be > inaccurate too. Probably sufficient to just work out by hand the amount of > memory which the network stack will need to allocate. I expect it'll be

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Andrew Morton
On Thu, 6 Dec 2007 10:52:42 -0500 Rik van Riel <[EMAIL PROTECTED]> wrote: > On Thu, 6 Dec 2007 03:55:11 -0800 > Andrew Morton <[EMAIL PROTECTED]> wrote: > > > - We a-priori decide to limit a particular stack's peak memory usage to > > 1MB > > > > - We empirically discover that the maximum amou

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Rik van Riel
On Thu, 6 Dec 2007 03:55:11 -0800 Andrew Morton <[EMAIL PROTECTED]> wrote: > - We a-priori decide to limit a particular stack's peak memory usage to > 1MB > > - We empirically discover that the maximum amount of memory which is > allocated by that stack on behalf of a single BIO is 16kb. (ie

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Andrew Morton
On Thu, 6 Dec 2007 01:48:52 -0800 Daniel Phillips <[EMAIL PROTECTED]> wrote: > On Wednesday 05 December 2007 23:31, Andrew Morton wrote: > > > > Rather than asking the stack "how much memory will this request consume" > > > > you could instead ask "how much memory are you currently using". > > > >

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-06 Thread Daniel Phillips
On Wednesday 05 December 2007 23:31, Andrew Morton wrote: > > > Rather than asking the stack "how much memory will this request consume" > > > you could instead ask "how much memory are you currently using". > > > > > > ie: on entry to the stack, do > > > > > > current->account_block_allocatio

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-05 Thread Andrew Morton
On Wed, 5 Dec 2007 22:21:44 -0800 Daniel Phillips <[EMAIL PROTECTED]> wrote: > On Wednesday 05 December 2007 17:24, Andrew Morton wrote: > > On Wed, 5 Dec 2007 16:03:01 -0800 Daniel Phillips <[EMAIL PROTECTED]> wrote: > > > ...a block device these days may not be just a single > > > device, but m

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-05 Thread Daniel Phillips
On Wednesday 05 December 2007 17:24, Andrew Morton wrote: > On Wed, 5 Dec 2007 16:03:01 -0800 Daniel Phillips <[EMAIL PROTECTED]> wrote: > > ...a block device these days may not be just a single > > device, but may be a stack of devices connected together by a generic > > mechanism such as device

Re: [RFC] [PATCH] A clean approach to writeout throttling

2007-12-05 Thread Andrew Morton
On Wed, 5 Dec 2007 16:03:01 -0800 Daniel Phillips <[EMAIL PROTECTED]> wrote: > Practice > > So here is the key idea of today's patch: it provides a simple mechanism > for imposing a limit on the amount of data that can be in flight to any > particular block device. > > The limit on in-flight d

[RFC] [PATCH] A clean approach to writeout throttling

2007-12-05 Thread Daniel Phillips
Good afternoon, According to me, each line of code removed from the kernel is worth ten lines added. If lines can be removed while at the same time improving performance, that is worth, ah, about 1,000 times more than lines added, correct? Or maybe 1,000,000 times as much, if removing lines