Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-09 Thread Stephen C. Tweedie
Hi, On Wed, 2005-03-09 at 13:28, Jan Kara wrote: > Hmm. I see for example a place at jbd/commit.c, line 287 (which you > did not change in your patch) which does this and doesn't seem to be > protected against journal_unmap_buffer() (but maybe I miss something). > Not that I'd find that race

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-09 Thread Jan Kara
> On Tue, 2005-03-08 at 15:12, Jan Kara wrote: > > > Isn't also the following scenario dangerous? > > > > __journal_unfile_buffer(jh); > > journal_remove_journal_head(bh); > > It depends. I think the biggest problem here is that there's really no > written rule protecting this stuff

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-09 Thread Stephen C. Tweedie
Hi, On Tue, 2005-03-08 at 15:12, Jan Kara wrote: > Isn't also the following scenario dangerous? > > __journal_unfile_buffer(jh); > journal_remove_journal_head(bh); It depends. I think the biggest problem here is that there's really no written rule protecting this stuff universally. But

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-09 Thread Stephen C. Tweedie
Hi, On Tue, 2005-03-08 at 15:12, Jan Kara wrote: Isn't also the following scenario dangerous? __journal_unfile_buffer(jh); journal_remove_journal_head(bh); It depends. I think the biggest problem here is that there's really no written rule protecting this stuff universally. But in

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-09 Thread Jan Kara
On Tue, 2005-03-08 at 15:12, Jan Kara wrote: Isn't also the following scenario dangerous? __journal_unfile_buffer(jh); journal_remove_journal_head(bh); It depends. I think the biggest problem here is that there's really no written rule protecting this stuff universally. But

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-09 Thread Stephen C. Tweedie
Hi, On Wed, 2005-03-09 at 13:28, Jan Kara wrote: Hmm. I see for example a place at jbd/commit.c, line 287 (which you did not change in your patch) which does this and doesn't seem to be protected against journal_unmap_buffer() (but maybe I miss something). Not that I'd find that race

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-08 Thread Jan Kara
Hello, > On Mon, 2005-03-07 at 21:08, Stephen C. Tweedie wrote: > > > Right, that was what I was thinking might be possible. But for now I've > > just done the simple patch --- make sure we don't clear > > jh->b_transaction when we're just refiling buffers from one list to > > another. That

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-08 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 21:08, Stephen C. Tweedie wrote: > Right, that was what I was thinking might be possible. But for now I've > just done the simple patch --- make sure we don't clear > jh->b_transaction when we're just refiling buffers from one list to > another. That should have the

[PATCH] invalidate/o_direct livelock {was Re: [RFC] ext3/jbd race: releasing in-use journal_heads}

2005-03-08 Thread Stephen C. Tweedie
Hi, On Tue, 2005-03-08 at 09:28, Stephen C. Tweedie wrote: > I think it should be OK just to move the page->mapping != mapping test > above the page>index > end test. Sure, if all the pages have been > stolen by the time we see them, then we'll repeat without advancing > "next"; but we're still

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-08 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 23:50, Andrew Morton wrote: > truncate_inode_pages_range() seems to dtrt here. Can we do it in the same > manner in invalidate_inode_pages2_range()? > > > Something like: > - if (page->mapping != mapping || page->index > end) { > +

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-08 Thread Suparna Bhattacharya
On Mon, Mar 07, 2005 at 11:37:42PM -0800, Andrew Morton wrote: > Suparna Bhattacharya <[EMAIL PROTECTED]> wrote: > > > > (let me know if the interface in the patch > > I just posted seems like the right direction to use when we go for the > > cleanup) > > Well what are the semantics? Pass in

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-08 Thread Suparna Bhattacharya
On Mon, Mar 07, 2005 at 11:37:42PM -0800, Andrew Morton wrote: Suparna Bhattacharya [EMAIL PROTECTED] wrote: (let me know if the interface in the patch I just posted seems like the right direction to use when we go for the cleanup) Well what are the semantics? Pass in an inclusive

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-08 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 23:50, Andrew Morton wrote: truncate_inode_pages_range() seems to dtrt here. Can we do it in the same manner in invalidate_inode_pages2_range()? Something like: - if (page-mapping != mapping || page-index end) { +

[PATCH] invalidate/o_direct livelock {was Re: [RFC] ext3/jbd race: releasing in-use journal_heads}

2005-03-08 Thread Stephen C. Tweedie
Hi, On Tue, 2005-03-08 at 09:28, Stephen C. Tweedie wrote: I think it should be OK just to move the page-mapping != mapping test above the pageindex end test. Sure, if all the pages have been stolen by the time we see them, then we'll repeat without advancing next; but we're still making

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-08 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 21:08, Stephen C. Tweedie wrote: Right, that was what I was thinking might be possible. But for now I've just done the simple patch --- make sure we don't clear jh-b_transaction when we're just refiling buffers from one list to another. That should have the

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-08 Thread Jan Kara
Hello, On Mon, 2005-03-07 at 21:08, Stephen C. Tweedie wrote: Right, that was what I was thinking might be possible. But for now I've just done the simple patch --- make sure we don't clear jh-b_transaction when we're just refiling buffers from one list to another. That should have

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
Suparna Bhattacharya <[EMAIL PROTECTED]> wrote: > > (let me know if the interface in the patch > I just posted seems like the right direction to use when we go for the > cleanup) Well what are the semantics? Pass in an inclusive max_index and the gang lookup functions terminate when they hit

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Suparna Bhattacharya
On Mon, Mar 07, 2005 at 10:46:18PM -0800, Andrew Morton wrote: > Suparna Bhattacharya <[EMAIL PROTECTED]> wrote: > > > > > > yup, looks like the same issue we hit in wait_on_page_writeback_range > > during AIO work - probably want to break out of the outer loop as well > > when this happens. >

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
Suparna Bhattacharya <[EMAIL PROTECTED]> wrote: > > > yup, looks like the same issue we hit in wait_on_page_writeback_range > during AIO work - probably want to break out of the outer loop as well > when this happens. The `next = page_index' before breaking will do that for us. > > How hard

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Suparna Bhattacharya
Just as an idea of what I meant (dug up an old WIP patch): --- radix-tree.c2004-04-01 10:32:15.384556136 +0530 +++ radix-tree.c.end2004-04-01 11:11:07.176069944 +0530 @@ -562,7 +562,8 @@ EXPORT_SYMBOL(radix_tree_gang_lookup); */ static unsigned int __lookup_tag(struct

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Suparna Bhattacharya
yup, looks like the same issue we hit in wait_on_page_writeback_range during AIO work - probably want to break out of the outer loop as well when this happens. >From the old changelog: >> >> wait_on_page_writeback_range shouldn't wait for pages beyond the >> specified range. Ideally, the

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 21:22, Stephen C. Tweedie wrote: > altgr-scrlck is showing a range of EIPs all in ext3_direct_IO-> > invalidate_inode_pages2_range(). I'm seeing > > invalidate_inode_pages2_range()->pagevec_lookup()->find_get_pages() In invalidate_inode_pages2_range(), what happens

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
"Stephen C. Tweedie" <[EMAIL PROTECTED]> wrote: > > In invalidate_inode_pages2_range(), what happens if we lookup a pagevec, > get a bunch of pages back, but all the pages in the vec are beyond the > end of the range we want? hmm, yes. Another one :( > @@ -271,12 +271,13 @@ int

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 21:11, Andrew Morton wrote: > > I'm having trouble testing it, though --- I seem to be getting livelocks > > in O_DIRECT running 400 fsstress processes in parallel; ring any bells? > > Nope. I dont think anyone has been that cruel to ext3 for a while. > I assume

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
"Stephen C. Tweedie" <[EMAIL PROTECTED]> wrote: > > I'm having trouble testing it, though --- I seem to be getting livelocks > in O_DIRECT running 400 fsstress processes in parallel; ring any bells? Nope. I dont think anyone has been that cruel to ext3 for a while. I assume this workload used

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 20:31, Andrew Morton wrote: > jbd_lock_bh_journal_head() is supposed to be a > finegrained innermost lock whose mandate is purely for atomicity of adding > and removing the journal_head and the b_jcount refcounting. I don't recall > there being any deeper meaning than

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
"Stephen C. Tweedie" <[EMAIL PROTECTED]> wrote: > > Trouble is, that's not enough; journal_put_journal_head() can nuke the > buffer with merely the bh_journal_head lock held. In the code above it > would be enough to take the journal_head_lock over the unfile/file pair. > > Andrew, can you

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 16:40, Stephen C. Tweedie wrote: > Andrew, can you remember why we ended up with both of those locks in the > first place? If we can do it, the efficient way out here is to abandon > the journal_head_lock and use the bh_state_lock for both. We already > hold that over

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Sat, 2005-03-05 at 00:04, Andrew Morton wrote: > Perhaps we could also fix this by elevating b_jcount whenever the jh is > being moved between lists? Possible. But jcount isn't atomic, and it requires the bh_journal_head lock to modify. Taking and dropping the lock twice around the

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 14:50, Jan Kara wrote: > I believe the other places should be safe (mostly by luck) as the > caller has made sure that __journal_remove_journal_head() won't do > anything (e.g. set b_transaction, b_next_transaction or such). Right; I've been looking through all the

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Jan Kara
> "Stephen C. Tweedie" <[EMAIL PROTECTED]> wrote: > > > > For the past few months there has been a slow but steady trickle of > > reports of oopses in kjournald. > > Yes, really tenuous stuff. Very glad if this is the fix! > > > Recently I got a couple of reports that > > were repeatable

Re: [Ext2-devel] [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Fri, 2005-03-04 at 23:17, Badari Pulavarty wrote: > I looked at few journalling bugs recently on RHEL4 testing here. > I am wondering if your patch fixes this following BUG also ? > I never got to bottom of some of these journal panics - > since they are not easily reproducible Right...

Re: [Ext2-devel] [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Fri, 2005-03-04 at 23:17, Badari Pulavarty wrote: I looked at few journalling bugs recently on RHEL4 testing here. I am wondering if your patch fixes this following BUG also ? I never got to bottom of some of these journal panics - since they are not easily reproducible Right... +

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Jan Kara
Stephen C. Tweedie [EMAIL PROTECTED] wrote: For the past few months there has been a slow but steady trickle of reports of oopses in kjournald. Yes, really tenuous stuff. Very glad if this is the fix! Recently I got a couple of reports that were repeatable enough to rerun with

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 14:50, Jan Kara wrote: I believe the other places should be safe (mostly by luck) as the caller has made sure that __journal_remove_journal_head() won't do anything (e.g. set b_transaction, b_next_transaction or such). Right; I've been looking through all the

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 16:40, Stephen C. Tweedie wrote: Andrew, can you remember why we ended up with both of those locks in the first place? If we can do it, the efficient way out here is to abandon the journal_head_lock and use the bh_state_lock for both. We already hold that over

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
Stephen C. Tweedie [EMAIL PROTECTED] wrote: Trouble is, that's not enough; journal_put_journal_head() can nuke the buffer with merely the bh_journal_head lock held. In the code above it would be enough to take the journal_head_lock over the unfile/file pair. Andrew, can you remember why

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 20:31, Andrew Morton wrote: jbd_lock_bh_journal_head() is supposed to be a finegrained innermost lock whose mandate is purely for atomicity of adding and removing the journal_head and the b_jcount refcounting. I don't recall there being any deeper meaning than

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
Stephen C. Tweedie [EMAIL PROTECTED] wrote: I'm having trouble testing it, though --- I seem to be getting livelocks in O_DIRECT running 400 fsstress processes in parallel; ring any bells? Nope. I dont think anyone has been that cruel to ext3 for a while. I assume this workload used to

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 21:11, Andrew Morton wrote: I'm having trouble testing it, though --- I seem to be getting livelocks in O_DIRECT running 400 fsstress processes in parallel; ring any bells? Nope. I dont think anyone has been that cruel to ext3 for a while. I assume this

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
Stephen C. Tweedie [EMAIL PROTECTED] wrote: In invalidate_inode_pages2_range(), what happens if we lookup a pagevec, get a bunch of pages back, but all the pages in the vec are beyond the end of the range we want? hmm, yes. Another one :( @@ -271,12 +271,13 @@ int

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Stephen C. Tweedie
Hi, On Mon, 2005-03-07 at 21:22, Stephen C. Tweedie wrote: altgr-scrlck is showing a range of EIPs all in ext3_direct_IO- invalidate_inode_pages2_range(). I'm seeing invalidate_inode_pages2_range()-pagevec_lookup()-find_get_pages() In invalidate_inode_pages2_range(), what happens if we

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Suparna Bhattacharya
yup, looks like the same issue we hit in wait_on_page_writeback_range during AIO work - probably want to break out of the outer loop as well when this happens. From the old changelog: wait_on_page_writeback_range shouldn't wait for pages beyond the specified range. Ideally, the

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Suparna Bhattacharya
Just as an idea of what I meant (dug up an old WIP patch): --- radix-tree.c2004-04-01 10:32:15.384556136 +0530 +++ radix-tree.c.end2004-04-01 11:11:07.176069944 +0530 @@ -562,7 +562,8 @@ EXPORT_SYMBOL(radix_tree_gang_lookup); */ static unsigned int __lookup_tag(struct

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
Suparna Bhattacharya [EMAIL PROTECTED] wrote: yup, looks like the same issue we hit in wait_on_page_writeback_range during AIO work - probably want to break out of the outer loop as well when this happens. The `next = page_index' before breaking will do that for us. How hard would it

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Suparna Bhattacharya
On Mon, Mar 07, 2005 at 10:46:18PM -0800, Andrew Morton wrote: Suparna Bhattacharya [EMAIL PROTECTED] wrote: yup, looks like the same issue we hit in wait_on_page_writeback_range during AIO work - probably want to break out of the outer loop as well when this happens. The `next =

Re: [Ext2-devel] Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-07 Thread Andrew Morton
Suparna Bhattacharya [EMAIL PROTECTED] wrote: (let me know if the interface in the patch I just posted seems like the right direction to use when we go for the cleanup) Well what are the semantics? Pass in an inclusive max_index and the gang lookup functions terminate when they hit an item

Re: [Ext2-devel] [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-04 Thread Badari Pulavarty
Stephen, I looked at few journalling bugs recently on RHEL4 testing here. I am wondering if your patch fixes this following BUG also ? I never got to bottom of some of these journal panics - since they are not easily reproducible + I don't understand journal code well enough :( Assertion

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-04 Thread Andrew Morton
"Stephen C. Tweedie" <[EMAIL PROTECTED]> wrote: > > For the past few months there has been a slow but steady trickle of > reports of oopses in kjournald. Yes, really tenuous stuff. Very glad if this is the fix! > Recently I got a couple of reports that > were repeatable enough to rerun with

[RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-04 Thread Stephen C. Tweedie
Hi all, For the past few months there has been a slow but steady trickle of reports of oopses in kjournald. Recently I got a couple of reports that were repeatable enough to rerun with extra debugging code. It turns out that we're releasing a journal_head while it is still linked onto the

[RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-04 Thread Stephen C. Tweedie
Hi all, For the past few months there has been a slow but steady trickle of reports of oopses in kjournald. Recently I got a couple of reports that were repeatable enough to rerun with extra debugging code. It turns out that we're releasing a journal_head while it is still linked onto the

Re: [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-04 Thread Andrew Morton
Stephen C. Tweedie [EMAIL PROTECTED] wrote: For the past few months there has been a slow but steady trickle of reports of oopses in kjournald. Yes, really tenuous stuff. Very glad if this is the fix! Recently I got a couple of reports that were repeatable enough to rerun with extra

Re: [Ext2-devel] [RFC] ext3/jbd race: releasing in-use journal_heads

2005-03-04 Thread Badari Pulavarty
Stephen, I looked at few journalling bugs recently on RHEL4 testing here. I am wondering if your patch fixes this following BUG also ? I never got to bottom of some of these journal panics - since they are not easily reproducible + I don't understand journal code well enough :( Assertion