Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-17 Thread Mike Galbraith
On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: > On Mon, 2012-07-16 at 13:03 -0400, Steven Rostedt wrote: > > On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: > > > > > > > > Ouch, you just turned the rt_read_lock() into a spin lock. If a higher > > > > > priority process

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-17 Thread Mike Galbraith
On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: On Mon, 2012-07-16 at 13:03 -0400, Steven Rostedt wrote: On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: Ouch, you just turned the rt_read_lock() into a spin lock. If a higher priority process preempted a lower

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Tue, 2012-07-17 at 00:34 -0400, Steven Rostedt wrote: > On Tue, 2012-07-17 at 00:27 -0400, Steven Rostedt wrote: > > > Actually, I was mistaken. I forgot that we defined 'cpu_chill()' as > > msleep(1) on RT, which would keep a deadlock from happening. > > Perhaps cpu_chill() isn't a good

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Tue, 2012-07-17 at 00:27 -0400, Steven Rostedt wrote: > On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: > > > > > There's that too. But the issue I was talking about is with all trylock > > > loops. As holding an rt-mutex now disables migration, if a high priority > > > process

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: > > > There's that too. But the issue I was talking about is with all trylock > > loops. As holding an rt-mutex now disables migration, if a high priority > > process preempts a task that holds the lock, and then the high prio task > >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 13:03 -0400, Steven Rostedt wrote: > On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: > > > > > > Ouch, you just turned the rt_read_lock() into a spin lock. If a higher > > > > priority process preempted a lower priority process that holds the same > > > > lock, it

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 11:43 -0400, Chris Mason wrote: > On Mon, Jul 16, 2012 at 04:55:44AM -0600, Mike Galbraith wrote: > > Seems btrfs isn't entirely convinced either. > > > > [ 2292.336229] use_block_rsv: 1810 callbacks suppressed > > [ 2292.336231] [ cut here ] > > [

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: > > > > Ouch, you just turned the rt_read_lock() into a spin lock. If a higher > > > priority process preempted a lower priority process that holds the same > > > lock, it will deadlock. > > > > Hm, how, it's doing cpu_chill()? > >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 18:26 +0200, Mike Galbraith wrote: > On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: > > On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: > > > > > > Great, thanks! I got stuck in bug land on Friday. You mentioned > > > > performance problems earlier on

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Chris Mason
On Mon, Jul 16, 2012 at 10:26:08AM -0600, Mike Galbraith wrote: > On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: > > On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: > > > > > > Great, thanks! I got stuck in bug land on Friday. You mentioned > > > > performance problems

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: > On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: > > > > Great, thanks! I got stuck in bug land on Friday. You mentioned > > > performance problems earlier on Saturday, did this improve performance? > > > > Yeah, the

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: > > Great, thanks! I got stuck in bug land on Friday. You mentioned > > performance problems earlier on Saturday, did this improve performance? > > Yeah, the read_trylock() seems to improve throughput. That's not > heavily tested, but

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Chris Mason
On Mon, Jul 16, 2012 at 04:55:44AM -0600, Mike Galbraith wrote: > On Sat, 2012-07-14 at 12:14 +0200, Mike Galbraith wrote: > > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > > Greetings, > > > > > > [ deadlocks

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Sat, 2012-07-14 at 12:14 +0200, Mike Galbraith wrote: > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > Greetings, > > > > [ deadlocks with btrfs and the recent RT kernels ] > > > > I talked with Thomas about

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Sat, 2012-07-14 at 12:14 +0200, Mike Galbraith wrote: On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: Greetings, [ deadlocks with btrfs and the recent RT kernels ] I talked with Thomas about this and I think

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Chris Mason
On Mon, Jul 16, 2012 at 04:55:44AM -0600, Mike Galbraith wrote: On Sat, 2012-07-14 at 12:14 +0200, Mike Galbraith wrote: On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: Greetings, [ deadlocks with btrfs and

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: Great, thanks! I got stuck in bug land on Friday. You mentioned performance problems earlier on Saturday, did this improve performance? Yeah, the read_trylock() seems to improve throughput. That's not heavily tested, but it

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: Great, thanks! I got stuck in bug land on Friday. You mentioned performance problems earlier on Saturday, did this improve performance? Yeah, the read_trylock() seems

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Chris Mason
On Mon, Jul 16, 2012 at 10:26:08AM -0600, Mike Galbraith wrote: On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: Great, thanks! I got stuck in bug land on Friday. You mentioned performance problems earlier on

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 18:26 +0200, Mike Galbraith wrote: On Mon, 2012-07-16 at 12:02 -0400, Steven Rostedt wrote: On Mon, 2012-07-16 at 04:02 +0200, Mike Galbraith wrote: Great, thanks! I got stuck in bug land on Friday. You mentioned performance problems earlier on Saturday, did

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: Ouch, you just turned the rt_read_lock() into a spin lock. If a higher priority process preempted a lower priority process that holds the same lock, it will deadlock. Hm, how, it's doing cpu_chill()? 'course PI is toast,

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 11:43 -0400, Chris Mason wrote: On Mon, Jul 16, 2012 at 04:55:44AM -0600, Mike Galbraith wrote: Seems btrfs isn't entirely convinced either. [ 2292.336229] use_block_rsv: 1810 callbacks suppressed [ 2292.336231] [ cut here ] [

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Mon, 2012-07-16 at 13:03 -0400, Steven Rostedt wrote: On Mon, 2012-07-16 at 18:36 +0200, Mike Galbraith wrote: Ouch, you just turned the rt_read_lock() into a spin lock. If a higher priority process preempted a lower priority process that holds the same lock, it will deadlock.

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Steven Rostedt
On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: There's that too. But the issue I was talking about is with all trylock loops. As holding an rt-mutex now disables migration, if a high priority process preempts a task that holds the lock, and then the high prio task starts

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Tue, 2012-07-17 at 00:27 -0400, Steven Rostedt wrote: On Tue, 2012-07-17 at 06:18 +0200, Mike Galbraith wrote: There's that too. But the issue I was talking about is with all trylock loops. As holding an rt-mutex now disables migration, if a high priority process preempts a task

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-16 Thread Mike Galbraith
On Tue, 2012-07-17 at 00:34 -0400, Steven Rostedt wrote: On Tue, 2012-07-17 at 00:27 -0400, Steven Rostedt wrote: Actually, I was mistaken. I forgot that we defined 'cpu_chill()' as msleep(1) on RT, which would keep a deadlock from happening. Perhaps cpu_chill() isn't a good name, as it

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-15 Thread Mike Galbraith
On Sun, 2012-07-15 at 13:56 -0400, Chris Mason wrote: > On Sat, Jul 14, 2012 at 04:14:43AM -0600, Mike Galbraith wrote: > > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > > Greetings, > > > > > > [ deadlocks with

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-15 Thread Chris Mason
On Sat, Jul 14, 2012 at 04:14:43AM -0600, Mike Galbraith wrote: > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > Greetings, > > > > [ deadlocks with btrfs and the recent RT kernels ] > > > > I talked with Thomas

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-15 Thread Chris Mason
On Sat, Jul 14, 2012 at 04:14:43AM -0600, Mike Galbraith wrote: On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: Greetings, [ deadlocks with btrfs and the recent RT kernels ] I talked with Thomas about this and I

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-15 Thread Mike Galbraith
On Sun, 2012-07-15 at 13:56 -0400, Chris Mason wrote: On Sat, Jul 14, 2012 at 04:14:43AM -0600, Mike Galbraith wrote: On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: Greetings, [ deadlocks with btrfs and the

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-14 Thread Mike Galbraith
On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > There is also a chunk of code in btrfs_clear_path_blocking that makes > sure to strictly honor top down locking order during the conversion. It > only does this when lockdep is enabled because in non-RT kernels we > don't need to worry

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-14 Thread Mike Galbraith
On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > Greetings, > > [ deadlocks with btrfs and the recent RT kernels ] > > I talked with Thomas about this and I think the problem is the > single-reader nature of the RW

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-14 Thread Mike Galbraith
On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: Greetings, [ deadlocks with btrfs and the recent RT kernels ] I talked with Thomas about this and I think the problem is the single-reader nature of the RW rwlocks. The

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-14 Thread Mike Galbraith
On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: There is also a chunk of code in btrfs_clear_path_blocking that makes sure to strictly honor top down locking order during the conversion. It only does this when lockdep is enabled because in non-RT kernels we don't need to worry about

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
Chris, On Fri, 13 Jul 2012, Chris Mason wrote: > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > Greetings, > > [ deadlocks with btrfs and the recent RT kernels ] > > I talked with Thomas about this and I think the problem is the > single-reader nature of the RW rwlocks.

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Chris Mason
On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > Greetings, [ deadlocks with btrfs and the recent RT kernels ] I talked with Thomas about this and I think the problem is the single-reader nature of the RW rwlocks. The lockdep report below mentions that btrfs is calling: > [

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Mike Galbraith
On Fri, 2012-07-13 at 06:47 -0400, Chris Mason wrote: > On Fri, Jul 13, 2012 at 04:26:26AM -0600, Thomas Gleixner wrote: > > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > > On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: > > > > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > > > > On

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Chris Mason
On Fri, Jul 13, 2012 at 04:26:26AM -0600, Thomas Gleixner wrote: > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: > > > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > > >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Fri, 13 Jul 2012, Mike Galbraith wrote: > On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: > > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > > Bingo, that makes it more likely that this is caused by copying w/o > > >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Mike Galbraith
On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: > On Fri, 13 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > Bingo, that makes it more likely that this is caused by copying w/o > > > initializing the lock and then freeing the original

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Chris Mason wrote: > On Thu, Jul 12, 2012 at 05:07:58AM -0600, Thomas Gleixner wrote: > > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > > crash> struct rt_mutex 0x8801770601c8 > > > struct rt_mutex { > > > wait_lock = { > > > raw_lock = { > > > slock = 7966 >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Fri, 13 Jul 2012, Mike Galbraith wrote: > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > Bingo, that makes it more likely that this is caused by copying w/o > > initializing the lock and then freeing the original structure. > > > > A quick check for memcpy finds that

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: > > > rawlock points to ...968 and the node_list to ...970. > > > > > > struct rt_mutex { > > > raw_spinlock_t

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: On Thu, 12 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: rawlock points to ...968 and the node_list to ...970. struct rt_mutex { raw_spinlock_t wait_lock;

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Fri, 13 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: Bingo, that makes it more likely that this is caused by copying w/o initializing the lock and then freeing the original structure. A quick check for memcpy finds that

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Chris Mason wrote: On Thu, Jul 12, 2012 at 05:07:58AM -0600, Thomas Gleixner wrote: On Thu, 12 Jul 2012, Mike Galbraith wrote: crash struct rt_mutex 0x8801770601c8 struct rt_mutex { wait_lock = { raw_lock = { slock = 7966 } },

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Mike Galbraith
On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: On Fri, 13 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: Bingo, that makes it more likely that this is caused by copying w/o initializing the lock and then freeing the original

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
On Fri, 13 Jul 2012, Mike Galbraith wrote: On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: On Fri, 13 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: Bingo, that makes it more likely that this is caused by copying w/o initializing

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Chris Mason
On Fri, Jul 13, 2012 at 04:26:26AM -0600, Thomas Gleixner wrote: On Fri, 13 Jul 2012, Mike Galbraith wrote: On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: On Fri, 13 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: Bingo, that

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Mike Galbraith
On Fri, 2012-07-13 at 06:47 -0400, Chris Mason wrote: On Fri, Jul 13, 2012 at 04:26:26AM -0600, Thomas Gleixner wrote: On Fri, 13 Jul 2012, Mike Galbraith wrote: On Fri, 2012-07-13 at 11:52 +0200, Thomas Gleixner wrote: On Fri, 13 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Chris Mason
On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: Greetings, [ deadlocks with btrfs and the recent RT kernels ] I talked with Thomas about this and I think the problem is the single-reader nature of the RW rwlocks. The lockdep report below mentions that btrfs is calling: [

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-13 Thread Thomas Gleixner
Chris, On Fri, 13 Jul 2012, Chris Mason wrote: On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: Greetings, [ deadlocks with btrfs and the recent RT kernels ] I talked with Thomas about this and I think the problem is the single-reader nature of the RW rwlocks. The

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Chris Mason
On Thu, Jul 12, 2012 at 05:07:58AM -0600, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > crash> struct rt_mutex 0x8801770601c8 > > struct rt_mutex { > > wait_lock = { > > raw_lock = { > > slock = 7966 > > } > > }, > > wait_list = { > >

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:48 +0200, Mike Galbraith wrote: > On Thu, 2012-07-12 at 15:43 +0200, Thomas Gleixner wrote: > > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > > A quick check for memcpy finds that __btrfs_close_devices()

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:43 +0200, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > > A quick check for memcpy finds that __btrfs_close_devices() does a > > > memcpy of btrfs_device structs w/o initializing the

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: > On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > > A quick check for memcpy finds that __btrfs_close_devices() does a > > memcpy of btrfs_device structs w/o initializing the lock in the new > > copy, but I have no idea whether that's the

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: > > > rawlock points to ...968 and the node_list to ...970. > > > > > > struct rt_mutex { > > > raw_spinlock_t

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: > On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: > > rawlock points to ...968 and the node_list to ...970. > > > > struct rt_mutex { > > raw_spinlock_t wait_lock; > > struct plist_head wait_list; > > > > The

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: > On Thu, 12 Jul 2012, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: > > > On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: > > > > Greetings, > > > > > > > > I'm chasing btrfs critters in an

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: > On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: > > On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: > > > Greetings, > > > > > > I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just > > > checked to see if they're

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: > crash> struct rt_mutex 0x8801770601c8 > struct rt_mutex { > wait_lock = { > raw_lock = { > slock = 7966 > } > }, > wait_list = { > node_list = { > next = 0x880175eedbe0, > prev = 0x880175eedbe0 > },

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: > On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: > > Greetings, > > > > I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just > > checked to see if they're alive in virgin latest/greatest rt kernel. > > > > Both

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: > Greetings, > > I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just > checked to see if they're alive in virgin latest/greatest rt kernel. > > Both are indeed alive and well, ie I didn't break it, nor did the > zillion

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: Greetings, I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just checked to see if they're alive in virgin latest/greatest rt kernel. Both are indeed alive and well, ie I didn't break it, nor did the zillion patches

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: Greetings, I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just checked to see if they're alive in virgin latest/greatest rt kernel. Both are indeed

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: crash struct rt_mutex 0x8801770601c8 struct rt_mutex { wait_lock = { raw_lock = { slock = 7966 } }, wait_list = { node_list = { next = 0x880175eedbe0, prev = 0x880175eedbe0 }, rawlock

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: Greetings, I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just checked to see if they're alive in virgin

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: On Thu, 12 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 10:44 +0200, Mike Galbraith wrote: On Thu, 2012-07-12 at 07:47 +0200, Mike Galbraith wrote: Greetings, I'm chasing btrfs critters in an enterprise 3.0-rt

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: rawlock points to ...968 and the node_list to ...970. struct rt_mutex { raw_spinlock_t wait_lock; struct plist_head wait_list; The raw_lock pointer

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: On Thu, 12 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 13:43 +0200, Thomas Gleixner wrote: rawlock points to ...968 and the node_list to ...970. struct rt_mutex { raw_spinlock_t wait_lock;

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Thomas Gleixner
On Thu, 12 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: A quick check for memcpy finds that __btrfs_close_devices() does a memcpy of btrfs_device structs w/o initializing the lock in the new copy, but I have no idea whether that's the place we

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:43 +0200, Thomas Gleixner wrote: On Thu, 12 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: A quick check for memcpy finds that __btrfs_close_devices() does a memcpy of btrfs_device structs w/o initializing the lock in

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Mike Galbraith
On Thu, 2012-07-12 at 15:48 +0200, Mike Galbraith wrote: On Thu, 2012-07-12 at 15:43 +0200, Thomas Gleixner wrote: On Thu, 12 Jul 2012, Mike Galbraith wrote: On Thu, 2012-07-12 at 15:31 +0200, Thomas Gleixner wrote: A quick check for memcpy finds that __btrfs_close_devices() does a

Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-12 Thread Chris Mason
On Thu, Jul 12, 2012 at 05:07:58AM -0600, Thomas Gleixner wrote: On Thu, 12 Jul 2012, Mike Galbraith wrote: crash struct rt_mutex 0x8801770601c8 struct rt_mutex { wait_lock = { raw_lock = { slock = 7966 } }, wait_list = { node_list = { next =

3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-11 Thread Mike Galbraith
Greetings, I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just checked to see if they're alive in virgin latest/greatest rt kernel. Both are indeed alive and well, ie I didn't break it, nor did the zillion patches in enterprise base kernel, so others may have an opportunity to

3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free!

2012-07-11 Thread Mike Galbraith
Greetings, I'm chasing btrfs critters in an enterprise 3.0-rt kernel, and just checked to see if they're alive in virgin latest/greatest rt kernel. Both are indeed alive and well, ie I didn't break it, nor did the zillion patches in enterprise base kernel, so others may have an opportunity to