Two issues. softlockup and oops in jbd. Safe to say the issue is between jbd and ocfs2. I'll need more info to proceed.
Do: $ objdump -DSl /lib/modules/`uname -r`/kernel/fs/jbd/jbd.ko >/tmp/jbd.out $ ./stat_sysdir -d device >/tmp/sysdir.out http://oss.oracle.com/~smushran/.debug/scripts/stat_sysdir.sh File a bugzilla (oss.oracle.com/bugzilla) and attach the above outputs. Sunil Alexandre Racine wrote: > > Hi Sunil, > > I just want to check something with you. Can you see with this crash > report if it is the kernel fault or ocfs2 fault? My server just > rebooted and I want to know who or what rebooted. Thanks. > > $ uname -a > > Linux PETER 2.6.23-gentoo-r8 #3 SMP Tue Mar 25 02:07:57 EDT 2008 > x86_64 Intel(R) Xeon(TM) CPU 3.40GHz GenuineIntel GNU/Linux > > Version of ocfs2, 1.39 here > http://bugs.gentoo.org/show_bug.cgi?id=193249#c25 > > Thanks, Dump below… > > ------------------------- > > Oct 6 13:04:59 PETER kernel BUG at fs/jbd/transaction.c:1161! > > Oct 6 13:04:59 PETER invalid opcode: 0000 [1] SMP > > Oct 6 13:04:59 PETER CPU 1 > > Oct 6 13:04:59 PETER Modules linked in: ocfs2_dlmfs ocfs2 ocfs2_dlm > ocfs2_nodemanager configfs iscsi_tcp libiscsi scsi_transport_iscsi > > Oct 6 13:04:59 PETER Pid: 9282, comm: unknown Not tainted > 2.6.23-gentoo-r8 #3 > > Oct 6 13:04:59 PETER RIP: 0010:[<ffffffff802fc2cf>] > [<ffffffff802fc2cf>] journal_dirty_metadata+0x143/0x1be > > Oct 6 13:04:59 PETER RSP: 0018:ffff81040289b918 EFLAGS: 00010292 > > Oct 6 13:04:59 PETER RAX: 0000000000000077 RBX: ffff8102c1300a88 RCX: > 0000000000000046 > > Oct 6 13:04:59 PETER RDX: 0000000000000005 RSI: 0000000000000096 RDI: > ffffffff807acd60 > > Oct 6 13:04:59 PETER RBP: ffff810134afd550 R08: ffffffff807acd68 R09: > 00000000ffffffff > > Oct 6 13:04:59 PETER R10: ffff81040289b88e R11: 0000000000000000 R12: > ffff810781b9a648 > > Oct 6 13:04:59 PETER R13: ffff8107db3cba00 R14: ffff8101dc44f1c0 R15: > ffff81040289ba48 > > Oct 6 13:04:59 PETER FS: 0000000000000000(0000) > GS:ffff8107e24d3a40(0063) knlGS:000000003fe436c0 > > Oct 6 13:04:59 PETER CS: 0010 DS: 002b ES: 002b CR0: 000000008005003b > > Oct 6 13:04:59 PETER CR2: 000000003f383000 CR3: 00000004e8786000 CR4: > 00000000000006e0 > > Oct 6 13:04:59 PETER DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > Oct 6 13:04:59 PETER DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > Oct 6 13:04:59 PETER Process unknown (pid: 9282, threadinfo > ffff81040289a000, task ffff81041e364790) > > Oct 6 13:04:59 PETER Stack: ffff810361ad6d30 ffff8102c1300a88 > ffff810781b9a648 ffff810016cb8000 > > Oct 6 13:04:59 PETER ffff810361ad6d30 ffffffff880aa2f3 > ffff810016cb8000 ffff8101dc44f3c0 > > Oct 6 13:04:59 PETER ffff8101dc44f1c0 ffffffff8808a73b > ffff8100215a3ec0 ffff81040289ba28 > > Oct 6 13:04:59 PETER Call Trace: > > Oct 6 13:04:59 PETER [<ffffffff880aa2f3>] > :ocfs2:ocfs2_journal_dirty+0x6a/0x11d > > Oct 6 13:04:59 PETER [<ffffffff8808a73b>] > :ocfs2:ocfs2_do_insert_extent+0x905/0xb43 > > Oct 6 13:04:59 PETER [<ffffffff8808ecd3>] > :ocfs2:ocfs2_insert_extent+0x600/0x70d > > Oct 6 13:04:59 PETER [<ffffffff8809ec09>] > :ocfs2:ocfs2_do_extend_allocation+0x370/0x54a > > Oct 6 13:04:59 PETER [<ffffffff880907eb>] > :ocfs2:ocfs2_write_begin_nolock+0x96b/0x1222 > > Oct 6 13:04:59 PETER [<ffffffff880928a9>] > :ocfs2:ocfs2_write_begin+0x1a9/0x274 > > Oct 6 13:04:59 PETER [<ffffffff880a41a4>] > :ocfs2:ocfs2_file_aio_write+0x65f/0xa2f > > Oct 6 13:04:59 PETER [<ffffffff880a77f6>] > :ocfs2:ocfs2_clear_inode+0x848/0x994 > > Oct 6 13:04:59 PETER [<ffffffff8027f908>] do_sync_write+0xc9/0x10c > > Oct 6 13:04:59 PETER [<ffffffff880a858c>] > :ocfs2:ocfs2_delete_inode+0x6b5/0x721 > > Oct 6 13:04:59 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:04:59 PETER [<ffffffff80288f48>] do_unlinkat+0xef/0x14b > > Oct 6 13:04:59 PETER [<ffffffff80280048>] vfs_write+0xad/0x136 > > Oct 6 13:04:59 PETER [<ffffffff80280585>] sys_write+0x45/0x6e > > Oct 6 13:04:59 PETER [<ffffffff80224e70>] sysenter_do_call+0x1b/0x67 > > Oct 6 13:04:59 PETER > > Oct 6 13:04:59 PETER Code: 0f 0b eb fe 48 83 7d 18 00 74 2c 49 c7 c0 > eb 8e 71 80 b9 90 > > Oct 6 13:04:59 PETER RIP [<ffffffff802fc2cf>] > journal_dirty_metadata+0x143/0x1be > > Oct 6 13:04:59 PETER RSP <ffff81040289b918> > > Oct 6 13:05:09 PETER BUG: soft lockup - CPU#0 stuck for 11s! > [kjournald:6988] > > Oct 6 13:05:09 PETER CPU 0: > > Oct 6 13:05:09 PETER Modules linked in: ocfs2_dlmfs ocfs2 ocfs2_dlm > ocfs2_nodemanager configfs iscsi_tcp libiscsi scsi_transport_iscsi > > Oct 6 13:05:09 PETER Pid: 6988, comm: kjournald Tainted: G D > 2.6.23-gentoo-r8 #3 > > Oct 6 13:05:09 PETER RIP: 0010:[<ffffffff80300f63>] > [<ffffffff80300f63>] journal_write_metadata_buffer+0x68/0x319 > > Oct 6 13:05:09 PETER RSP: 0018:ffff8107d76f5de0 EFLAGS: 00000206 > > Oct 6 13:05:09 PETER RAX: 0000000000398021 RBX: ffff8107db3cba00 RCX: > 0000000000000cb6 > > Oct 6 13:05:09 PETER RDX: 0000000000000cb7 RSI: ffffffff808c3480 RDI: > ffff8102b19f9338 > > Oct 6 13:05:09 PETER RBP: 0000000000000000 R08: ffff8107dadb1598 R09: > ffff8107dadb15a0 > > Oct 6 13:05:09 PETER R10: ffff8102bff05c28 R11: ffffffff88092288 R12: > 0000000000008287 > > Oct 6 13:05:09 PETER R13: 0000000000008287 R14: ffff8107dadb15b0 R15: > 0000000000000000 > > Oct 6 13:05:09 PETER FS: 0000000000000000(0000) > GS:ffffffff8081c000(0000) knlGS:0000000000000000 > > Oct 6 13:05:09 PETER CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Oct 6 13:05:09 PETER CR2: 00002aadb762d9d0 CR3: 00000004777bc000 CR4: > 00000000000006e0 > > Oct 6 13:05:09 PETER DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > Oct 6 13:05:09 PETER DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > Oct 6 13:05:09 PETER > > Oct 6 13:05:09 PETER Call Trace: > > Oct 6 13:05:09 PETER [<ffffffff8030010d>] journal_bmap+0x22/0x7a > > Oct 6 13:05:09 PETER [<ffffffff802fcf83>] > journal_commit_transaction+0x795/0x1016 > > Oct 6 13:05:09 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:09 PETER [<ffffffff8030066f>] kjournald+0xb9/0x212 > > Oct 6 13:05:09 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:09 PETER [<ffffffff803005b6>] kjournald+0x0/0x212 > > Oct 6 13:05:09 PETER [<ffffffff80247908>] kthread+0x47/0x73 > > Oct 6 13:05:09 PETER [<ffffffff8020c188>] child_rip+0xa/0x12 > > Oct 6 13:05:09 PETER [<ffffffff80443d4b>] tg3_start_xmit_dma_bug+0x0/0x74b > > Oct 6 13:05:09 PETER [<ffffffff802478c1>] kthread+0x0/0x73 > > Oct 6 13:05:09 PETER [<ffffffff8020c17e>] child_rip+0x0/0x12 > > Oct 6 13:05:21 PETER BUG: soft lockup - CPU#0 stuck for 11s! > [kjournald:6988] > > Oct 6 13:05:21 PETER CPU 0: > > Oct 6 13:05:21 PETER Modules linked in: ocfs2_dlmfs ocfs2 ocfs2_dlm > ocfs2_nodemanager configfs iscsi_tcp libiscsi scsi_transport_iscsi > > Oct 6 13:05:21 PETER Pid: 6988, comm: kjournald Tainted: G D > 2.6.23-gentoo-r8 #3 > > Oct 6 13:05:21 PETER RIP: 0010:[<ffffffff80300f63>] > [<ffffffff80300f63>] journal_write_metadata_buffer+0x68/0x319 > > Oct 6 13:05:21 PETER RSP: 0018:ffff8107d76f5de0 EFLAGS: 00000206 > > Oct 6 13:05:21 PETER RAX: 0000000000398021 RBX: ffff8107db3cba00 RCX: > 0000000000000cb6 > > Oct 6 13:05:21 PETER RDX: 0000000000000cb7 RSI: ffffffff808c3480 RDI: > ffff8102b19f9338 > > Oct 6 13:05:21 PETER RBP: 0000000000000000 R08: ffff8107dadb1598 R09: > ffff8107dadb15a0 > > Oct 6 13:05:21 PETER R10: ffff8102bff05c28 R11: ffffffff88092288 R12: > 0000000000008287 > > Oct 6 13:05:21 PETER R13: 0000000000008287 R14: ffff8107dadb15b0 R15: > 0000000000000000 > > Oct 6 13:05:21 PETER FS: 0000000000000000(0000) > GS:ffffffff8081c000(0000) knlGS:0000000000000000 > > Oct 6 13:05:21 PETER CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Oct 6 13:05:21 PETER CR2: 00002aadb762d9d0 CR3: 00000004777bc000 CR4: > 00000000000006e0 > > Oct 6 13:05:21 PETER DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > Oct 6 13:05:21 PETER DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > Oct 6 13:05:21 PETER > > Oct 6 13:05:21 PETER Call Trace: > > Oct 6 13:05:21 PETER [<ffffffff8030010d>] journal_bmap+0x22/0x7a > > Oct 6 13:05:21 PETER [<ffffffff802fcf83>] > journal_commit_transaction+0x795/0x1016 > > Oct 6 13:05:21 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:21 PETER [<ffffffff8030066f>] kjournald+0xb9/0x212 > > Oct 6 13:05:21 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:21 PETER [<ffffffff803005b6>] kjournald+0x0/0x212 > > Oct 6 13:05:21 PETER [<ffffffff80247908>] kthread+0x47/0x73 > > Oct 6 13:05:21 PETER [<ffffffff8020c188>] child_rip+0xa/0x12 > > Oct 6 13:05:21 PETER [<ffffffff80443d4b>] tg3_start_xmit_dma_bug+0x0/0x74b > > Oct 6 13:05:21 PETER [<ffffffff802478c1>] kthread+0x0/0x73 > > Oct 6 13:05:21 PETER [<ffffffff8020c17e>] child_rip+0x0/0x12 > > Oct 6 13:05:21 PETER > > Oct 6 13:05:32 PETER BUG: soft lockup - CPU#0 stuck for 11s! > [kjournald:6988] > > Oct 6 13:05:32 PETER CPU 0: > > Oct 6 13:05:32 PETER Modules linked in: ocfs2_dlmfs ocfs2 ocfs2_dlm > ocfs2_nodemanager configfs iscsi_tcp libiscsi scsi_transport_iscsi > > Oct 6 13:05:32 PETER Pid: 6988, comm: kjournald Tainted: G D > 2.6.23-gentoo-r8 #3 > > Oct 6 13:05:32 PETER RIP: 0010:[<ffffffff80300f63>] > [<ffffffff80300f63>] journal_write_metadata_buffer+0x68/0x319 > > Oct 6 13:05:32 PETER RSP: 0018:ffff8107d76f5de0 EFLAGS: 00000206 > > Oct 6 13:05:32 PETER RAX: 0000000000398021 RBX: ffff8107db3cba00 RCX: > 0000000000000cb6 > > Oct 6 13:05:32 PETER RDX: 0000000000000cb7 RSI: ffffffff808c3480 RDI: > ffff8102b19f9338 > > Oct 6 13:05:32 PETER RBP: 0000000000000000 R08: ffff8107dadb1598 R09: > ffff8107dadb15a0 > > Oct 6 13:05:32 PETER R10: ffff8102bff05c28 R11: ffffffff88092288 R12: > 0000000000008287 > > Oct 6 13:05:32 PETER R13: 0000000000008287 R14: ffff8107dadb15b0 R15: > 0000000000000000 > > Oct 6 13:05:32 PETER FS: 0000000000000000(0000) > GS:ffffffff8081c000(0000) knlGS:0000000000000000 > > Oct 6 13:05:32 PETER CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Oct 6 13:05:32 PETER CR2: 00002aadb762d9d0 CR3: 00000004777bc000 CR4: > 00000000000006e0 > > Oct 6 13:05:32 PETER DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > Oct 6 13:05:32 PETER DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > Oct 6 13:05:32 PETER Call Trace: > > Oct 6 13:05:32 PETER [<ffffffff8030010d>] journal_bmap+0x22/0x7a > > Oct 6 13:05:32 PETER [<ffffffff802fcf83>] > journal_commit_transaction+0x795/0x1016 > > Oct 6 13:05:32 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:32 PETER [<ffffffff8030066f>] kjournald+0xb9/0x212 > > Oct 6 13:05:32 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:32 PETER [<ffffffff803005b6>] kjournald+0x0/0x212 > > Oct 6 13:05:32 PETER [<ffffffff80247908>] kthread+0x47/0x73 > > Oct 6 13:05:32 PETER [<ffffffff8020c188>] child_rip+0xa/0x12 > > Oct 6 13:05:32 PETER [<ffffffff80443d4b>] tg3_start_xmit_dma_bug+0x0/0x74b > > Oct 6 13:05:32 PETER [<ffffffff802478c1>] kthread+0x0/0x73 > > Oct 6 13:05:32 PETER [<ffffffff8020c17e>] child_rip+0x0/0x12 > > Oct 6 13:05:32 PETER > > Oct 6 13:05:44 PETER BUG: soft lockup - CPU#0 stuck for 11s! > [kjournald:6988] > > Oct 6 13:05:44 PETER CPU 0: > > Oct 6 13:05:44 PETER Modules linked in: ocfs2_dlmfs ocfs2 ocfs2_dlm > ocfs2_nodemanager configfs iscsi_tcp libiscsi scsi_transport_iscsi > > Oct 6 13:05:44 PETER Pid: 6988, comm: kjournald Tainted: G D > 2.6.23-gentoo-r8 #3 > > Oct 6 13:05:44 PETER RIP: 0010:[<ffffffff80300f63>] > [<ffffffff80300f63>] journal_write_metadata_buffer+0x68/0x319 > > Oct 6 13:05:44 PETER RSP: 0018:ffff8107d76f5de0 EFLAGS: 00000206 > > Oct 6 13:05:44 PETER RAX: 0000000000398021 RBX: ffff8107db3cba00 RCX: > 0000000000000cb6 > > Oct 6 13:05:44 PETER RDX: 0000000000000cb7 RSI: ffffffff808c3480 RDI: > ffff8102b19f9338 > > Oct 6 13:05:44 PETER RBP: 0000000000000000 R08: ffff8107dadb1598 R09: > ffff8107dadb15a0 > > Oct 6 13:05:44 PETER R10: ffff8102bff05c28 R11: ffffffff88092288 R12: > 0000000000008287 > > Oct 6 13:05:44 PETER R13: 0000000000008287 R14: ffff8107dadb15b0 R15: > 0000000000000000 > > Oct 6 13:05:44 PETER FS: 0000000000000000(0000) > GS:ffffffff8081c000(0000) knlGS:0000000000000000 > > Oct 6 13:05:44 PETER CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Oct 6 13:05:44 PETER CR2: 00002aadb762d9d0 CR3: 00000004777bc000 CR4: > 00000000000006e0 > > Oct 6 13:05:44 PETER DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > Oct 6 13:05:44 PETER DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > Oct 6 13:05:44 PETER > > Oct 6 13:05:44 PETER Call Trace: > > Oct 6 13:05:44 PETER [<ffffffff8030010d>] journal_bmap+0x22/0x7a > > Oct 6 13:05:44 PETER [<ffffffff802fcf83>] > journal_commit_transaction+0x795/0x1016 > > Oct 6 13:05:44 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:44 PETER [<ffffffff8030066f>] kjournald+0xb9/0x212 > > Oct 6 13:05:44 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:44 PETER [<ffffffff803005b6>] kjournald+0x0/0x212 > > Oct 6 13:05:44 PETER [<ffffffff80247908>] kthread+0x47/0x73 > > Oct 6 13:05:44 PETER [<ffffffff8020c188>] child_rip+0xa/0x12 > > Oct 6 13:05:44 PETER [<ffffffff80443d4b>] tg3_start_xmit_dma_bug+0x0/0x74b > > Oct 6 13:05:44 PETER [<ffffffff802478c1>] kthread+0x0/0x73 > > Oct 6 13:05:44 PETER [<ffffffff8020c17e>] child_rip+0x0/0x12 > > Oct 6 13:05:56 PETER BUG: soft lockup - CPU#0 stuck for 11s! > [kjournald:6988] > > Oct 6 13:05:56 PETER CPU 0: > > Oct 6 13:05:56 PETER Modules linked in: ocfs2_dlmfs ocfs2 ocfs2_dlm > ocfs2_nodemanager configfs iscsi_tcp libiscsi scsi_transport_iscsi > > Oct 6 13:05:56 PETER Pid: 6988, comm: kjournald Tainted: G D > 2.6.23-gentoo-r8 #3 > > Oct 6 13:05:56 PETER RIP: 0010:[<ffffffff80300f63>] > [<ffffffff80300f63>] journal_write_metadata_buffer+0x68/0x319 > > Oct 6 13:05:56 PETER RSP: 0018:ffff8107d76f5de0 EFLAGS: 00000206 > > Oct 6 13:05:56 PETER RAX: 0000000000398021 RBX: ffff8107db3cba00 RCX: > 0000000000000cb6 > > Oct 6 13:05:56 PETER RDX: 0000000000000cb7 RSI: ffffffff808c3480 RDI: > ffff8102b19f9338 > > Oct 6 13:05:56 PETER RBP: 0000000000000000 R08: ffff8107dadb1598 R09: > ffff8107dadb15a0 > > Oct 6 13:05:56 PETER R10: ffff8102bff05c28 R11: ffffffff88092288 R12: > 0000000000008287 > > Oct 6 13:05:56 PETER R13: 0000000000008287 R14: ffff8107dadb15b0 R15: > 0000000000000000 > > Oct 6 13:05:56 PETER FS: 0000000000000000(0000) > GS:ffffffff8081c000(0000) knlGS:0000000000000000 > > Oct 6 13:05:56 PETER CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > > Oct 6 13:05:56 PETER CR2: 00002aadb762d9d0 CR3: 00000004777bc000 CR4: > 00000000000006e0 > > Oct 6 13:05:56 PETER DR0: 0000000000000000 DR1: 0000000000000000 DR2: > 0000000000000000 > > Oct 6 13:05:56 PETER DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: > 0000000000000400 > > Oct 6 13:05:56 PETER > > Oct 6 13:05:56 PETER Call Trace: > > Oct 6 13:05:56 PETER [<ffffffff8030010d>] journal_bmap+0x22/0x7a > > Oct 6 13:05:56 PETER [<ffffffff802fcf83>] > journal_commit_transaction+0x795/0x1016 > > Oct 6 13:05:56 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:56 PETER [<ffffffff8030066f>] kjournald+0xb9/0x212 > > Oct 6 13:05:56 PETER [<ffffffff80247a26>] > autoremove_wake_function+0x0/0x2e > > Oct 6 13:05:56 PETER [<ffffffff803005b6>] kjournald+0x0/0x212 > > Oct 6 13:05:56 PETER [<ffffffff80247908>] kthread+0x47/0x73 > > Oct 6 13:05:56 PETER [<ffffffff8020c188>] child_rip+0xa/0x12 > > Oct 6 13:05:56 PETER [<ffffffff80443d4b>] tg3_start_xmit_dma_bug+0x0/0x74b > > Oct 6 13:05:56 PETER [<ffffffff802478c1>] kthread+0x0/0x73 > > Oct 6 13:05:56 PETER [<ffffffff8020c17e>] child_rip+0x0/0x12 > > Alexandre Racine > > [EMAIL PROTECTED] > > 514-461-1300 poste 3304 > > ------------------------------------------------------------------------ > > _______________________________________________ > Ocfs2-users mailing list > [email protected] > http://oss.oracle.com/mailman/listinfo/ocfs2-users _______________________________________________ Ocfs2-users mailing list [email protected] http://oss.oracle.com/mailman/listinfo/ocfs2-users
