Re:
http://lkml.org/lkml/2007/1/28/146
Just got a similar OOPS on a system under heavy load (transcode + p2p),
with kernel 2.6.20.3 / x86_64 (in free_block).
[<ffffffff802c5b95>] free_block+0xae/0x13c
[<ffffffff802c5cb6>] drain_array+0x93/0xd1
[<ffffffff802c6820>] cache_reap+0xea/0x239
[<ffffffff802c6736>] cache_reap+0x0/0x239
[<ffffffff8024a644>] run_workqueue+0x95/0x140
[<ffffffff802471e5>] worker_thread+0x0/0x150
[<ffffffff802472ff>] worker_thread+0x11a/0x150
[<ffffffff80284806>] default_wake_function+0x0/0xe
[<ffffffff802319e7>] kthread+0xd1/0x100
[<ffffffff8025aec8>] child_rip+0xa/0x12
[<ffffffff80231916>] kthread+0x0/0x100
[<ffffffff8025aebe>] child_rip+0x0/0x12
Not sure this is relevant, but transcode uses nice/ionice:
ionice -n 7 -c 3 nice -n 19 transcode [...]
No forced pre-emption, 100HZ, etc...
Note: this is not an SMP system, just a SMP kernel. So if it is a race,
it is not an SMP triggered race. (or I could just be unlucky again -
damn those cosmic rays)
Antoine
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: list_del corruption. next->prev
should be ffff8100007c6000, but was ffffffffffffffff
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: ------------[ cut here
]------------
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: kernel BUG at
lib/list_debug.c:72!
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: invalid opcode: 0000 [1] SMP
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: CPU 0
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: Modules linked in: ipt_LOG
ipt_MASQUERADE iptable_nat nf_nat ip6table_filter ip6_tables iptable_filter
ip_tables pegasus autofs4 hidp rfcomm l2cap bluetooth sunrpc
nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack
nfnetlink ip6t_REJECT xt_tcpudp x_tables ipv6 dm_mirror dm_multipath dm_mod
raid0 video thermal sbs processor i2c_ec dock button battery asus_acpi
backlight ac lp bt878 pata_via snd_hda_intel snd_hda_codec snd_bt87x
snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss
snd_mixer_oss snd_pcm snd_timer snd soundcore ide_cd sg shpchp parport_pc
snd_page_alloc pata_ali i2c_viapro sata_uli via_rhine mii cdrom parport tuner
tvaudio bttv k8temp hwmon video_buf pcspkr ir_common compat_ioctl32
i2c_algo_bit btcx_risc tveeprom i2c_core videodev v4l2_common serio_raw
v4l1_compat sata_sil ata_generic sata_via libata sd_mod scsi_mod raid456 xor
raid1 ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: Pid: 5, comm: events/0 Not
tainted 2.6.20.3 #5
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RIP: 0010:[<ffffffff803332ae>]
[<ffffffff803332ae>] list_del+0x3f/0x5b
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RSP: 0018:ffff81000148ddb0
EFLAGS: 00010082
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RAX: 0000000000000058 RBX:
ffff8100007c6000 RCX: ffffffff8051b718
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RDX: ffffffff8051b718 RSI:
0000000000000082 RDI: ffffffff8051b700
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RBP: ffff81000140ff40 R08:
ffffffff8051b718 R09: 0000000000000010
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: R10: 0000000000000000 R11:
0000000000000000 R12: ffff8100007c6f00
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: R13: ffff810001420580 R14:
ffff810001425218 R15: 000000000000003e
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: FS: 0000000041001940(0000)
GS:ffffffff8055f000(0000) knlGS:00000000f7f328e0
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: CS: 0010 DS: 0018 ES: 0018 CR0:
000000008005003b
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: CR2: 00002b7cf8af2000 CR3:
000000000532c000 CR4: 00000000000006e0
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: Process events/0 (pid: 5,
threadinfo ffff81000148c000, task ffff810001476820)
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: Stack: ffff8100066cf000
ffffffff802c5b95 0000006000000000 ffff810001425028
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: 0000000000000060
ffff810001425000 ffff81000140ff80 0000000000000000
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: ffff810001420580
ffffffff802c5cb6 00000001ffffff86 0000000000000000
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: Call Trace:
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff802c5b95>]
free_block+0xae/0x13c
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff802c5cb6>]
drain_array+0x93/0xd1
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff802c6820>]
cache_reap+0xea/0x239
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff802c6736>]
cache_reap+0x0/0x239
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff8024a644>]
run_workqueue+0x95/0x140
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff802471e5>]
worker_thread+0x0/0x150
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff802472ff>]
worker_thread+0x11a/0x150
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff80284806>]
default_wake_function+0x0/0xe
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff802319e7>]
kthread+0xd1/0x100
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff8025aec8>]
child_rip+0xa/0x12
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff80231916>]
kthread+0x0/0x100
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff8025aebe>]
child_rip+0x0/0x12
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel:
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel:
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: Code: 0f 0b eb fe 48 89 48 08 48
89 01 48 c7 47 08 00 02 20 00 48
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RIP [<ffffffff803332ae>]
list_del+0x3f/0x5b
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RSP <ffff81000148ddb0>
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: NMI Watchdog detected LOCKUP on
CPU 0
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: CPU 0
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: Modules linked in: ipt_LOG
ipt_MASQUERADE iptable_nat nf_nat ip6table_filter ip6_tables iptable_filter
ip_tables pegasus autofs4 hidp rfcomm l2cap bluetooth sunrpc
nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 xt_state nf_conntrack
nfnetlink ip6t_REJECT xt_tcpudp x_tables ipv6 dm_mirror dm_multipath dm_mod
raid0 video thermal sbs processor i2c_ec dock button battery asus_acpi
backlight ac lp bt878 pata_via snd_hda_intel snd_hda_codec snd_bt87x
snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss
snd_mixer_oss snd_pcm snd_timer snd soundcore ide_cd sg shpchp parport_pc
snd_page_alloc pata_ali i2c_viapro sata_uli via_rhine mii cdrom parport tuner
tvaudio bttv k8temp hwmon video_buf pcspkr ir_common compat_ioctl32
i2c_algo_bit btcx_risc tveeprom i2c_core videodev v4l2_common serio_raw
v4l1_compat sata_sil ata_generic sata_via libata sd_mod scsi_mod raid456 xor
raid1 ext3 jbd ehci_hcd ohci_hcd uhci_hcd
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: Pid: 211, comm: kswapd0 Not
tainted 2.6.20.3 #5
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RIP: 0010:[<ffffffff80207756>]
[<ffffffff80207756>] _raw_spin_lock+0x76/0xf3
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RSP: 0000:ffff81000fe1db10
EFLAGS: 00000097
Mar 21 06:15:27 x1-6-00-18-f3-f6-fa-fb kernel: RAX: 0000000000000008 RBX:
ffff81000140ff80 RCX: 0000000027ec466b
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: RDX: 000000000000288f RSI:
ffffffff804c5865 RDI: 0000000000000001
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: RBP: 0000000019ba8b10 R08:
ffff8100056d9698 R09: ffff81000140fe03
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: R10: ffff8100056d9698 R11:
ffffffff88032aba R12: 0000000000000001
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: R13: 0000000077020c98 R14:
ffff810001420580 R15: ffff81000140ff80
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: FS: 00002ba96ff4b600(0000)
GS:ffffffff8055f000(0000) knlGS:00000000f7f328e0
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: CS: 0010 DS: 0018 ES: 0018 CR0:
000000008005003b
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: CR2: 00002aeee011afd0 CR3:
000000000532c000 CR4: 00000000000006e0
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: Process kswapd0 (pid: 211,
threadinfo ffff81000fe1c000, task ffff81000fc97040)
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: Stack: ffff81000140fde0
ffff81000140ff40 000000000000003c ffff810001421400
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: 0000000000000000
ffffffff802c5d28 ffff81000fe1dea0 ffff810001421400
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: 0000000000000000
ffff81000c15df68 ffff810001420580 0000000000000246
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: Call Trace:
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff802c5d28>]
cache_flusharray+0x34/0xae
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff80207678>]
kmem_cache_free+0x1e6/0x203
Mar 21 06:15:28 x1-6-00-18-f3-f6-fa-fb kernel: [<ffffffff80226799>]
free_buffer_head+0x24/0x3d
