Yesterday evening, I had a series of kernel BUG events logged. They didn't actually _crash_ my workstation but I rebooted it anyway. Last night at 02:27:09 another BUG halted the machine. Attached file BUGS.txt summarizes the BUGs logged. Attached files messages.*.txt have the full messages.
I presume it'd be useful to report these upstream. What's the proper method for doing so? Regards, David
Jan 13 21:35:04 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:36:10 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:40:32 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:50:21 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:55:49 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:59:05 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:01:16 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:11:06 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:13:17 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:15:03 osage kernel BUG at mm/rmap.c:725! Jan 13 22:20:55 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:25:02 osage kernel BUG at mm/rmap.c:725! Jan 14 02:26:21 osage BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
Jan 13 20:59:06 osage 0 akff80045d)<>tc:<>ff80db63 ff20002000000002 ffff061d<>00000000 ff8014e8ffffffff 00000000<>ff80db63 ff8014e0000003cd 00000000<>alTae Jan 13 20:59:06 osage 0 <ffff061d]?umpva+0x400/0x706 Jan 13 20:59:06 osage [<ffffffff80265013>] ? exit_mmap+0xa2/0x119 Jan 13 20:59:06 osage [<ffffffff8022dd2e>] ? mmput+0x2c/0xa2 Jan 13 20:59:06 osage [<ffffffff80231148>] ? exit_mm+0xee/0xf9 Jan 13 20:59:06 osage [<ffffffff80232505>] ? do_exit+0x1b2/0x728 Jan 13 20:59:06 osage [<ffffffff80232ae1>] ? do_group_exit+0x66/0x96 Jan 13 20:59:06 osage [<ffff000b]?sse_alfspt+x601 Jan 13 20:59:06 osage 0Cd:8 82 df f4 b8 00 00 88 07 94 b4 04 5c 41 88 05 8c 76 94 0e 25 ef 0>0 bf b7 85 b5 3e 1f e8 60 91 4 Jan 13 20:59:06 osage 1RP <ffff065d]pg_eoera+xf01a<>RP<ff8014d8 Jan 13 20:59:06 osage 4-- n rc 7d053000]- Jan 13 20:59:06 osage 1Fxn eusv al u eoti edd Jan 13 21:00:01 osage cron[12841]: (root) CMD (rm -f /var/spool/cron/lastrun/cron.hourly) Jan 13 21:00:01 osage cron[12843]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:00:01 osage cron[12839]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 21:00:01 osage sSMTP[12855]: Sent mail for hea...@osage (221 Bye) Jan 13 21:05:01 osage cron[12861]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:05:04 osage sSMTP[12954]: Sent mail for hea...@osage (221 Bye) Jan 13 21:10:01 osage cron[12958]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:10:01 osage cron[12956]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 21:15:01 osage cron[13062]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:15:01 osage sSMTP[13064]: Sent mail for hea...@osage (221 Bye) Jan 13 21:20:01 osage cron[13068]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:20:01 osage cron[13066]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 21:20:01 osage 0D2 00000000<>R:00000000 R:0000ff0f R:00000000 Jan 13 21:20:01 osage 4Poessnmi pd 38,traif ff80fa00 akff800f69)<>tc:<>00000000 ff80fac8ffffffff 0000000000000 Jan 13 21:20:01 osage ffff8800a61f9138 ffff88008fca3c60 00000000002b0296 0000000000000000 Jan 13 21:20:01 osage 0000000101424080 00007fc1b814a000 ffff8800115147f8 ffff88001625a830 Jan 13 21:20:01 osage Call Trace: Jan 13 21:20:01 osage [<ffffffff80265013>] ? exit_mmap+0xa2/0x119 Jan 13 21:20:01 osage [<fffffff82d2> mu+xc0a Jan 13 21:20:01 osage 0 <ffff0318]?ei_m0e/x9<>[ffff8220> oei+xb/x2 Jan 13 21:20:01 osage 0 <ffff03f7]?_dqeesga+xe01c<>[ffff822e> ogopei+x609 Jan 13 21:20:01 osage 0 <ffff03fc]?gtsga_odlvr02502f<>[ffff82aa> ontf_eue08/xa Jan 13 21:20:01 osage 0 <ffff020f]?ts_qlc+x007 Jan 13 21:20:01 osage 0 <ffff0214]?tyt_aeu+x6/x7 Jan 13 21:20:01 osage 0 <ffff03f5]?sga_aeu+x002 Jan 13 21:20:01 osage 0 <ffff034b]?sn_inl01f01b<>[ffff82c9> oivldo+x508 Jan 13 21:20:01 osage 0 <ffff078f]?vsra+x1/x3 Jan 13 21:20:01 osage 0 <ffff006d]?rtn_inl03/x0<>oe 5c 42 c8 04 b4 84 b5 04 1e c4 38 80 00 83 07 c4 1e c4 3c 04 95 0<6 31 17 6f c2 8e 48 58 5f f0 04 <>I [ffff821f> na_ms033076<>RP<ff80fab8 Jan 13 21:25:01 osage cron[13082]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:25:01 osage sSMTP[13084]: Sent mail for hea...@osage (221 Bye) Jan 13 21:27:58 osage 0C2 ff204f38<>-[edtae14d65c3c --<>iigrcriefutbtrbo snee!<6>emacs[1587] general protection ip:7fbe08b5d419 sp:7fff14e12740 error:0 in libc-2.6.1.so[7fbe08ae8000+136000] Jan 13 21:27:58 osage f7c8 R14: 00007fbe01e00000 R15: ffff88002802d340 Jan 13 21:27:58 osage FS: 00007fbe0ccdb6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 21:27:58 osage CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jan 13 21:27:58 osage CR2: 00007fbe08af71a0 CR3: 0000000011785000 CR4: 00000000000006e0 Jan 13 21:27:58 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 21:27:58 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 21:27:58 osage Process emacs (pid: 1587, threadinfo ffff88000bb40000, task ffff8800060405d0) Jan 13 21:27:58 osage Stack: Jan 13 21:27:58 osage ffff88011eaa6848 ffffe200018b2650 0000000000000000 ffffffff8026112d Jan 13 21:27:58 osage 0000000000000000 ffff88000bb41c58 ffffffffffffffff 0000000000000000 Jan 13 21:27:58 osage ffff88008a100720 ffff88000bb41c60 0000000000210000 0000000000000000 Jan 13 21:27:58 osage Call Trace: Jan 13 21:27:58 osage [<ffffffff8026112d>] ? unmap_vmas+0x400/0x706 Jan 13 21:27:58 osage [<ffffffff80265013>] ? exit_mmap+0xa2/0x119 Jan 13 21:27:58 osage [<ffffffff8022dd2e>] ? mmput+0x2c/0xa2 Jan 13 21:27:58 osage [<ffffffff80231148>] ? exit_mm+0xee/0xf9 Jan 13 21:27:58 osage [<ffffffff80232505>] ? do_exit+0x1b2/0x728 Jan 13 21:27:58 osage [<ffffffff80238fa7>] ? __dequeue_signal+0xee/0x11c Jan 13 21:27:58 osage [<ffffffff80232ae1>] ? do_group_exit+0x66/0x96 Jan 13 21:27:58 osage [<ffffffff8023af3c>] ? gtsga_odlvr02502f<>[ffff82aa> ontf_eue08/xa Jan 13 21:27:58 osage 0 <ffff020f]?ts_qlc+x007 Jan 13 21:27:58 osage 0 <ffff03f5]?sga_aeu+x002 Jan 13 21:27:58 osage 0 <ffff034b]?sn_inl01f01b<>[ffff82b9> eitsga+xd08 Jan 13 21:27:58 osage 0Cd:8 82 df f4 b8 00 00 88 07 94 b4 04 5c 41 88 05 8c 76 94 0e 25 ef 0>0 bf b7 85 b5 3e 1f e8 60 91 4 Jan 13 21:27:58 osage 1RP <ffff065d]pg_eoera+xf01a<>RP<ff80b4b8 Jan 13 21:27:58 osage 4-- n rc 7d053000]- Jan 13 21:27:58 osage 1Fxn eusv al u eoti edd Jan 13 21:30:01 osage cron[13090]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 21:30:01 osage cron[13091]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:35:01 osage cron[13122]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:35:02 osage sSMTP[13124]: Sent mail for hea...@osage (221 Bye) Jan 13 21:35:04 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:35:04 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:35:04 osage CPU 1: Jan 13 21:35:04 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:35:04 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 21:35:04 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 21:35:04 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 21:35:04 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 21:35:04 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 21:35:04 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 21:35:04 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 21:35:04 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 21:35:04 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 21:35:04 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 21:35:04 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 21:35:04 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 21:35:04 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 21:35:04 osage Call Trace: Jan 13 21:35:04 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 21:35:04 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 21:35:04 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 21:35:04 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 21:35:04 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 21:35:04 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 21:35:04 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 21:35:04 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 21:35:04 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 21:36:10 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:36:10 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:36:10 osage CPU 1: Jan 13 21:36:10 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:36:10 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 21:36:10 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 21:36:10 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 21:36:10 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 21:36:10 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 21:36:10 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 21:36:10 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 21:36:10 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 21:36:10 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 21:36:10 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 21:36:10 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 21:36:10 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 21:36:10 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 21:36:10 osage Call Trace: Jan 13 21:36:10 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 21:36:10 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 21:36:10 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 21:36:10 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 21:36:10 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 21:36:10 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 21:36:10 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 21:36:10 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 21:36:10 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 21:40:01 osage cron[13126]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 21:40:01 osage cron[13131]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:40:03 osage rrdtool[13227]: segfault at 7fd12668c8c6 ip 00007fd136d7ee20 sp 00007fff3ef84558 error 4 in ld-2.6.1.so[7fd136d6b000+1b000] Jan 13 21:40:03 osage system_health.p[13132]: segfault at 34fff43a81 ip 00007f9aa8d90f28 sp 00007fffb12480c0 error 4 in libpython2.6.so.1.0[7f9aa8c99000+14b000] Jan 13 21:40:03 osage sSMTP[13231]: Sent mail for hea...@osage (221 Bye) Jan 13 21:40:32 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:40:32 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:40:32 osage CPU 1: Jan 13 21:40:32 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:40:32 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 21:40:32 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 21:40:32 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 21:40:32 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 21:40:32 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 21:40:32 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 21:40:32 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 21:40:32 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 21:40:32 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 21:40:32 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 21:40:32 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 21:40:32 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 21:40:32 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 21:40:32 osage Call Trace: Jan 13 21:40:32 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 21:40:32 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 21:40:32 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 21:40:32 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 21:40:32 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 21:40:32 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 21:40:32 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 21:40:32 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 21:40:32 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 21:45:01 osage cron[13233]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:45:02 osage rrdtool[13271]: segfault at 7fa4af06c230 ip 00007fa4af06c230 sp 00007ffff7ef48e8 error 14 in libart_lgpl_2.so.2.3.20[7fa4eebed000+17000] Jan 13 21:50:01 osage cron[13327]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 21:50:01 osage cron[13332]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:50:02 osage sSMTP[13336]: Sent mail for hea...@osage (221 Bye) Jan 13 21:50:21 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:50:21 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:50:21 osage CPU 1: Jan 13 21:50:21 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:50:21 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 21:50:21 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 21:50:21 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 21:50:21 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 21:50:21 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 21:50:21 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 21:50:21 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 21:50:21 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 21:50:21 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 21:50:21 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 21:50:21 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 21:50:21 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 21:50:21 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 21:50:21 osage Call Trace: Jan 13 21:50:21 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 21:50:21 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 21:50:21 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 21:50:21 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 21:50:21 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 21:50:21 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 21:50:21 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 21:50:21 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 21:50:21 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 21:55:01 osage cron[13343]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 21:55:01 osage sSMTP[13345]: Sent mail for hea...@osage (221 Bye) Jan 13 21:55:49 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:55:49 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:55:49 osage CPU 1: Jan 13 21:55:49 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:55:49 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 21:55:49 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 21:55:49 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 21:55:49 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 21:55:49 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 21:55:49 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 21:55:49 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 21:55:49 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 21:55:49 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 21:55:49 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 21:55:49 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 21:55:49 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 21:55:49 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 21:55:49 osage Call Trace: Jan 13 21:55:49 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 21:55:49 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 21:55:49 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 21:55:49 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 21:55:49 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 21:55:49 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 21:55:49 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 21:55:49 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 21:55:49 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 21:58:01 osage cron[13347]: (root) CMD (/usr/sbin/ntpdate -s nic) Jan 13 21:58:01 osage ntpdate[13347]: adjust time server 192.168.1.2 offset -0.047710 sec Jan 13 21:59:05 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 21:59:05 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:59:05 osage CPU 1: Jan 13 21:59:05 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 21:59:05 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 21:59:05 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 21:59:05 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 21:59:05 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 21:59:05 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 21:59:05 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 21:59:05 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 21:59:05 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 21:59:05 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 21:59:05 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 21:59:05 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 21:59:05 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 21:59:05 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 21:59:05 osage Call Trace: Jan 13 21:59:05 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 21:59:05 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 21:59:05 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 21:59:05 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 21:59:05 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 21:59:05 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 21:59:05 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 21:59:05 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 21:59:05 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 22:00:01 osage cron[13352]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 22:00:01 osage cron[13357]: (root) CMD (rm -f /var/spool/cron/lastrun/cron.hourly) Jan 13 22:00:01 osage cron[13359]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 22:00:01 osage system_health.p[13360]: segfault at 7f34d08eb758 ip 00007f34d25fe418 sp 00007fffdaaebda0 error 4 in libpython2.6.so.1.0[7f34d253e000+14b000] Jan 13 22:00:01 osage sSMTP[13367]: Sent mail for hea...@osage (221 Bye) Jan 13 22:00:38 osage su[10577]: pam_unix(su:session): session closed for user root Jan 13 22:01:16 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:01:16 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:01:16 osage CPU 1: Jan 13 22:01:16 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:01:16 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 22:01:16 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 22:01:16 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 22:01:16 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 22:01:16 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 22:01:16 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 22:01:16 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 22:01:16 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 22:01:16 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 22:01:16 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 22:01:16 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 22:01:16 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 22:01:16 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 22:01:16 osage Call Trace: Jan 13 22:01:16 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 22:01:16 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 22:01:16 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 22:01:16 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 22:01:16 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 22:01:16 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 22:01:16 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 22:01:16 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 22:01:16 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 22:05:01 osage cron[13371]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 22:10:01 osage cron[13466]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 22:10:01 osage cron[13470]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 22:10:01 osage sSMTP[13475]: Sent mail for hea...@osage (221 Bye) Jan 13 22:11:06 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:11:06 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:11:06 osage CPU 1: Jan 13 22:11:06 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:11:06 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 22:11:06 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 22:11:06 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 22:11:06 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 22:11:06 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 22:11:06 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 22:11:06 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 22:11:06 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 22:11:06 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 22:11:06 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 22:11:06 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 22:11:06 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 22:11:06 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 22:11:06 osage Call Trace: Jan 13 22:11:06 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 22:11:06 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 22:11:06 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 22:11:06 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 22:11:06 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 22:11:06 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 22:11:06 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 22:11:06 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 22:11:06 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 22:13:17 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:13:17 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:13:17 osage CPU 1: Jan 13 22:13:17 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:13:17 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 22:13:17 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 22:13:17 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 22:13:17 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 22:13:17 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 22:13:17 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 22:13:17 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 22:13:17 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 22:13:17 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 22:13:17 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 22:13:17 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 22:13:17 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 22:13:17 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 22:13:17 osage Call Trace: Jan 13 22:13:17 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 22:13:17 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 22:13:17 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 22:13:17 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 22:13:17 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 22:13:17 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 22:13:17 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 22:13:17 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 22:13:17 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 22:15:01 osage cron[13482]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 22:15:03 osage Eeek! page_mapcount(page) went negative! (-1) Jan 13 22:15:03 osage page pfn = 5739 Jan 13 22:15:03 osage page->flags = 400000000004001c Jan 13 22:15:03 osage page->count = 0 Jan 13 22:15:03 osage page->mapping = 0000000000000000 Jan 13 22:15:03 osage vma->vm_ops = 0x0 Jan 13 22:15:03 osage ------------[ cut here ]------------ Jan 13 22:15:03 osage kernel BUG at mm/rmap.c:725! Jan 13 22:15:03 osage invalid opcode: 0000 [#6] SMP Jan 13 22:15:03 osage last sysfs file: /sys/devices/pci0000:00/0000:00:14.4/0000:03:05.2/usb2/manufacturer Jan 13 22:15:03 osage CPU 0 Jan 13 22:15:03 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:15:03 osage Pid: 13534, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 22:15:03 osage RIP: 0010:[<ffffffff8026852d>] [<ffffffff8026852d>] page_remove_rmap+0xff/0x11a Jan 13 22:15:03 osage RSP: 0018:ffff880001599dc8 EFLAGS: 00010246 Jan 13 22:15:03 osage RAX: 0000000000000000 RBX: ffffe20000131478 RCX: 000000000001c456 Jan 13 22:15:03 osage RDX: ffff8800a7a6d000 RSI: 0000000000000046 RDI: ffffffff805d6cf4 Jan 13 22:15:03 osage RBP: ffff880016b25330 R08: ffff8800155236c8 R09: 0000000000000000 Jan 13 22:15:03 osage R10: 00000000ffffffff R11: 0000000000000000 R12: 00007faddb4d9000 Jan 13 22:15:03 osage R13: ffff8800155236c8 R14: 00007faddb57c000 R15: ffff880028022340 Jan 13 22:15:03 osage FS: 00007faddb57c6f0(0000) GS:ffffffff80557980(0000) knlGS:00000000f7139b20 Jan 13 22:15:03 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 22:15:03 osage CR2: 0000000000651000 CR3: 00000000bbf29000 CR4: 00000000000006e0 Jan 13 22:15:03 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 22:15:03 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 22:15:03 osage Process rrdtool (pid: 13534, threadinfo ffff880001598000, task ffff88000158e110) Jan 13 22:15:03 osage Stack: Jan 13 22:15:03 osage ffffe200004b1c90 ffffe20000131478 0000000000000020 ffffffff8026112d Jan 13 22:15:03 osage 0000000000000000 ffff880001599ec0 00007faddb57c000 00007faddb4b6000 Jan 13 22:15:03 osage ffff880016b25330 ffff880001599ec8 00000000003dc000 0000000000000000 Jan 13 22:15:03 osage Call Trace: Jan 13 22:15:03 osage [<ffffffff8026112d>] ? unmap_vmas+0x400/0x706 Jan 13 22:15:03 osage [<ffffffff80264ea5>] ? unmap_region+0xb3/0x126 Jan 13 22:15:03 osage [<ffffffff80265a6c>] ? do_munmap+0x246/0x2b0 Jan 13 22:15:03 osage [<ffffffff8042c8b1>] ? __down_write_nested+0x12/0x93 Jan 13 22:15:03 osage [<ffffffff802668b3>] ? sys_munmap+0x3b/0x55 Jan 13 22:15:03 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 22:15:03 osage Code: 80 e8 27 5d fe ff 48 8b 85 90 00 00 00 48 85 c0 74 19 48 8b 40 20 48 85 c0 74 10 48 8b 70 58 48 c7 c7 66 f9 4a 80 e8 02 5d fe ff <0f> 0b eb fe 8b 77 18 5a 5b 5d 83 e6 01 f7 de 83 c6 08 e9 16 64 Jan 13 22:15:03 osage RIP [<ffffffff8026852d>] page_remove_rmap+0xff/0x11a Jan 13 22:15:03 osage RSP <ffff880001599dc8> Jan 13 22:15:03 osage ---[ end trace 174dd06553c030c0 ]--- Jan 13 22:20:01 osage cron[13536]: (root) CMD (test -x /usr/sbin/run-crons && /usr/sbin/run-crons ) Jan 13 22:20:01 osage cron[13541]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 22:20:02 osage sSMTP[13550]: Sent mail for hea...@osage (221 Bye) Jan 13 22:20:55 osage BUG: soft lockup - CPU#1 stuck for 61s! [rrdtool:13119] Jan 13 22:20:55 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:20:55 osage CPU 1: Jan 13 22:20:55 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:20:55 osage Pid: 13119, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 22:20:55 osage RIP: 0010:[<ffffffff80334455>] [<ffffffff80334455>] __read_lock_failed+0x5/0x14 Jan 13 22:20:55 osage RSP: 0018:ffff88010735fdb8 EFLAGS: 00000297 Jan 13 22:20:55 osage RAX: ffff8800114a32d0 RBX: ffff88010735fe68 RCX: ffff88010735fe68 Jan 13 22:20:55 osage RDX: 0000000000000101 RSI: ffff880109ea8000 RDI: ffff88010f87cc44 Jan 13 22:20:55 osage RBP: ffff88010735fe68 R08: 0000000000008001 R09: 0000000000000000 Jan 13 22:20:55 osage R10: 0000000000000000 R11: 0000000000000246 R12: ffffffff80257602 Jan 13 22:20:55 osage R13: 000000000000001f R14: 0000000000000002 R15: 000000000000001e Jan 13 22:20:55 osage FS: 00007f441185b6f0(0000) GS:ffff88011f878ac0(0000) knlGS:00000000f426eb90 Jan 13 22:20:55 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 22:20:55 osage CR2: 0000000000635000 CR3: 000000011a1cb000 CR4: 00000000000006e0 Jan 13 22:20:55 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 22:20:55 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 22:20:55 osage Call Trace: Jan 13 22:20:55 osage [<ffffffff8042cb25>] ? _read_lock+0xb/0xc Jan 13 22:20:55 osage [<ffffffff8027ddc4>] ? do_path_lookup+0x3f/0x121 Jan 13 22:20:55 osage [<ffffffff8027e96c>] ? path_lookup_open+0x54/0x95 Jan 13 22:20:55 osage [<ffffffff8027ebbb>] ? do_filp_open+0x9d/0x782 Jan 13 22:20:55 osage [<ffffffff8026349f>] free_pgtables+0x87/0x99 Jan 13 22:20:55 osage [<ffffffff80264ed7>] ? unmap_region+0xe5/0x126 Jan 13 22:20:55 osage [<ffffffff80285f5c>] ? alloc_fd+0x69/0x10c Jan 13 22:20:55 osage [<ffffffff802739c9>] ? do_sys_open+0x48/0xcc Jan 13 22:20:55 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 22:25:01 osage cron[13554]: (health) CMD (/home/health/bin/system_health.py --log --graph > /dev/null 2>&1) Jan 13 22:25:02 osage Eeek! page_mapcount(page) went negative! (-2) Jan 13 22:25:02 osage page pfn = 2db7a Jan 13 22:25:02 osage page->flags = 4000000000000864 Jan 13 22:25:02 osage page->count = 2 Jan 13 22:25:02 osage page->mapping = ffff88011f4028c0 Jan 13 22:25:02 osage vma->vm_ops = 0xffffffff80516c00 Jan 13 22:25:02 osage vma->vm_ops->fault = filemap_fault+0x0/0x347 Jan 13 22:25:02 osage vma->vm_file->f_op->mmap = generic_file_mmap+0x0/0x47 Jan 13 22:25:02 osage ------------[ cut here ]------------ Jan 13 22:25:02 osage kernel BUG at mm/rmap.c:725! Jan 13 22:25:02 osage invalid opcode: 0000 [#7] SMP Jan 13 22:25:02 osage last sysfs file: /sys/devices/pci0000:00/0000:00:14.4/0000:03:05.2/usb2/manufacturer Jan 13 22:25:02 osage CPU 0 Jan 13 22:25:02 osage Modules linked in: fan r8169 thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 13 22:25:02 osage Pid: 13618, comm: rrdtool Tainted: G D W 2.6.28-gentoo #11 Jan 13 22:25:02 osage RIP: 0010:[<ffffffff8026852d>] [<ffffffff8026852d>] page_remove_rmap+0xff/0x11a Jan 13 22:25:02 osage RSP: 0018:ffff88000167bd88 EFLAGS: 00010292 Jan 13 22:25:02 osage RAX: 000000000000003b RBX: ffffe20000a002b0 RCX: 0000000000020a8f Jan 13 22:25:02 osage RDX: ffff8800a7a6d000 RSI: 0000000000000046 RDI: ffffffff805d6cf4 Jan 13 22:25:02 osage RBP: ffff8800841f4678 R08: ffff88001324b568 R09: 0000000000000000 Jan 13 22:25:02 osage R10: 00000000ffffffff R11: 0000000000000000 R12: 00007fd4ec6ad000 Jan 13 22:25:02 osage R13: ffff88001324b568 R14: 00007fd4ec7db000 R15: ffff880028022340 Jan 13 22:25:02 osage FS: 00007fd4ed2b96f0(0000) GS:ffffffff80557980(0000) knlGS:00000000f7139b20 Jan 13 22:25:02 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 13 22:25:02 osage CR2: 00007fd4ed0dc000 CR3: 000000001709a000 CR4: 00000000000006e0 Jan 13 22:25:02 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 13 22:25:02 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 13 22:25:02 osage Process rrdtool (pid: 13618, threadinfo ffff88000167a000, task ffff8800016726d0) Jan 13 22:25:02 osage Stack: Jan 13 22:25:02 osage ffff8800193bfd08 ffffe20000a002b0 0000000000000020 ffffffff8026112d Jan 13 22:25:02 osage 0000000000000000 ffff88000167be78 ffffffffffffffff 0000000000000000 Jan 13 22:25:02 osage ffff8800841f4678 ffff88000167be80 000000000038b979 0000000000000000 Jan 13 22:25:02 osage Call Trace: Jan 13 22:25:02 osage [<ffffffff8026112d>] ? unmap_vmas+0x400/0x706 Jan 13 22:25:02 osage [<ffffffff80265013>] ? exit_mmap+0xa2/0x119 Jan 13 22:25:02 osage [<ffffffff8022dd2e>] ? mmput+0x2c/0xa2 Jan 13 22:25:02 osage [<ffffffff80231148>] ? exit_mm+0xee/0xf9 Jan 13 22:25:02 osage [<ffffffff80232505>] ? do_exit+0x1b2/0x728 Jan 13 22:25:02 osage [<ffffffff80232ae1>] ? do_group_exit+0x66/0x96 Jan 13 22:25:02 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 13 22:25:02 osage Code: 80 e8 27 5d fe ff 48 8b 85 90 00 00 00 48 85 c0 74 19 48 8b 40 20 48 85 c0 74 10 48 8b 70 58 48 c7 c7 66 f9 4a 80 e8 02 5d fe ff <0f> 0b eb fe 8b 77 18 5a 5b 5d 83 e6 01 f7 de 83 c6 08 e9 16 64 Jan 13 22:25:02 osage RIP [<ffffffff8026852d>] page_remove_rmap+0xff/0x11a Jan 13 22:25:02 osage RSP <ffff88000167bd88> Jan 13 22:25:02 osage ---[ end trace 174dd06553c030c0 ]--- Jan 13 22:25:02 osage Fixing recursive fault but reboot is needed!
Jan 14 02:26:01 osage /usr/bin/crontab[14032]: (root) LIST (nobody) Jan 14 02:26:20 osage sSMTP[15408]: Sent mail for r...@osage (221 Bye) Jan 14 02:26:20 osage /usr/bin/crontab[15459]: (root) LIST (nobody) Jan 14 02:26:21 osage BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 Jan 14 02:26:21 osage IP: [<ffffffff80282e9f>] __d_lookup+0xd8/0xf9 Jan 14 02:26:21 osage PGD d7b66067 PUD d7b9e067 PMD 0 Jan 14 02:26:21 osage Oops: 0000 [#1] SMP Jan 14 02:26:21 osage last sysfs file: /sys/devices/pci0000:00/0000:00:12.0/host3/target3:0:0/3:0:0:0/model Jan 14 02:26:21 osage CPU 0 Jan 14 02:26:21 osage Modules linked in: r8169 fan thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 14 02:26:21 osage Pid: 15520, comm: chkrootkit Not tainted 2.6.28-gentoo #11 Jan 14 02:26:21 osage RIP: 0010:[<ffffffff80282e9f>] [<ffffffff80282e9f>] __d_lookup+0xd8/0xf9 Jan 14 02:26:21 osage RSP: 0018:ffff8800c243bd18 EFLAGS: 00010202 Jan 14 02:26:21 osage RAX: 0000000000000010 RBX: fffffffffffffff8 RCX: 0000000000000013 Jan 14 02:26:21 osage RDX: 000013de8de1ff3b RSI: 00c3832e8bc182c7 RDI: ffff88011f4019e0 Jan 14 02:26:21 osage RBP: 0000000000000010 R08: 0000000000000000 R09: ffff8800c243ba00 Jan 14 02:26:21 osage R10: 0000000000000000 R11: 0000000000000002 R12: ffff88011f4019e0 Jan 14 02:26:21 osage R13: ffff8800c243bd98 R14: 00000000085f05c9 R15: 0000000000000005 Jan 14 02:26:21 osage FS: 00007f255a2476f0(0000) GS:ffffffff80557980(0000) knlGS:00000000f67bab90 Jan 14 02:26:21 osage CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Jan 14 02:26:21 osage CR2: 0000000000000010 CR3: 00000000c2530000 CR4: 00000000000006e0 Jan 14 02:26:21 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 14 02:26:21 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 14 02:26:21 osage Process chkrootkit (pid: 15520, threadinfo ffff8800c243a000, task ffff880106520b90) Jan 14 02:26:21 osage Stack: Jan 14 02:26:21 osage ffff8800c243bda8 0000000000001d42 ffff8800c243bd98 ffff88011f4019e0 Jan 14 02:26:21 osage ffff88011fabd6f0 ffff88011fabd6c0 ffff8800c243bda8 ffffffff80282ee6 Jan 14 02:26:21 osage ffff8800c243bd98 0000000000003ca1 ffff88011f82db80 ffffffff802acc34 Jan 14 02:26:21 osage Call Trace: Jan 14 02:26:21 osage [<ffffffff80282ee6>] ? d_lookup+0x26/0x38 Jan 14 02:26:21 osage [<ffffffff802acc34>] ? proc_flush_task+0x14e/0x26e Jan 14 02:26:21 osage [<ffffffff80229c79>] ? __dequeue_entity+0x1c/0x32 Jan 14 02:26:21 osage [<ffffffff80231487>] ? release_task+0x20/0x2de Jan 14 02:26:21 osage [<ffffffff80231bb7>] ? wait_consider_task+0x472/0x7c6 Jan 14 02:26:21 osage [<ffffffff8023200a>] ? do_wait+0xff/0x314 Jan 14 02:26:21 osage [<ffffffff8022a1c6>] ? default_wake_function+0x0/0xe Jan 14 02:26:21 osage [<ffffffff802322a9>] ? sys_wait4+0x8a/0xa1 Jan 14 02:26:21 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 14 02:26:21 osage Code: 48 8b 34 24 44 89 fa e8 5c f4 0a 00 85 c0 75 0b f0 ff 03 fe 43 08 48 89 d8 eb 24 fe 43 08 48 8b 6d 00 48 85 ed 74 16 48 8d 5d e8 <48> 8b 45 00 44 39 73 30 0f 18 08 75 e6 e9 7a ff ff ff 31 c0 41 Jan 14 02:26:21 osage RIP [<ffffffff80282e9f>] __d_lookup+0xd8/0xf9 Jan 14 02:26:21 osage RSP <ffff8800c243bd18> Jan 14 02:26:21 osage CR2: 0000000000000010 Jan 14 02:26:21 osage ---[ end trace 98b4fc7bcb1d49a6 ]--- Jan 14 02:26:21 osage BUG: unable to handle kernel NULL pointer dereference at 0000000000000010 Jan 14 02:26:21 osage IP: [<ffffffff80282e9f>] __d_lookup+0xd8/0xf9 Jan 14 02:26:21 osage PGD d791f067 PUD c2581067 PMD 0 Jan 14 02:26:21 osage Oops: 0000 [#2] SMP Jan 14 02:26:21 osage last sysfs file: /sys/devices/pci0000:00/0000:00:12.0/host3/target3:0:0/3:0:0:0/model Jan 14 02:26:21 osage CPU 0 Jan 14 02:26:21 osage Modules linked in: r8169 fan thermal processor mii button ohci_hcd uhci_hcd usb_storage ehci_hcd usbcore sg ahci scsi_wait_scan libata Jan 14 02:26:21 osage Pid: 15629, comm: ps Tainted: G D 2.6.28-gentoo #11 Jan 14 02:26:21 osage RIP: 0010:[<ffffffff80282e9f>] [<ffffffff80282e9f>] __d_lookup+0xd8/0xf9 Jan 14 02:26:21 osage RSP: 0018:ffff8800c25e3d98 EFLAGS: 00010202 Jan 14 02:26:21 osage RAX: 0000000000000010 RBX: fffffffffffffff8 RCX: 0000000000000013 Jan 14 02:26:21 osage RDX: 000013de8de1ff3b RSI: 00c3832e8bc182c7 RDI: ffff88011f4019e0 Jan 14 02:26:21 osage RBP: 0000000000000010 R08: 0000000000000005 R09: ffffffff802ac6cb Jan 14 02:26:21 osage R10: 0000000000000000 R11: 0000000000000002 R12: ffff88011f4019e0 Jan 14 02:26:21 osage R13: ffff8800c25e3e18 R14: 00000000085f05c9 R15: 0000000000000005 Jan 14 02:26:21 osage FS: 00007f2851f356f0(0000) GS:ffffffff80557980(0000) knlGS:00000000f713eb20 Jan 14 02:26:21 osage CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Jan 14 02:26:21 osage CR2: 0000000000000010 CR3: 00000000d78d7000 CR4: 00000000000006e0 Jan 14 02:26:21 osage DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 14 02:26:21 osage DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 14 02:26:21 osage Process ps (pid: 15629, threadinfo ffff8800c25e2000, task ffff880106520b90) Jan 14 02:26:21 osage Stack: Jan 14 02:26:21 osage ffff8800c25e3ea8 0000000000001d42 ffff8800c25e3e18 ffff88011f4019e0 Jan 14 02:26:21 osage ffff88011f4019e0 0000000000000005 ffff8800c25e3ea8 ffffffff80282ee6 Jan 14 02:26:21 osage ffff8800c25e3ea8 ffff8800c25e3e18 ffffffff802ac6cb ffffffff802ab5ef Jan 14 02:26:21 osage Call Trace: Jan 14 02:26:21 osage [<ffffffff80282ee6>] ? d_lookup+0x26/0x38 Jan 14 02:26:21 osage [<ffffffff802ac6cb>] ? proc_pid_instantiate+0x0/0x99 Jan 14 02:26:21 osage [<ffffffff802ab5ef>] ? proc_fill_cache+0x73/0x137 Jan 14 02:26:21 osage [<ffffffff80280440>] ? filldir+0x0/0xb1 Jan 14 02:26:21 osage [<ffffffff802ade29>] ? proc_pid_readdir+0x118/0x19a Jan 14 02:26:21 osage [<ffffffff80280440>] ? filldir+0x0/0xb1 Jan 14 02:26:21 osage [<ffffffff80280440>] ? filldir+0x0/0xb1 Jan 14 02:26:21 osage [<ffffffff80280556>] ? vfs_readdir+0x65/0x97 Jan 14 02:26:21 osage [<ffffffff802807b5>] ? sys_getdents+0x75/0xbc Jan 14 02:26:21 osage [<ffffffff8020b0cb>] ? system_call_fastpath+0x16/0x1b Jan 14 02:26:21 osage Code: 48 8b 34 24 44 89 fa e8 5c f4 0a 00 85 c0 75 0b f0 ff 03 fe 43 08 48 89 d8 eb 24 fe 43 08 48 8b 6d 00 48 85 ed 74 16 48 8d 5d e8 <48> 8b 45 00 44 39 73 30 0f 18 08 75 e6 e9 7a ff ff ff 31 c0 41 Jan 14 02:26:21 osage RIP [<ffffffff80282e9f>] __d_lookup+0xd8/0xf9 Jan 14 02:26:21 osage RSP <ffff8800c25e3d98> Jan 14 02:26:21 osage CR2: 0000000000000010 Jan 14 02:26:21 osage ---[ end trace 98b4fc7bcb1d49a6 ]---
