[Bug 999755] Re: Kernel crash in rb_next doing ohai loops
We think that this seems to be an interplay between the kernel bug, ohai, and Ruby 1.9.3. I was unable to reproduce the crash using the opscode omnibus installer (http://www.opscode.com/blog/2012/06/29 /omnibus-chef-packaging/) which uses a bundled ruby 1.9.2, so you could potentially try that as an alternative to patching OHAI. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/999755 Title: Kernel crash in rb_next doing ohai loops To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/999755/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 999755] Re: Kernel crash on EC2 m1.large instances
I've just reproduced this crash using the stock 3.2.0-24-39 kernel on VirtualBox on OS X (Lion). I created a 2-CPU VM using the latest VirtualBox (4.1.16 r78094), for Ubuntu 64-bit, default 8GB disk. The steps I followed were: * Install 64-bit 12.04 Server LTS, minimal install from ISO downloaded from ubuntu.com/download/server * apt-get dist-upgrade and reboot * apt-get install ruby1.9.3 screen * gem install chef * In a screen session: * while true; do ohai; done After a while (less than an hour) I had reproduced the stack trace from this bug. I added console=ttyS0 to the startup parameters and had VirtualBox log the serial output to a file so I could capture the stack trace (below). This means this bug is nothing to do with Xen, and hopefully can be reproduced by developers more easily. Stefan: If you could build a regular kernel I would be happy to try and capture more info for you. Thanks, Gavin. [ 1119.141743] BUG: unable to handle kernel NULL pointer dereference at 0010 [ 1119.145307] IP: [81311c81] rb_next+0x1/0x50 [ 1119.145307] PGD 1be80067 PUD 1f17d067 PMD 0 [ 1119.145307] Oops: [#1] SMP [ 1119.145307] CPU 0 [ 1119.145307] Modules linked in: vesafb ext2 snd_intel8x0 psmouse snd_ac97_codec ac97_bus snd_pcm snd_timer ppdev snd soundcore joydev snd_page_alloc serio_raw i2c_piix4 parport_pc mac_hid lp parport usbhid hid e1000 [ 1119.145307] [ 1119.145307] Pid: 6768, comm: ohai Not tainted 3.2.0-24-generic #39-Ubuntu innotek GmbH VirtualBox [ 1119.145307] RIP: 0010:[81311c81] [81311c81] rb_next+0x1/0x50 [ 1119.145307] RSP: 0018:88001bd8dc18 EFLAGS: 00010046 [ 1119.145307] RAX: RBX: 88001f232400 RCX: [ 1119.145307] RDX: fff0 RSI: RDI: 0010 [ 1119.145307] RBP: 88001bd8dc48 R08: 0001 R09: [ 1119.145307] R10: 88001fd13780 R11: 0001 R12: [ 1119.145307] R13: R14: R15: 015010f8 [ 1119.145307] FS: 7f6a721d7700() GS:88001fc0() knlGS: [ 1119.145307] CS: 0010 DS: ES: CR0: 80050033 [ 1119.145307] CR2: 0010 CR3: 1ce72000 CR4: 06f0 [ 1119.145307] DR0: DR1: DR2: [ 1119.145307] DR3: DR6: 0ff0 DR7: 0400 [ 1119.145307] Process ohai (pid: 6768, threadinfo 88001bd8c000, task 88001cafdbc0) [ 1119.145307] Stack: [ 1119.145307] 88001bd8dc48 8104ff39 88001f232400 88001fc13780 [ 1119.145307] 88001f233a00 88001bd8dc78 810561d8 [ 1119.145307] 88001bd8dc78 88001fc13780 88001cafdf80 [ 1119.145307] Call Trace: [ 1119.145307] [8104ff39] ? pick_next_entity+0xb9/0xe0 [ 1119.145307] [810561d8] pick_next_task_fair+0x38/0x70 [ 1119.145307] [81659fbc] __schedule+0x14c/0x6f0 [ 1119.145307] [81045d18] ? flush_tlb_page+0x48/0xb0 [ 1119.145307] [8165a88f] schedule+0x3f/0x60 [ 1119.145307] [811811f9] pipe_wait+0x59/0x80 [ 1119.145307] [8108aff0] ? add_wait_queue+0x60/0x60 [ 1119.145307] [8118196a] pipe_read+0x1da/0x330 [ 1119.145307] [81177562] do_sync_read+0xd2/0x110 [ 1119.145307] [8129cdb3] ? security_file_permission+0x93/0xb0 [ 1119.145307] [811779e1] ? rw_verify_area+0x61/0xf0 [ 1119.145307] [81177ec0] vfs_read+0xb0/0x180 [ 1119.145307] [81177fda] sys_read+0x4a/0x90 [ 1119.145307] [81664d82] system_call_fastpath+0x16/0x1b [ 1119.145307] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 48 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47 [ 1119.145307] RIP [81311c81] rb_next+0x1/0x50 [ 1119.145307] RSP 88001bd8dc18 [ 1119.145307] CR2: 0010 [ 1119.145307] ---[ end trace 787431227c69a127 ]--- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/999755 Title: Kernel crash on EC2 m1.large instances To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/999755/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 999755] Re: Kernel crash on EC2 VirtualBox
We've got small EC2 instances (single processor) that haven't exhibited this behaviour, but we get it with large EC2 instances (2 CPUs); the VirtualBox machine I just reproduced it with was specifically set to 2 CPUS. It seems to me that this bug might only occur on multi-cpu boxes? ** Summary changed: - Kernel crash on EC2 m1.large instances + Kernel crash on EC2 VirtualBox -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/999755 Title: Kernel crash on EC2 VirtualBox To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/999755/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 999755] Re: Kernel crash on EC2 VirtualBox
And we've just reproduced on EC2 with the debug kernel: [248587.286290] [ cut here ] [248587.286765] kernel BUG at /home/smb/precise-amd64/ubuntu-2.6/kernel/sched_fair.c:1239! [248587.286775] invalid opcode: [#1] SMP [248587.286783] CPU 0 [248587.286786] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables bnep rfcomm bluetooth parport_pc ppdev lp parport isofs acpiphp [248587.286822] [248587.286827] Pid: 18805, comm: ohai Not tainted 3.2.0-24-virtual #37+lp999755dbg1 [248587.286836] RIP: e030:[81050cd5] [81050cd5] pick_next_entity+0x105/0x110 [248587.286849] RSP: e02b:8801d02d7c28 EFLAGS: 00010096 [248587.286854] RAX: 002d RBX: 8801d20df800 RCX: 0003 [248587.286860] RDX: RSI: 81e000a0 RDI: 0004 [248587.286866] RBP: 8801d02d7c48 R08: 000a R09: [248587.286872] R10: R11: R12: 8801dff866c0 [248587.286878] R13: 8801d20de600 R14: R15: 01f41018 [248587.286889] FS: 7f5ec2010700() GS:8801dff73000() knlGS: [248587.286895] CS: e033 DS: ES: CR0: 8005003b [248587.286901] CR2: 01f09f30 CR3: 0001cec3e000 CR4: 2660 [248587.286908] DR0: DR1: DR2: [248587.286914] DR3: DR6: 0ff0 DR7: 0400 [248587.286921] Process ohai (pid: 18805, threadinfo 8801d02d6000, task 8801cf10db80) [248587.286928] Stack: [248587.286931] 8801d20df800 8801dff866c0 8801d20de600 [248587.286944] 8801d02d7c78 810544e8 8801d02d7c78 8801dff866c0 [248587.286956] 8801cf10df28 8801d02d7cf8 81652f3c [248587.286968] Call Trace: [248587.286976] [810544e8] pick_next_task_fair+0x38/0x70 [248587.286984] [81652f3c] __schedule+0x14c/0x6f0 [248587.286992] [8165564e] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [248587.286999] [816535af] schedule+0x3f/0x60 [248587.287006] [8117e149] pipe_wait+0x59/0x80 [248587.287014] [81089370] ? add_wait_queue+0x60/0x60 [248587.287021] [8117e8aa] pipe_read+0x1da/0x330 [248587.287028] [81174552] do_sync_read+0xd2/0x110 [248587.287036] [8100a25d] ? xen_force_evtchn_callback+0xd/0x10 [248587.287043] [8100aa32] ? check_events+0x12/0x20 [248587.287051] [81298d63] ? security_file_permission+0x93/0xb0 [248587.287057] [811749d1] ? rw_verify_area+0x61/0xf0 [248587.287063] [81174eb0] vfs_read+0xb0/0x180 [248587.287069] [81174fca] sys_read+0x4a/0x90 [248587.287076] [8165da42] system_call_fastpath+0x16/0x1b [248587.287081] Code: 89 df e8 69 c1 5e 00 48 8b 73 38 48 c7 c7 7b ef 9f 81 31 c0 e8 98 c9 5e 00 48 8b 73 10 48 c7 c7 98 ef 9f 81 31 c0 e8 86 c9 5e 00 0f 0b 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89 [248587.287170] RIP [81050cd5] pick_next_entity+0x105/0x110 [248587.287177] RSP 8801d02d7c28 [248587.287194] ---[ end trace 124f7d4d99f55a46 ]--- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/999755 Title: Kernel crash on EC2 VirtualBox To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/999755/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 999755] Re: Kernel crash on EC2 VirtualBox
And another one from the debug kernel on EC2, with a slightly different call stack: [ 4389.480352] [ cut here ] [ 4389.480884] kernel BUG at /home/smb/precise-amd64/ubuntu-2.6/kernel/sched_fair.c:1239! [ 4389.480894] invalid opcode: [#1] SMP [ 4389.480902] CPU 0 [ 4389.480905] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables bnep rfcomm bluetooth parport_pc ppdev lp parport isofs acpiphp [ 4389.480945] [ 4389.480949] Pid: 24612, comm: ohai Not tainted 3.2.0-24-virtual #37+lp999755dbg1 [ 4389.480958] RIP: e030:[81050cd5] [81050cd5] pick_next_entity+0x105/0x110 [ 4389.480972] RSP: e02b:8801d0cdd818 EFLAGS: 00010092 [ 4389.480977] RAX: 002c RBX: 8801d1106e00 RCX: 0003 [ 4389.480983] RDX: RSI: 81e000a0 RDI: 0004 [ 4389.480988] RBP: 8801d0cdd838 R08: 000a R09: [ 4389.480994] R10: R11: R12: 8801dff866c0 [ 4389.481000] R13: 8801d1107a00 R14: 0008 R15: 8801d1e97800 [ 4389.481009] FS: 7f09fcc8a700() GS:8801dff73000() knlGS: [ 4389.481015] CS: e033 DS: ES: CR0: 8005003b [ 4389.481021] CR2: 012dbf30 CR3: 0001d0702000 CR4: 2660 [ 4389.481028] DR0: DR1: DR2: [ 4389.481035] DR3: DR6: 0ff0 DR7: 0400 [ 4389.481041] Process ohai (pid: 24612, threadinfo 8801d0cdc000, task 8801d2905b80) [ 4389.481047] Stack: [ 4389.481051] 8801d1106e00 8801dff866c0 8801d1107a00 0008 [ 4389.481063] 8801d0cdd868 810544e8 8801d0cdd868 8801dff866c0 [ 4389.481075] 8801d2905f28 8801d0cdd8e8 81652f3c [ 4389.481086] Call Trace: [ 4389.481093] [810544e8] pick_next_task_fair+0x38/0x70 [ 4389.481101] [81652f3c] __schedule+0x14c/0x6f0 [ 4389.481108] [816535af] schedule+0x3f/0x60 [ 4389.481114] [816546cd] schedule_hrtimeout_range_clock+0x14d/0x170 [ 4389.481124] [8100aa1f] ? xen_restore_fl_direct_reloc+0x4/0x4 [ 4389.481131] [8165564e] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [ 4389.481139] [8108935d] ? add_wait_queue+0x4d/0x60 [ 4389.481146] [81654703] schedule_hrtimeout_range+0x13/0x20 [ 4389.481154] [811877d9] poll_schedule_timeout+0x49/0x70 [ 4389.481160] [81188356] do_select+0x4d6/0x600 [ 4389.481167] [811878e0] ? poll_freewait+0xe0/0xe0 [ 4389.481173] [811879d0] ? __pollwait+0xf0/0xf0 [ 4389.481179] [81005191] ? __raw_callee_save_xen_pte_val+0x11/0x1e [ 4389.481186] [81054c3e] ? update_curr+0x21e/0x230 [ 4389.481192] [8103cc65] ? pvclock_clocksource_read+0x55/0xf0 [ 4389.481199] [810553db] ? check_preempt_wakeup+0x15b/0x230 [ 4389.481207] [8104ed74] ? check_preempt_curr+0x84/0xa0 [ 4389.481214] [8104edcd] ? ttwu_do_wakeup+0x3d/0x120 [ 4389.481222] [8130fd49] ? put_dec+0x59/0x60 [ 4389.481228] [81310c1f] ? number.isra.2+0x31f/0x350 [ 4389.481236] [81323436] ? nla_parse+0x86/0xe0 [ 4389.481242] [8165564e] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [ 4389.481250] [8105e610] ? try_to_wake_up+0x190/0x200 [ 4389.481257] [81188641] core_sys_select+0x1c1/0x330 [ 4389.481263] [8100aa32] ? check_events+0x12/0x20 [ 4389.481269] [8100a25d] ? xen_force_evtchn_callback+0xd/0x10 [ 4389.481276] [8100aa32] ? check_events+0x12/0x20 [ 4389.481282] [8100aa1f] ? xen_restore_fl_direct_reloc+0x4/0x4 [ 4389.481289] [81004c62] ? xen_mc_flush+0xb2/0x1c0 [ 4389.481295] [8100aa1f] ? xen_restore_fl_direct_reloc+0x4/0x4 [ 4389.481302] [811889eb] sys_select+0xbb/0x100 [ 4389.481308] [8105edf7] ? schedule_tail+0x27/0xb0 [ 4389.481314] [8165da42] system_call_fastpath+0x16/0x1b [ 4389.481319] Code: 89 df e8 69 c1 5e 00 48 8b 73 38 48 c7 c7 7b ef 9f 81 31 c0 e8 98 c9 5e 00 48 8b 73 10 48 c7 c7 98 ef 9f 81 31 c0 e8 86 c9 5e 00 0f 0b 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89 [ 4389.481407] RIP [81050cd5] pick_next_entity+0x105/0x110 [ 4389.484006] RSP 8801d0cdd818 [ 4389.484006] ---[ end trace 7ee7cea7516c9821 ]--- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/999755 Title: Kernel crash on EC2 VirtualBox To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/999755/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 999755] Re: Kernel crash on EC2 m1.large instances
We've also seen this on the -24.38 and -24.39 kernels now: [56843.390534] BUG: unable to handle kernel NULL pointer dereference at 0010 [56843.390551] IP: [8130d901] rb_next+0x1/0x50 [56843.390566] PGD 1d20a7067 PUD 1d29a2067 PMD 0 [56843.390575] Oops: [#1] SMP [56843.390583] CPU 1 [56843.390585] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables parport_pc ppdev lp parport isofs acpiphp [56843.390616] [56843.390621] Pid: 1340, comm: chef-client Not tainted 3.2.0-24-virtual #39-Ubuntu [56843.390629] RIP: e030:[8130d901] [8130d901] rb_next+0x1/0x50 [56843.390638] RSP: e02b:8801d050dc18 EFLAGS: 00010046 [56843.390642] RAX: RBX: 8801d24c8e00 RCX: [56843.390648] RDX: fff0 RSI: RDI: 0010 [56843.390654] RBP: 8801d050dc48 R08: R09: [56843.390659] R10: 8801dff866c0 R11: 0001 R12: [56843.390664] R13: R14: R15: 011168a8 [56843.390674] FS: 7fda5feaa700() GS:8801dff8f000() knlGS: [56843.390681] CS: e033 DS: ES: CR0: 8005003b [56843.390686] CR2: 0010 CR3: 0001d28cc000 CR4: 2660 [56843.390692] DR0: DR1: DR2: [56843.390698] DR3: DR6: 0ff0 DR7: 0400 [56843.390704] Process chef-client (pid: 1340, threadinfo 8801d050c000, task 8801d2405b80) [56843.390711] Stack: [56843.390714] 8801d050dc48 8104ee19 8801d24c8e00 8801dffa26c0 [56843.390726] 8801d24c9400 8801d050dc78 810545e8 [56843.390737] 8801d050dc78 8801dffa26c0 0001 8801d2405f28 [56843.390748] Call Trace: [56843.390758] [8104ee19] ? pick_next_entity+0xb9/0xe0 [56843.390765] [810545e8] pick_next_task_fair+0x38/0x70 [56843.390774] [816530dc] __schedule+0x14c/0x6f0 [56843.390781] [816557ee] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [56843.390789] [8165374f] schedule+0x3f/0x60 [56843.390797] [8117e209] pipe_wait+0x59/0x80 [56843.390804] [81089470] ? add_wait_queue+0x60/0x60 [56843.390810] [8117e96a] pipe_read+0x1da/0x330 [56843.390817] [81174612] do_sync_read+0xd2/0x110 [56843.390824] [8100a25d] ? xen_force_evtchn_callback+0xd/0x10 [56843.390831] [8100aa32] ? check_events+0x12/0x20 [56843.390838] [81298dc3] ? security_file_permission+0x93/0xb0 [56843.390845] [81174a91] ? rw_verify_area+0x61/0xf0 [56843.390851] [81174f70] vfs_read+0xb0/0x180 [56843.390857] [8117508a] sys_read+0x4a/0x90 [56843.390863] [8165dbc2] system_call_fastpath+0x16/0x1b [56843.390869] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 48 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47 [56843.390960] RIP [8130d901] rb_next+0x1/0x50 [56843.390967] RSP 8801d050dc18 [56843.390970] CR2: 0010 [56843.390990] ---[ end trace ddb8957a1d924ccc ]--- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/999755 Title: Kernel crash on EC2 m1.large instances To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/999755/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 999755] Re: Kernel crash on EC2 m1.large instances
Triggered this again by running ohai in a continuous loop, took about 24 hours to occur: [18438803.627371] BUG: unable to handle kernel NULL pointer dereference at 0010 [18438803.627388] IP: [8130d7f1] rb_next+0x1/0x50 [18438803.627402] PGD 1d0efa067 PUD 1d232d067 PMD 0 [18438803.627411] Oops: [#1] SMP [18438803.627419] CPU 1 [18438803.627422] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables isofs acpiphp [18438803.627447] [18438803.627452] Pid: 29083, comm: ohai Not tainted 3.2.0-23-virtual #36-Ubuntu [18438803.627460] RIP: e030:[8130d7f1] [8130d7f1] rb_next+0x1/0x50 [18438803.627469] RSP: e02b:8801d225d808 EFLAGS: 00010046 [18438803.627473] RAX: RBX: 8801d2232400 RCX: [18438803.627479] RDX: fff0 RSI: 8801dffa2760 RDI: 0010 [18438803.627485] RBP: 8801d225d838 R08: R09: [18438803.627490] R10: 8801dff866c0 R11: R12: [18438803.627497] R13: R14: 0280 R15: 8801d0992300 [18438803.627508] FS: 7f34206c2700() GS:8801dff8f000() knlGS: [18438803.627515] CS: e033 DS: ES: CR0: 8005003b [18438803.627521] CR2: 0010 CR3: 0001d0e9e000 CR4: 2660 [18438803.627527] DR0: DR1: DR2: [18438803.627534] DR3: DR6: 0ff0 DR7: 0400 [18438803.627541] Process ohai (pid: 29083, threadinfo 8801d225c000, task 8801d260adc0) [18438803.627547] Stack: [18438803.627551] 8801d225d838 8104ece9 8801d2232400 8801dffa26c0 [18438803.627562] 8801d0f8fc00 0280 8801d225d868 810544b8 [18438803.627573] 8801d225d868 8801dffa26c0 0001 8801d260b168 [18438803.627584] Call Trace: [18438803.627596] [8104ece9] ? pick_next_entity+0xb9/0xe0 [18438803.627604] [810544b8] pick_next_task_fair+0x38/0x70 [18438803.627861] [81652ddc] __schedule+0x14c/0x6f0 [18438803.627874] [8111d335] ? prep_new_page+0x145/0x1e0 [18438803.627881] [8165344f] schedule+0x3f/0x60 [18438803.627889] [8165454c] schedule_hrtimeout_range_clock+0x12c/0x170 [18438803.627901] [8108c890] ? update_rmtp+0x70/0x70 [18438803.627908] [8108d684] ? hrtimer_start_range_ns+0x14/0x20 [18438803.627916] [816545a3] schedule_hrtimeout_range+0x13/0x20 [18438803.627927] [811877a9] poll_schedule_timeout+0x49/0x70 [18438803.627934] [81188326] do_select+0x4d6/0x600 [18438803.627942] [811878b0] ? poll_freewait+0xe0/0xe0 [18438803.627949] [811879a0] ? __pollwait+0xf0/0xf0 [18438803.627956] [811879a0] ? __pollwait+0xf0/0xf0 [18438803.627966] [8100a25d] ? xen_force_evtchn_callback+0xd/0x10 [18438803.627974] [8100aa32] ? check_events+0x12/0x20 [18438803.627981] [8100a25d] ? xen_force_evtchn_callback+0xd/0x10 [18438803.627988] [8100aa32] ? check_events+0x12/0x20 [18438803.627995] [8100aa1f] ? xen_restore_fl_direct_reloc+0x4/0x4 [18438803.628003] [81006d1d] ? xen_flush_tlb_single+0xbd/0x210 [18438803.628013] [81306dbd] ? cpumask_any_but+0x2d/0x40 [18438803.628022] [81044b98] ? flush_tlb_page+0x48/0xb0 [18438803.628030] [810438ac] ? ptep_set_access_flags+0x6c/0x70 [18438803.628038] [81138c52] ? do_wp_page+0x382/0x740 [18438803.628045] [81006739] ? pte_mfn_to_pfn+0x89/0xf0 [18438803.628053] [81005209] ? __raw_callee_save_xen_pmd_val+0x11/0x1e [18438803.628061] [81188611] core_sys_select+0x1c1/0x330 [18438803.628069] [8113af98] ? handle_mm_fault+0x1f8/0x350 [18438803.628076] [8103cc65] ? pvclock_clocksource_read+0x55/0xf0 [18438803.628085] [8100a540] ? xen_clocksource_read+0x20/0x30 [18438803.628092] [8100a629] ? xen_clocksource_get_cycles+0x9/0x10 [18438803.628101] [810933ed] ? ktime_get_ts+0xad/0xe0 [18438803.628108] [811889bb] sys_select+0xbb/0x100 [18438803.628117] [8165d8c2] system_call_fastpath+0x16/0x1b [18438803.628123] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 48 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47 [18438803.628207] RIP [8130d7f1] rb_next+0x1/0x50 [18438803.628215] RSP 8801d225d808 [18438803.628219] CR2: 0010 [18438803.628229] ---[ end trace 6e3e751b67665edf ]--- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/999755 Title: Kernel crash on EC2 m1.large instances To manage notifications about this bug go
[Bug 999755] Re: Kernel crash on EC2 m1.large instances
I've reproduced this by running the OHAI command from the OpsCode Chef ohai gem (0.6.12) in a loop, although it took nearly 2 days before it triggered. Basically I ran `gem install ohai; while true; do ohai; done` in a screen session. The stack trace is: [18362917.357055] BUG: unable to handle kernel NULL pointer dereference at 0010 [18362917.357079] IP: [8130d7f1] rb_next+0x1/0x50 [18362917.357098] PGD 1d098d067 PUD 1d045b067 PMD 0 [18362917.357110] Oops: [#1] SMP [18362917.357122] CPU 0 [18362917.357126] Modules linked in: ipt_REJECT xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables isofs acpiphp [18362917.357152] [18362917.357157] Pid: 21217, comm: ohai Not tainted 3.2.0-23-virtual #36-Ubuntu [18362917.357166] RIP: e030:[8130d7f1] [8130d7f1] rb_next+0x1/0x50 [18362917.357176] RSP: e02b:8801d22f1808 EFLAGS: 00010046 [18362917.357181] RAX: RBX: 8801d0842600 RCX: [18362917.357187] RDX: fff0 RSI: RDI: 0010 [18362917.357193] RBP: 8801d22f1838 R08: R09: [18362917.357199] R10: 8801dffa26c0 R11: 0001 R12: [18362917.357207] R13: R14: 0008 R15: 8801d0dc2d00 [18362917.357218] FS: 7fcdb3810700() GS:8801dff73000() knlGS: [18362917.357225] CS: e033 DS: ES: CR0: 8005003b [18362917.357232] CR2: 0010 CR3: 0001d2641000 CR4: 2660 [18362917.357240] DR0: DR1: DR2: [18362917.357246] DR3: DR6: 0ff0 DR7: 0400 [18362917.357253] Process ohai (pid: 21217, threadinfo 8801d22f, task 8801d0ad44a0) [18362917.357261] Stack: [18362917.357265] 8801d22f1838 8104ece9 8801d0842600 8801dff866c0 [18362917.357277] 8801d0842e00 0008 8801d22f1868 810544b8 [18362917.357289] 8801d22f1868 8801dff866c0 8801d0ad4848 [18362917.357301] Call Trace: [18362917.357314] [8104ece9] ? pick_next_entity+0xb9/0xe0 [18362917.357322] [810544b8] pick_next_task_fair+0x38/0x70 [18362917.357331] [81652ddc] __schedule+0x14c/0x6f0 [18362917.357341] [8100a25d] ? xen_force_evtchn_callback+0xd/0x10 [18362917.357348] [8165344f] schedule+0x3f/0x60 [18362917.357355] [8165456d] schedule_hrtimeout_range_clock+0x14d/0x170 [18362917.357365] [8100aa1f] ? xen_restore_fl_direct_reloc+0x4/0x4 [18362917.357373] [816554ee] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [18362917.357383] [8108932d] ? add_wait_queue+0x4d/0x60 [18362917.357389] [816545a3] schedule_hrtimeout_range+0x13/0x20 [18362917.357400] [811877a9] poll_schedule_timeout+0x49/0x70 [18362917.357408] [81188326] do_select+0x4d6/0x600 [18362917.357414] [811878b0] ? poll_freewait+0xe0/0xe0 [18362917.357422] [811879a0] ? __pollwait+0xf0/0xf0 [18362917.357431] [81306cd6] ? cpumask_next_and+0x36/0x50 [18362917.357438] [81052124] ? select_idle_sibling+0x174/0x220 [18362917.357445] [8130bbdb] ? radix_tree_lookup+0xb/0x10 [18362917.357453] [810d61c7] ? irq_to_desc+0x17/0x20 [18362917.357461] [810d902e] ? irq_get_irq_data+0xe/0x10 [18362917.357472] [813a404e] ? info_for_irq+0xe/0x30 [18362917.357478] [81306cd6] ? cpumask_next_and+0x36/0x50 [18362917.357487] [810592d1] ? find_busiest_group+0x171/0xbb0 [18362917.357494] [81188611] core_sys_select+0x1c1/0x330 [18362917.357501] [816554ee] ? _raw_spin_unlock_irqrestore+0x1e/0x30 [18362917.357511] [8163d029] ? idle_balance+0xf0/0x11b [18362917.357517] [8100a25d] ? xen_force_evtchn_callback+0xd/0x10 [18362917.357524] [8100aa32] ? check_events+0x12/0x20 [18362917.357531] [8100aa1f] ? xen_restore_fl_direct_reloc+0x4/0x4 [18362917.357538] [81004c62] ? xen_mc_flush+0xb2/0x1c0 [18362917.357545] [8100aa1f] ? xen_restore_fl_direct_reloc+0x4/0x4 [18362917.357552] [811889bb] sys_select+0xbb/0x100 [18362917.357559] [8105edc7] ? schedule_tail+0x27/0xb0 [18362917.357568] [8165d8c2] system_call_fastpath+0x16/0x1b [18362917.357573] Code: 89 06 48 8b 47 08 48 89 46 08 48 8b 47 10 48 89 46 10 c3 0f 1f 80 00 00 00 00 48 89 32 eb b2 0f 1f 00 48 89 70 10 eb a9 66 90 55 48 8b 17 48 89 e5 48 89 d0 48 83 e0 fc 48 39 c7 74 34 48 8b 47 [18362917.357653] RIP [8130d7f1] rb_next+0x1/0x50 [18362917.357660] RSP 8801d22f1808 [18362917.357664] CR2: 0010 [18362917.357673] ---[ end trace de16620c8d9e9c7c ]--- -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu.
[Bug 999755] Re: Kernel crash on EC2 m1.large instances
BTW Xen version from dmesg is: Xen version: 3.4.3-2.6.18 (preserve-AD) This is on EC2 so we have no control over this. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/999755 Title: Kernel crash on EC2 m1.large instances To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/999755/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
