On 01/31/2012 04:06 PM, Bram De Wilde wrote:
> Hi,
>
> We are setting up a ubuntu / openstack / KVM based private cloud. Of the 3
> servers involved in this configuration 1 has been crashing frequently over
> the last month.
> Since it is the only server responsible for the virtualization (so the only
> one running kvm virtual machines) my guess was to find some help with you
> guys. We are running the same install on 2 other machines (not running
> virtual machines!) who have been completely stable over the last three
> months, so a hardware issue might also be at play here?
>
> I have collected some stack traces over the last month and was hoping one of
> you might point us in the right direction as to what is going wrong on our
> system.
> We are running a default ubuntu 11.10 server install. The kernel has been
> upgraded from the 3.0.0-12 release to the 3.0.0-14 release over the last
> month, but we keep getting these crashes. We are now running the
> 3.2.2-030202-generic hoping that this is a kernel problem, and it is fixed in
> this latest version.
>
> If any additional information is required or I should file an official bug
> with this information, please let me know...
File a bug please.
Meanwhile:
> Jan 27 13:41:28 cmggcn01 kernel: [871350.761867] general protection fault:
> 0000 [#2] SMP
> Jan 27 13:41:28 cmggcn01 kernel: [871350.790117] CPU 14
> Jan 27 13:41:28 cmggcn01 kernel: [871350.790387] Modules linked in: btrfs
> zlib_deflate libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
> reiserfs ebt_arp ebt_ip 8021q garp ip6table_filter ip6_tables ebtable_nat
> ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
> xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp
> iptable_filter ip_tables x_tables bridge stp kvm_intel kvm nbd vesafb ib_iser
> rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp
> libiscsi scsi_transport_iscsi psmouse dcdbas dm_multipath serio_raw joydev
> ghes hed acpi_power_meter bonding lp parport i7core_edac edac_core ses
> enclosure usbhid hid megaraid_sas bnx2
> Jan 27 13:41:28 cmggcn01 kernel: [871351.005151]
> Jan 27 13:41:28 cmggcn01 kernel: [871351.036187] Pid: 90, comm: kswapd0
> Tainted: G D 3.0.0-14-server #23-Ubuntu Dell Inc. PowerEdge
> R710/0MD99X
> Jan 27 13:41:28 cmggcn01 kernel: [871351.072809] RIP:
> 0010:[<ffffffffa01a5890>] [<ffffffffa01a5890>] kvm_unmap_rmapp+0x20/0x60
> [kvm]
> Jan 27 13:41:28 cmggcn01 kernel: [871351.105190] RSP: 0018:ffff8817f3e27a60
> EFLAGS: 00010202
> Jan 27 13:41:28 cmggcn01 kernel: [871351.141329] RAX: 00008817f5d067f8 RBX:
> ffffc9001fd41ff8 RCX: ffffffffa01a58d0
> Jan 27 13:41:28 cmggcn01 kernel: [871351.179076] RDX: 0000000000000000 RSI:
> 0000000000000000 RDI: 00008817f5d067f8
> Jan 27 13:41:28 cmggcn01 kernel: [871351.212086] RBP: ffff8817f3e27a80 R08:
> ffff8817f315b3e0 R09: 0000000000000100
> Jan 27 13:41:28 cmggcn01 kernel: [871351.245788] R10: 000000000000000e R11:
> 0000000000000002 R12: ffff8817f2f0c000
> Jan 27 13:41:28 cmggcn01 kernel: [871351.277514] R13: 0000000000000000 R14:
> ffff880be235e000 R15: 00000000000d3cff
> Jan 27 13:41:28 cmggcn01 kernel: [871351.308421] FS: 0000000000000000(0000)
> GS:ffff88183fce0000(0000) knlGS:0000000000000000
> Jan 27 13:41:28 cmggcn01 kernel: [871351.339685] CS: 0010 DS: 0000 ES: 0000
> CR0: 000000008005003b
> Jan 27 13:41:28 cmggcn01 kernel: [871351.370089] CR2: 00007f8836442000 CR3:
> 0000000001c03000 CR4: 00000000000026e0
> Jan 27 13:41:28 cmggcn01 kernel: [871351.399771] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
> Jan 27 13:41:28 cmggcn01 kernel: [871351.428208] DR3: 0000000000000000 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
> Jan 27 13:41:28 cmggcn01 kernel: [871351.456153] Process kswapd0 (pid: 90,
> threadinfo ffff8817f3e26000, task ffff8817f63ac560)
> Jan 27 13:41:28 cmggcn01 kernel: [871351.484943] Stack:
> Jan 27 13:41:28 cmggcn01 kernel: [871351.512984] 0000000000000000
> ffffc9001fd41ff8 0000000000000001 00007f834a87e000
> Jan 27 13:41:28 cmggcn01 kernel: [871351.542025] ffff8817f3e27aa0
> ffffffffa01a5945 ffff880be235e060 0000000000000001
> Jan 27 13:41:28 cmggcn01 kernel: [871351.571050] ffff8817f3e27b10
> ffffffffa01a1dd9 ffff8817f3e27ae0 ffffffffa01a58d0
<snip>
> Jan 27 13:41:28 cmggcn01 kernel: [871352.080582] Code: e7 d0 e8 e0 66 90 e9
> a2 fe ff ff 55 48 89 e5 41 55 41 54 53 48 83 ec 08 66 66 66 66 90 45 31 ed 49
> 89 fc 48 89 f3 eb 20 0f 1f 00 <f6> 00 01 74 35 48 8b 15 74 7a 02 00 48 89 c6
> 4c 89 e7 41 bd 01
0: e8 e0 66 90 e9 callq 0xffffffffe99066e5
5: a2 fe ff ff 55 48 89 mov %al,0x41e5894855fffffe
c: e5 41
e: 55 push %rbp
f: 41 54 push %r12
11: 53 push %rbx
12: 48 83 ec 08 sub $0x8,%rsp
16: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax
1b: 45 31 ed xor %r13d,%r13d
1e: 49 89 fc mov %rdi,%r12
21: 48 89 f3 mov %rsi,%rbx
24: eb 20 jmp 0x46
26: 0f 1f 00 nopl (%rax)
29: f6 00 01 testb $0x1,(%rax)
^ dies here, %rax is non-canonical.
2c: 74 35 je 0x63
2e: 48 8b 15 74 7a 02 00 mov 0x27a74(%rip),%rdx # 0x27aa9
35: 48 89 c6 mov %rax,%rsi
38: 4c 89 e7 mov %r12,%rdi
static int kvm_unmap_rmapp(struct kvm *kvm, unsigned long *rmapp,
unsigned long data)
{
u64 *spte;
int need_tlb_flush = 0;
while ((spte = rmap_next(kvm, rmapp, NULL))) {
BUG_ON(!(*spte & PT_PRESENT_MASK));
^ here, when fetching *spte.
rmap_printk("kvm_rmap_unmap_hva: spte %p %llx\n", spte, *spte);
drop_spte(kvm, spte);
need_tlb_flush = 1;
}
return need_tlb_flush;
Looks like a use-after-free with the two bytes at offset 6 zeroed.
If this is reproducible, please rerun with the host kernel parameter
slub_debug=FZPU.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
