Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Thanks! Thanks, -Kai On 3/14/2017 5:57 AM, Paolo Bonzini wrote: On 13/03/2017 15:58, fangying wrote: Hi, Huang Kai After weeks of intensive testing, we think the problem is solved and this issue can be closed. Thanks for the update. We got to the same conclusion. Paolo
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 13/03/2017 15:58, fangying wrote: > Hi, Huang Kai > > After weeks of intensive testing, we think the problem is solved and > this issue can be closed. Thanks for the update. We got to the same conclusion. Paolo
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Hi, Huang Kai After weeks of intensive testing, we think the problem is solved and this issue can be closed. On 2017/2/27 15:38, Huang, Kai wrote: On 2/25/2017 2:44 PM, Herongguang (Stephen) wrote: On 2017/2/24 23:14, Paolo Bonzini wrote: On 24/02/2017 16:10, Chris Friesen wrote: On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote: On 2017/2/22 22:43, Paolo Bonzini wrote: Hopefully Gaohuai and Rongguang can help with this too. Paolo Yes, we are looking into and testing this. I think this can result in any memory corruption, if VM1 writes its PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1 is not registered yet), then VM1 is destroyed (hence its PML buffer is freed back to kernel), after that, VM2 starts migration, so CPU logs VM2’s dirty GFNS into a freed memory, results in any memory corruption. As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec) is eligible to back port to kernel stable. Are we expecting that fix to resolve the original issue, or is it a separate issue that needs fixing in stable? It should be the original issue. Paolo . Yes, I agree, though we are still testing. Hi Stephen, Sorry for late reply. I was taking the whole week off last week. How's the test going? Thanks, -Kai .
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 2/25/2017 2:44 PM, Herongguang (Stephen) wrote: On 2017/2/24 23:14, Paolo Bonzini wrote: On 24/02/2017 16:10, Chris Friesen wrote: On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote: On 2017/2/22 22:43, Paolo Bonzini wrote: Hopefully Gaohuai and Rongguang can help with this too. Paolo Yes, we are looking into and testing this. I think this can result in any memory corruption, if VM1 writes its PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1 is not registered yet), then VM1 is destroyed (hence its PML buffer is freed back to kernel), after that, VM2 starts migration, so CPU logs VM2’s dirty GFNS into a freed memory, results in any memory corruption. As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec) is eligible to back port to kernel stable. Are we expecting that fix to resolve the original issue, or is it a separate issue that needs fixing in stable? It should be the original issue. Paolo . Yes, I agree, though we are still testing. Hi Stephen, Sorry for late reply. I was taking the whole week off last week. How's the test going? Thanks, -Kai
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 2017/2/24 23:14, Paolo Bonzini wrote: On 24/02/2017 16:10, Chris Friesen wrote: On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote: On 2017/2/22 22:43, Paolo Bonzini wrote: Hopefully Gaohuai and Rongguang can help with this too. Paolo Yes, we are looking into and testing this. I think this can result in any memory corruption, if VM1 writes its PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1 is not registered yet), then VM1 is destroyed (hence its PML buffer is freed back to kernel), after that, VM2 starts migration, so CPU logs VM2’s dirty GFNS into a freed memory, results in any memory corruption. As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec) is eligible to back port to kernel stable. Are we expecting that fix to resolve the original issue, or is it a separate issue that needs fixing in stable? It should be the original issue. Paolo . Yes, I agree, though we are still testing.
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 24/02/2017 16:10, Chris Friesen wrote: > On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote: > >> On 2017/2/22 22:43, Paolo Bonzini wrote: > >>> Hopefully Gaohuai and Rongguang can help with this too. >>> >>> Paolo >> >> Yes, we are looking into and testing this. >> >> I think this can result in any memory corruption, if VM1 writes its >> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1 >> is not registered yet), then VM1 is destroyed (hence its PML buffer >> is freed back to kernel), after that, VM2 starts migration, so CPU >> logs VM2’s dirty GFNS into a freed memory, results in any memory >> corruption. >> >> As its severity, this commit >> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec) >> >> is eligible to back port to kernel stable. > > Are we expecting that fix to resolve the original issue, or is it a > separate issue that needs fixing in stable? It should be the original issue. Paolo
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 02/23/2017 08:23 PM, Herongguang (Stephen) wrote: On 2017/2/22 22:43, Paolo Bonzini wrote: Hopefully Gaohuai and Rongguang can help with this too. Paolo . Yes, we are looking into and testing this. I think this can result in any memory corruption, if VM1 writes its PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1 is not registered yet), then VM1 is destroyed (hence its PML buffer is freed back to kernel), after that, VM2 starts migration, so CPU logs VM2’s dirty GFNS into a freed memory, results in any memory corruption. As its severity, this commit (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec) is eligible to back port to kernel stable. Are we expecting that fix to resolve the original issue, or is it a separate issue that needs fixing in stable? Chris
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On Fri, Feb 24, 2017 at 11:00:32AM +0100, Paolo Bonzini wrote:
>
>
> On 24/02/2017 10:59, Greg KH wrote:
> > On Fri, Feb 24, 2017 at 05:35:17PM +0800, Herongguang (Stephen) wrote:
> >>
> >>
> >> On 2017/2/24 10:23, Herongguang (Stephen) wrote:
> >>>
> >>>
> >>> On 2017/2/22 22:43, Paolo Bonzini wrote:
>
>
> On 22/02/2017 14:31, Chris Friesen wrote:
> >>>
> >>
> >> Can you reproduce it with kernel 4.8+? I'm suspecting commmit
> >> 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
> >> 2016-07-14) to be the fix.
> >
> > I can't easily try with a newer kernel, the software package we're using
> > has kernel patches that would have to be ported.
> >
> > I'm at a conference, don't really have time to set up a pair of test
> > machines from scratch with a custom kernel.
>
> Hopefully Gaohuai and Rongguang can help with this too.
>
> Paolo
>
> .
>
> >>> Yes, we are looking into and testing this.
> >>>
> >>> I think this can result in any memory corruption, if VM1 writes its
> >>> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
> >>> is not registered yet), then VM1 is destroyed (hence its PML buffer
> >>> is freed back to kernel), after that, VM2 starts migration, so CPU
> >>> logs VM2’s dirty GFNS into a freed memory, results in any memory
> >>> corruption.
> >>>
> >>> As its severity, this commit
> >>> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
> >>> is eligible to back port to kernel stable.
> >>
> >> Hi, Greg, can you cherry pick commit
> >> 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec to 4.4-y?
> >
> > If the KVM maintainers say it is ok to do so, yes, I will.
>
> Yes, he beat me by minutes. :)
Heh, ok, I'll go add it to the recently-announced 4.4.52-rc1 release.
thanks,
greg k-h
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 24/02/2017 10:59, Greg KH wrote:
> On Fri, Feb 24, 2017 at 05:35:17PM +0800, Herongguang (Stephen) wrote:
>>
>>
>> On 2017/2/24 10:23, Herongguang (Stephen) wrote:
>>>
>>>
>>> On 2017/2/22 22:43, Paolo Bonzini wrote:
On 22/02/2017 14:31, Chris Friesen wrote:
>>>
>>
>> Can you reproduce it with kernel 4.8+? I'm suspecting commmit
>> 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
>> 2016-07-14) to be the fix.
>
> I can't easily try with a newer kernel, the software package we're using
> has kernel patches that would have to be ported.
>
> I'm at a conference, don't really have time to set up a pair of test
> machines from scratch with a custom kernel.
Hopefully Gaohuai and Rongguang can help with this too.
Paolo
.
>>> Yes, we are looking into and testing this.
>>>
>>> I think this can result in any memory corruption, if VM1 writes its
>>> PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
>>> is not registered yet), then VM1 is destroyed (hence its PML buffer
>>> is freed back to kernel), after that, VM2 starts migration, so CPU
>>> logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
>>>
>>> As its severity, this commit
>>> (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
>>> is eligible to back port to kernel stable.
>>
>> Hi, Greg, can you cherry pick commit
>> 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec to 4.4-y?
>
> If the KVM maintainers say it is ok to do so, yes, I will.
Yes, he beat me by minutes. :)
Paolo
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On Fri, Feb 24, 2017 at 05:35:17PM +0800, Herongguang (Stephen) wrote:
>
>
> On 2017/2/24 10:23, Herongguang (Stephen) wrote:
> >
> >
> > On 2017/2/22 22:43, Paolo Bonzini wrote:
> > >
> > >
> > > On 22/02/2017 14:31, Chris Friesen wrote:
> > > > > >
> > > > >
> > > > > Can you reproduce it with kernel 4.8+? I'm suspecting commmit
> > > > > 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
> > > > > 2016-07-14) to be the fix.
> > > >
> > > > I can't easily try with a newer kernel, the software package we're using
> > > > has kernel patches that would have to be ported.
> > > >
> > > > I'm at a conference, don't really have time to set up a pair of test
> > > > machines from scratch with a custom kernel.
> > >
> > > Hopefully Gaohuai and Rongguang can help with this too.
> > >
> > > Paolo
> > >
> > > .
> > >
> > Yes, we are looking into and testing this.
> >
> > I think this can result in any memory corruption, if VM1 writes its
> > PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
> > is not registered yet), then VM1 is destroyed (hence its PML buffer
> > is freed back to kernel), after that, VM2 starts migration, so CPU
> > logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
> >
> > As its severity, this commit
> > (http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
> > is eligible to back port to kernel stable.
>
> Hi, Greg, can you cherry pick commit 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec
> to 4.4-y?
If the KVM maintainers say it is ok to do so, yes, I will.
thanks,
greg k-h
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 2017/2/24 10:23, Herongguang (Stephen) wrote:
On 2017/2/22 22:43, Paolo Bonzini wrote:
On 22/02/2017 14:31, Chris Friesen wrote:
Can you reproduce it with kernel 4.8+? I'm suspecting commmit
4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
2016-07-14) to be the fix.
I can't easily try with a newer kernel, the software package we're using
has kernel patches that would have to be ported.
I'm at a conference, don't really have time to set up a pair of test
machines from scratch with a custom kernel.
Hopefully Gaohuai and Rongguang can help with this too.
Paolo
.
Yes, we are looking into and testing this.
I think this can result in any memory corruption, if VM1 writes its
PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
is not registered yet), then VM1 is destroyed (hence its PML buffer
is freed back to kernel), after that, VM2 starts migration, so CPU
logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
As its severity, this commit
(http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
is eligible to back port to kernel stable.
Hi, Greg, can you cherry pick commit 4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec
to 4.4-y?
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 2017/2/22 22:43, Paolo Bonzini wrote:
On 22/02/2017 14:31, Chris Friesen wrote:
Can you reproduce it with kernel 4.8+? I'm suspecting commmit
4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
2016-07-14) to be the fix.
I can't easily try with a newer kernel, the software package we're using
has kernel patches that would have to be ported.
I'm at a conference, don't really have time to set up a pair of test
machines from scratch with a custom kernel.
Hopefully Gaohuai and Rongguang can help with this too.
Paolo
.
Yes, we are looking into and testing this.
I think this can result in any memory corruption, if VM1 writes its
PML buffer into VM2’s VMCS (since sched_in/sched_out notifier of VM1
is not registered yet), then VM1 is destroyed (hence its PML buffer
is freed back to kernel), after that, VM2 starts migration, so CPU
logs VM2’s dirty GFNS into a freed memory, results in any memory corruption.
As its severity, this commit
(http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=4e59516a12a6ef6dcb660cb3a3f70c64bd60cfec)
is eligible to back port to kernel stable.
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 02/22/2017 05:15 AM, Paolo Bonzini wrote:
On 22/02/2017 04:08, Chris Friesen wrote:
On 02/19/2017 10:38 PM, Han, Huaitong wrote:
Hi, Gaohuai
I tried to debug the problem, and I found the indirect cause may be that
the rmap value is not cleared when KVM mmu page is freed. I have read
code without the root cause. Can you stable reproduce the the issue?
Many guesses need to be verified.
In both cases it seems to have been triggered by repeatedly
live-migrating a KVM virtual machine between two hypervisors with
Broadwell CPUs running the latest CentOS 7.
It's a race of some sort, it doesn't happen every time.
Can you reproduce it with kernel 4.8+? I'm suspecting commmit
4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
2016-07-14) to be the fix.
I can't easily try with a newer kernel, the software package we're using has
kernel patches that would have to be ported.
I'm at a conference, don't really have time to set up a pair of test machines
from scratch with a custom kernel.
Chris
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 22/02/2017 14:31, Chris Friesen wrote:
>>>
>>
>> Can you reproduce it with kernel 4.8+? I'm suspecting commmit
>> 4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
>> 2016-07-14) to be the fix.
>
> I can't easily try with a newer kernel, the software package we're using
> has kernel patches that would have to be ported.
>
> I'm at a conference, don't really have time to set up a pair of test
> machines from scratch with a custom kernel.
Hopefully Gaohuai and Rongguang can help with this too.
Paolo
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 22/02/2017 04:08, Chris Friesen wrote:
> On 02/19/2017 10:38 PM, Han, Huaitong wrote:
>> Hi, Gaohuai
>>
>> I tried to debug the problem, and I found the indirect cause may be that
>> the rmap value is not cleared when KVM mmu page is freed. I have read
>> code without the root cause. Can you stable reproduce the the issue?
>> Many guesses need to be verified.
>
> In both cases it seems to have been triggered by repeatedly
> live-migrating a KVM virtual machine between two hypervisors with
> Broadwell CPUs running the latest CentOS 7.
>
> It's a race of some sort, it doesn't happen every time.
Can you reproduce it with kernel 4.8+? I'm suspecting commmit
4e59516a12a6 ("kvm: vmx: ensure VMCS is current while enabling PML",
2016-07-14) to be the fix.
Paolo
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
On 02/19/2017 10:38 PM, Han, Huaitong wrote: Hi, Gaohuai I tried to debug the problem, and I found the indirect cause may be that the rmap value is not cleared when KVM mmu page is freed. I have read code without the root cause. Can you stable reproduce the the issue? Many guesses need to be verified. In both cases it seems to have been triggered by repeatedly live-migrating a KVM virtual machine between two hypervisors with Broadwell CPUs running the latest CentOS 7. It's a race of some sort, it doesn't happen every time. Chris
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Hi, Gaohuai I tried to debug the problem, and I found the indirect cause may be that the rmap value is not cleared when KVM mmu page is freed. I have read code without the root cause. Can you stable reproduce the the issue? Many guesses need to be verified. On Mon, 2017-02-20 at 10:17 +0800, hangaohuai wrote: > Hi, Kai Huang and Xiao Guangrong. > > For the problem mentioned above, there may be a bug related to PML and > probably on Broadwell CPUs. > > I've been reading the code for PML for days, but I haven't found any clews. > Do you have any idea about this BUG ? > > Hope you can help! > > On 2017/2/10 23:28, Chris Friesen wrote: > > > > Well, not so much solved as worked around it. > > > > It seems that the problem only showed up on Broadwell, which made us wonder > > about something hardware specific. > > > > Setting "kvm-intel.eptad=0" in the kernel boot args seems to mask the > > problem for us. > > > > Chris > > > > On 02/10/2017 03:11 AM, Herongguang (Stephen) wrote: > >> Hi, Chris Friesen, did you solve the problem? > >> > >> On 2017/2/9 22:37, Herongguang (Stephen) wrote: > >>> Hi. > >>> I had a problem when I just repeatedly live migrate a vm between two > >>> compute > >>> nodes. > >>> The phenomenon was that the KVM module was crashed and then the host > >>> rebooted. > >>> However I cannot reliably trigger this BUG. > >>> > >>> The backtrace is the same as > >>> http://www.spinics.net/lists/kvm/msg138475.html. > >>> > >>> The crash is triggered when function __rmap_clear_dirty is invoked and an > >>> invalid page(0x7f183000) is visited. > >>> The value %rdi is 0x7f183000, which is obviously not a valid > >>> kernel > >>> pointer for x86_64. > >>> The assembly of __rmap_clear_dirty is: > >>> 0xa04d9ac0 <__rmap_clear_dirty>:callq > >>> 0x816543d0 > >>> > >>> 0xa04d9ac5 <__rmap_clear_dirty+5>: push %rbp > >>> 0xa04d9ac6 <__rmap_clear_dirty+6>: mov %rsp,%rbp > >>> 0xa04d9ac9 <__rmap_clear_dirty+9>: push %rbx > >>> 0xa04d9aca <__rmap_clear_dirty+10>: sub $0x18,%rsp > >>> 0xa04d9ace <__rmap_clear_dirty+14>: mov (%rsi),%rdi > >>> 0xa04d9ad1 <__rmap_clear_dirty+17>: mov %gs:0x28,%rax > >>> 0xa04d9ada <__rmap_clear_dirty+26>: mov %rax,-0x10(%rbp) > >>> 0xa04d9ade <__rmap_clear_dirty+30>: xor %eax,%eax > >>> 0xa04d9ae0 <__rmap_clear_dirty+32>: test %rdi,%rdi > >>> 0xa04d9ae3 <__rmap_clear_dirty+35>: je 0xa04d9b78 > >>> 0xa04d9ae9 <__rmap_clear_dirty+41>: test $0x1,%dil > >>> 0xa04d9aed <__rmap_clear_dirty+45>: je 0xa04d9b98 > >>> 0xa04d9af3 <__rmap_clear_dirty+51>: and > >>> $0xfffe,%rdi > >>> 0xa04d9af7 <__rmap_clear_dirty+55>: movl $0x0,-0x18(%rbp) > >>> 0xa04d9afe <__rmap_clear_dirty+62>: mov %rdi,-0x20(%rbp) > >>> 0xa04d9b02 <__rmap_clear_dirty+66>: mov (%rdi),%rdi > >>> 0xa04d9b05 <__rmap_clear_dirty+69>: test %rdi,%rdi > >>> ... > >>> > >>> The details of the environment is: > >>> Host info: x86_64 CentOS 7 kernel (3.10.0-327.36.58.10_2.x86_64, slightly > >>> modified). The CPU is Broadwell Intel(R) Xeon(R) CPU E5-2618L v4 @ > >>> 2.20GHz. > >>> Kmod info: version is 4.4.36 > >>> What I know is that the parameter PML(Page Modification Logging) is > >>> enabled by > >>> default. > >>> # cat /sys/module/kvm_intel/parameters/pml > >>> # Y > >>> > >>> Below is the crash message: > >>> [1548777.924180] kvm: zapping shadow pages for mmio generation wraparound > >>> [1548777.947844] HTB: quantum of class 10001 is big. Consider r2q change. > >>> [1548778.185389] kvm: zapping shadow pages for mmio generation wraparound > >>> [1548778.994801] BUG: unable to handle kernel paging request at > >>> 7f183000 > >>> [1548779.002135] IP: [] __rmap_clear_dirty+0x4a/0xf0 > >>> [kvm] > >>> [1548779.009151] PGD 1f8452067 PUD 0 > >>> [1548779.012774] Thread overran stack, or stack corrupted > >>> [1548779.018076] Oops: [#1] SMP > >>> [1548779.027050] collected_len = 1048570, LOG_BUF_LEN_LOCAL = 1048576 > >>> [1548779.042039] kbox: no notify die func register. no need to notify > >>> [1548779.048392] do nothing after die! > >>> [1548779.052071] Modules linked in: kboxdriver(O) kbox(O) sch_htb > >>> ip_set_hash_net ip6table_filter ip6_tables iptable_filter igb_uio(OE) uio > >>> bridge dm_service_time dm_multipath iscsi_tcp libiscsi_tcp libiscsi > >>> scsi_transport_iscsi 8021q garp stp mrp llc vfat fat isofs ext4 jbd2 xfs > >>> sha512_generic dev_connlimit(O) bum(O) ip_set nfnetlink prio(O) nat(O) > >>> vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre ib_uverbs(OVE) > >>> hotpatch(OE) > >>> sigma_serial(O) pmcint(O) guest_kbox_ram(O) signo_catch(O) mlx4_ib(OVE) > >>> mlx4_en(OVE)
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Hi, Kai Huang and Xiao Guangrong. For the problem mentioned above, there may be a bug related to PML and probably on Broadwell CPUs. I've been reading the code for PML for days, but I haven't found any clews. Do you have any idea about this BUG ? Hope you can help! On 2017/2/10 23:28, Chris Friesen wrote: > > Well, not so much solved as worked around it. > > It seems that the problem only showed up on Broadwell, which made us wonder > about something hardware specific. > > Setting "kvm-intel.eptad=0" in the kernel boot args seems to mask the problem > for us. > > Chris > > On 02/10/2017 03:11 AM, Herongguang (Stephen) wrote: >> Hi, Chris Friesen, did you solve the problem? >> >> On 2017/2/9 22:37, Herongguang (Stephen) wrote: >>> Hi. >>> I had a problem when I just repeatedly live migrate a vm between two compute >>> nodes. >>> The phenomenon was that the KVM module was crashed and then the host >>> rebooted. >>> However I cannot reliably trigger this BUG. >>> >>> The backtrace is the same as >>> http://www.spinics.net/lists/kvm/msg138475.html. >>> >>> The crash is triggered when function __rmap_clear_dirty is invoked and an >>> invalid page(0x7f183000) is visited. >>> The value %rdi is 0x7f183000, which is obviously not a valid kernel >>> pointer for x86_64. >>> The assembly of __rmap_clear_dirty is: >>> 0xa04d9ac0 <__rmap_clear_dirty>:callq >>> 0x816543d0 >>> >>> 0xa04d9ac5 <__rmap_clear_dirty+5>: push %rbp >>> 0xa04d9ac6 <__rmap_clear_dirty+6>: mov %rsp,%rbp >>> 0xa04d9ac9 <__rmap_clear_dirty+9>: push %rbx >>> 0xa04d9aca <__rmap_clear_dirty+10>: sub $0x18,%rsp >>> 0xa04d9ace <__rmap_clear_dirty+14>: mov (%rsi),%rdi >>> 0xa04d9ad1 <__rmap_clear_dirty+17>: mov %gs:0x28,%rax >>> 0xa04d9ada <__rmap_clear_dirty+26>: mov %rax,-0x10(%rbp) >>> 0xa04d9ade <__rmap_clear_dirty+30>: xor %eax,%eax >>> 0xa04d9ae0 <__rmap_clear_dirty+32>: test %rdi,%rdi >>> 0xa04d9ae3 <__rmap_clear_dirty+35>: je 0xa04d9b78 >>> 0xa04d9ae9 <__rmap_clear_dirty+41>: test $0x1,%dil >>> 0xa04d9aed <__rmap_clear_dirty+45>: je 0xa04d9b98 >>> 0xa04d9af3 <__rmap_clear_dirty+51>: and >>> $0xfffe,%rdi >>> 0xa04d9af7 <__rmap_clear_dirty+55>: movl $0x0,-0x18(%rbp) >>> 0xa04d9afe <__rmap_clear_dirty+62>: mov %rdi,-0x20(%rbp) >>> 0xa04d9b02 <__rmap_clear_dirty+66>: mov (%rdi),%rdi >>> 0xa04d9b05 <__rmap_clear_dirty+69>: test %rdi,%rdi >>> ... >>> >>> The details of the environment is: >>> Host info: x86_64 CentOS 7 kernel (3.10.0-327.36.58.10_2.x86_64, slightly >>> modified). The CPU is Broadwell Intel(R) Xeon(R) CPU E5-2618L v4 @ 2.20GHz. >>> Kmod info: version is 4.4.36 >>> What I know is that the parameter PML(Page Modification Logging) is enabled >>> by >>> default. >>> # cat /sys/module/kvm_intel/parameters/pml >>> # Y >>> >>> Below is the crash message: >>> [1548777.924180] kvm: zapping shadow pages for mmio generation wraparound >>> [1548777.947844] HTB: quantum of class 10001 is big. Consider r2q change. >>> [1548778.185389] kvm: zapping shadow pages for mmio generation wraparound >>> [1548778.994801] BUG: unable to handle kernel paging request at >>> 7f183000 >>> [1548779.002135] IP: [] __rmap_clear_dirty+0x4a/0xf0 [kvm] >>> [1548779.009151] PGD 1f8452067 PUD 0 >>> [1548779.012774] Thread overran stack, or stack corrupted >>> [1548779.018076] Oops: [#1] SMP >>> [1548779.027050] collected_len = 1048570, LOG_BUF_LEN_LOCAL = 1048576 >>> [1548779.042039] kbox: no notify die func register. no need to notify >>> [1548779.048392] do nothing after die! >>> [1548779.052071] Modules linked in: kboxdriver(O) kbox(O) sch_htb >>> ip_set_hash_net ip6table_filter ip6_tables iptable_filter igb_uio(OE) uio >>> bridge dm_service_time dm_multipath iscsi_tcp libiscsi_tcp libiscsi >>> scsi_transport_iscsi 8021q garp stp mrp llc vfat fat isofs ext4 jbd2 xfs >>> sha512_generic dev_connlimit(O) bum(O) ip_set nfnetlink prio(O) nat(O) >>> vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre ib_uverbs(OVE) hotpatch(OE) >>> sigma_serial(O) pmcint(O) guest_kbox_ram(O) signo_catch(O) mlx4_ib(OVE) >>> mlx4_en(OVE) vxlan ip6_udp_tunnel udp_tunnel ib_sa(OVE) ib_mad(OVE) ptp >>> ib_core(OVE) ib_addr(OVE) pps_core ib_netlink(OVE) intel_powerclamp coretemp >>> intel_rapl crc32_pclmul crc32c_intel ipmi_devintf(OVE) ghash_clmulni_intel >>> mlx4_core(OVE) aesni_intel kvm_intel(O) igb(OVE) lrw gf128mul i2c_algo_bit >>> glue_helper >>> [1548779.125577] iTCO_wdt ablk_helper kvm(O) iTCO_vendor_support sg cryptd >>> compat(OVE) dca sb_edac i2c_i801 kbox_pci(OVE) i2c_core edac_core pcspkr >>> lpc_ich shpchp mfd_core ipmi_si(OVE) ipmi_msghandler(OVE)
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Hi, Chris Friesen. We notice that you can reliably trigger the BUG during the live-migration stress test, however we can't. Could you descripe your test steps so that we can re-trigger the BUG and get more information about it ? On 2017/2/10 23:28, Chris Friesen wrote: > > Well, not so much solved as worked around it. > > It seems that the problem only showed up on Broadwell, which made us wonder > about something hardware specific. > > Setting "kvm-intel.eptad=0" in the kernel boot args seems to mask the problem > for us. > > Chris > > On 02/10/2017 03:11 AM, Herongguang (Stephen) wrote: >> Hi, Chris Friesen, did you solve the problem? >> >> On 2017/2/9 22:37, Herongguang (Stephen) wrote: >>> Hi. >>> I had a problem when I just repeatedly live migrate a vm between two compute >>> nodes. >>> The phenomenon was that the KVM module was crashed and then the host >>> rebooted. >>> However I cannot reliably trigger this BUG. >>> >>> The backtrace is the same as >>> http://www.spinics.net/lists/kvm/msg138475.html. >>> >>> The crash is triggered when function __rmap_clear_dirty is invoked and an >>> invalid page(0x7f183000) is visited. >>> The value %rdi is 0x7f183000, which is obviously not a valid kernel >>> pointer for x86_64. >>> The assembly of __rmap_clear_dirty is: >>> 0xa04d9ac0 <__rmap_clear_dirty>:callq >>> 0x816543d0 >>> >>> 0xa04d9ac5 <__rmap_clear_dirty+5>: push %rbp >>> 0xa04d9ac6 <__rmap_clear_dirty+6>: mov %rsp,%rbp >>> 0xa04d9ac9 <__rmap_clear_dirty+9>: push %rbx >>> 0xa04d9aca <__rmap_clear_dirty+10>: sub $0x18,%rsp >>> 0xa04d9ace <__rmap_clear_dirty+14>: mov (%rsi),%rdi >>> 0xa04d9ad1 <__rmap_clear_dirty+17>: mov %gs:0x28,%rax >>> 0xa04d9ada <__rmap_clear_dirty+26>: mov %rax,-0x10(%rbp) >>> 0xa04d9ade <__rmap_clear_dirty+30>: xor %eax,%eax >>> 0xa04d9ae0 <__rmap_clear_dirty+32>: test %rdi,%rdi >>> 0xa04d9ae3 <__rmap_clear_dirty+35>: je 0xa04d9b78 >>> 0xa04d9ae9 <__rmap_clear_dirty+41>: test $0x1,%dil >>> 0xa04d9aed <__rmap_clear_dirty+45>: je 0xa04d9b98 >>> 0xa04d9af3 <__rmap_clear_dirty+51>: and >>> $0xfffe,%rdi >>> 0xa04d9af7 <__rmap_clear_dirty+55>: movl $0x0,-0x18(%rbp) >>> 0xa04d9afe <__rmap_clear_dirty+62>: mov %rdi,-0x20(%rbp) >>> 0xa04d9b02 <__rmap_clear_dirty+66>: mov (%rdi),%rdi >>> 0xa04d9b05 <__rmap_clear_dirty+69>: test %rdi,%rdi >>> ... >>> >>> The details of the environment is: >>> Host info: x86_64 CentOS 7 kernel (3.10.0-327.36.58.10_2.x86_64, slightly >>> modified). The CPU is Broadwell Intel(R) Xeon(R) CPU E5-2618L v4 @ 2.20GHz. >>> Kmod info: version is 4.4.36 >>> What I know is that the parameter PML(Page Modification Logging) is enabled >>> by >>> default. >>> # cat /sys/module/kvm_intel/parameters/pml >>> # Y >>> >>> Below is the crash message: >>> [1548777.924180] kvm: zapping shadow pages for mmio generation wraparound >>> [1548777.947844] HTB: quantum of class 10001 is big. Consider r2q change. >>> [1548778.185389] kvm: zapping shadow pages for mmio generation wraparound >>> [1548778.994801] BUG: unable to handle kernel paging request at >>> 7f183000 >>> [1548779.002135] IP: [] __rmap_clear_dirty+0x4a/0xf0 [kvm] >>> [1548779.009151] PGD 1f8452067 PUD 0 >>> [1548779.012774] Thread overran stack, or stack corrupted >>> [1548779.018076] Oops: [#1] SMP >>> [1548779.027050] collected_len = 1048570, LOG_BUF_LEN_LOCAL = 1048576 >>> [1548779.042039] kbox: no notify die func register. no need to notify >>> [1548779.048392] do nothing after die! >>> [1548779.052071] Modules linked in: kboxdriver(O) kbox(O) sch_htb >>> ip_set_hash_net ip6table_filter ip6_tables iptable_filter igb_uio(OE) uio >>> bridge dm_service_time dm_multipath iscsi_tcp libiscsi_tcp libiscsi >>> scsi_transport_iscsi 8021q garp stp mrp llc vfat fat isofs ext4 jbd2 xfs >>> sha512_generic dev_connlimit(O) bum(O) ip_set nfnetlink prio(O) nat(O) >>> vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre ib_uverbs(OVE) hotpatch(OE) >>> sigma_serial(O) pmcint(O) guest_kbox_ram(O) signo_catch(O) mlx4_ib(OVE) >>> mlx4_en(OVE) vxlan ip6_udp_tunnel udp_tunnel ib_sa(OVE) ib_mad(OVE) ptp >>> ib_core(OVE) ib_addr(OVE) pps_core ib_netlink(OVE) intel_powerclamp coretemp >>> intel_rapl crc32_pclmul crc32c_intel ipmi_devintf(OVE) ghash_clmulni_intel >>> mlx4_core(OVE) aesni_intel kvm_intel(O) igb(OVE) lrw gf128mul i2c_algo_bit >>> glue_helper >>> [1548779.125577] iTCO_wdt ablk_helper kvm(O) iTCO_vendor_support sg cryptd >>> compat(OVE) dca sb_edac i2c_i801 kbox_pci(OVE) i2c_core edac_core pcspkr >>> lpc_ich shpchp mfd_core ipmi_si(OVE) ipmi_msghandler(OVE) nf_conntrack_ipv4 >>> nf_defrag_ipv4
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
> Well, not so much solved as worked around it. > > It seems that the problem only showed up on Broadwell, which made us wonder > about something hardware specific. Yes, the bug seems to be related to PML. I am looking at the code, but I haven't found anything yet. Paolo
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Well, not so much solved as worked around it. It seems that the problem only showed up on Broadwell, which made us wonder about something hardware specific. Setting "kvm-intel.eptad=0" in the kernel boot args seems to mask the problem for us. Chris On 02/10/2017 03:11 AM, Herongguang (Stephen) wrote: Hi, Chris Friesen, did you solve the problem? On 2017/2/9 22:37, Herongguang (Stephen) wrote: Hi. I had a problem when I just repeatedly live migrate a vm between two compute nodes. The phenomenon was that the KVM module was crashed and then the host rebooted. However I cannot reliably trigger this BUG. The backtrace is the same as http://www.spinics.net/lists/kvm/msg138475.html. The crash is triggered when function __rmap_clear_dirty is invoked and an invalid page(0x7f183000) is visited. The value %rdi is 0x7f183000, which is obviously not a valid kernel pointer for x86_64. The assembly of __rmap_clear_dirty is: 0xa04d9ac0 <__rmap_clear_dirty>:callq 0x816543d0 0xa04d9ac5 <__rmap_clear_dirty+5>: push %rbp 0xa04d9ac6 <__rmap_clear_dirty+6>: mov %rsp,%rbp 0xa04d9ac9 <__rmap_clear_dirty+9>: push %rbx 0xa04d9aca <__rmap_clear_dirty+10>: sub $0x18,%rsp 0xa04d9ace <__rmap_clear_dirty+14>: mov (%rsi),%rdi 0xa04d9ad1 <__rmap_clear_dirty+17>: mov %gs:0x28,%rax 0xa04d9ada <__rmap_clear_dirty+26>: mov %rax,-0x10(%rbp) 0xa04d9ade <__rmap_clear_dirty+30>: xor %eax,%eax 0xa04d9ae0 <__rmap_clear_dirty+32>: test %rdi,%rdi 0xa04d9ae3 <__rmap_clear_dirty+35>: je 0xa04d9b78 0xa04d9ae9 <__rmap_clear_dirty+41>: test $0x1,%dil 0xa04d9aed <__rmap_clear_dirty+45>: je 0xa04d9b98 0xa04d9af3 <__rmap_clear_dirty+51>: and $0xfffe,%rdi 0xa04d9af7 <__rmap_clear_dirty+55>: movl $0x0,-0x18(%rbp) 0xa04d9afe <__rmap_clear_dirty+62>: mov %rdi,-0x20(%rbp) 0xa04d9b02 <__rmap_clear_dirty+66>: mov (%rdi),%rdi 0xa04d9b05 <__rmap_clear_dirty+69>: test %rdi,%rdi ... The details of the environment is: Host info: x86_64 CentOS 7 kernel (3.10.0-327.36.58.10_2.x86_64, slightly modified). The CPU is Broadwell Intel(R) Xeon(R) CPU E5-2618L v4 @ 2.20GHz. Kmod info: version is 4.4.36 What I know is that the parameter PML(Page Modification Logging) is enabled by default. # cat /sys/module/kvm_intel/parameters/pml # Y Below is the crash message: [1548777.924180] kvm: zapping shadow pages for mmio generation wraparound [1548777.947844] HTB: quantum of class 10001 is big. Consider r2q change. [1548778.185389] kvm: zapping shadow pages for mmio generation wraparound [1548778.994801] BUG: unable to handle kernel paging request at 7f183000 [1548779.002135] IP: [] __rmap_clear_dirty+0x4a/0xf0 [kvm] [1548779.009151] PGD 1f8452067 PUD 0 [1548779.012774] Thread overran stack, or stack corrupted [1548779.018076] Oops: [#1] SMP [1548779.027050] collected_len = 1048570, LOG_BUF_LEN_LOCAL = 1048576 [1548779.042039] kbox: no notify die func register. no need to notify [1548779.048392] do nothing after die! [1548779.052071] Modules linked in: kboxdriver(O) kbox(O) sch_htb ip_set_hash_net ip6table_filter ip6_tables iptable_filter igb_uio(OE) uio bridge dm_service_time dm_multipath iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 8021q garp stp mrp llc vfat fat isofs ext4 jbd2 xfs sha512_generic dev_connlimit(O) bum(O) ip_set nfnetlink prio(O) nat(O) vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre ib_uverbs(OVE) hotpatch(OE) sigma_serial(O) pmcint(O) guest_kbox_ram(O) signo_catch(O) mlx4_ib(OVE) mlx4_en(OVE) vxlan ip6_udp_tunnel udp_tunnel ib_sa(OVE) ib_mad(OVE) ptp ib_core(OVE) ib_addr(OVE) pps_core ib_netlink(OVE) intel_powerclamp coretemp intel_rapl crc32_pclmul crc32c_intel ipmi_devintf(OVE) ghash_clmulni_intel mlx4_core(OVE) aesni_intel kvm_intel(O) igb(OVE) lrw gf128mul i2c_algo_bit glue_helper [1548779.125577] iTCO_wdt ablk_helper kvm(O) iTCO_vendor_support sg cryptd compat(OVE) dca sb_edac i2c_i801 kbox_pci(OVE) i2c_core edac_core pcspkr lpc_ich shpchp mfd_core ipmi_si(OVE) ipmi_msghandler(OVE) nf_conntrack_ipv4 nf_defrag_ipv4 vhost_net(O) tun(O) vhost(O) macvtap macvlan vfio_pci irqbypass vfio_iommu_type1 vfio xt_sctp nf_conntrack_proto_sctp nf_nat_proto_sctp nf_nat nf_conntrack sctp libcrc32c ip_tables ext3 mbcache jbd sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul crct10dif_common libahci mpt2sas(OVE) libata raid_class scsi_transport_sas dm_mod [last unloaded: kbox] [1548779.178704] CPU: 35 PID: 33769 Comm: migration Tainted: G OE V--- 3.10.0-327.36.58.10_2.x86_64 #1 [1548779.189855] Hardware name: HUAWEI TECHNOLOGIES CO.,LTD. CH80GPUB8/CH80GPUB8, BIOS GPUBV201 06/18/2015 [1548779.199585] task:
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Hi, Chris Friesen, did you solve the problem? On 2017/2/9 22:37, Herongguang (Stephen) wrote: Hi. I had a problem when I just repeatedly live migrate a vm between two compute nodes. The phenomenon was that the KVM module was crashed and then the host rebooted. However I cannot reliably trigger this BUG. The backtrace is the same as http://www.spinics.net/lists/kvm/msg138475.html. The crash is triggered when function __rmap_clear_dirty is invoked and an invalid page(0x7f183000) is visited. The value %rdi is 0x7f183000, which is obviously not a valid kernel pointer for x86_64. The assembly of __rmap_clear_dirty is: 0xa04d9ac0 <__rmap_clear_dirty>:callq 0x816543d0 0xa04d9ac5 <__rmap_clear_dirty+5>: push %rbp 0xa04d9ac6 <__rmap_clear_dirty+6>: mov %rsp,%rbp 0xa04d9ac9 <__rmap_clear_dirty+9>: push %rbx 0xa04d9aca <__rmap_clear_dirty+10>: sub $0x18,%rsp 0xa04d9ace <__rmap_clear_dirty+14>: mov (%rsi),%rdi 0xa04d9ad1 <__rmap_clear_dirty+17>: mov %gs:0x28,%rax 0xa04d9ada <__rmap_clear_dirty+26>: mov %rax,-0x10(%rbp) 0xa04d9ade <__rmap_clear_dirty+30>: xor %eax,%eax 0xa04d9ae0 <__rmap_clear_dirty+32>: test %rdi,%rdi 0xa04d9ae3 <__rmap_clear_dirty+35>: je 0xa04d9b78 0xa04d9ae9 <__rmap_clear_dirty+41>: test $0x1,%dil 0xa04d9aed <__rmap_clear_dirty+45>: je 0xa04d9b98 0xa04d9af3 <__rmap_clear_dirty+51>: and $0xfffe,%rdi 0xa04d9af7 <__rmap_clear_dirty+55>: movl $0x0,-0x18(%rbp) 0xa04d9afe <__rmap_clear_dirty+62>: mov %rdi,-0x20(%rbp) 0xa04d9b02 <__rmap_clear_dirty+66>: mov (%rdi),%rdi 0xa04d9b05 <__rmap_clear_dirty+69>: test %rdi,%rdi ... The details of the environment is: Host info: x86_64 CentOS 7 kernel (3.10.0-327.36.58.10_2.x86_64, slightly modified). The CPU is Broadwell Intel(R) Xeon(R) CPU E5-2618L v4 @ 2.20GHz. Kmod info: version is 4.4.36 What I know is that the parameter PML(Page Modification Logging) is enabled by default. # cat /sys/module/kvm_intel/parameters/pml # Y Below is the crash message: [1548777.924180] kvm: zapping shadow pages for mmio generation wraparound [1548777.947844] HTB: quantum of class 10001 is big. Consider r2q change. [1548778.185389] kvm: zapping shadow pages for mmio generation wraparound [1548778.994801] BUG: unable to handle kernel paging request at 7f183000 [1548779.002135] IP: [] __rmap_clear_dirty+0x4a/0xf0 [kvm] [1548779.009151] PGD 1f8452067 PUD 0 [1548779.012774] Thread overran stack, or stack corrupted [1548779.018076] Oops: [#1] SMP [1548779.027050] collected_len = 1048570, LOG_BUF_LEN_LOCAL = 1048576 [1548779.042039] kbox: no notify die func register. no need to notify [1548779.048392] do nothing after die! [1548779.052071] Modules linked in: kboxdriver(O) kbox(O) sch_htb ip_set_hash_net ip6table_filter ip6_tables iptable_filter igb_uio(OE) uio bridge dm_service_time dm_multipath iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 8021q garp stp mrp llc vfat fat isofs ext4 jbd2 xfs sha512_generic dev_connlimit(O) bum(O) ip_set nfnetlink prio(O) nat(O) vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre ib_uverbs(OVE) hotpatch(OE) sigma_serial(O) pmcint(O) guest_kbox_ram(O) signo_catch(O) mlx4_ib(OVE) mlx4_en(OVE) vxlan ip6_udp_tunnel udp_tunnel ib_sa(OVE) ib_mad(OVE) ptp ib_core(OVE) ib_addr(OVE) pps_core ib_netlink(OVE) intel_powerclamp coretemp intel_rapl crc32_pclmul crc32c_intel ipmi_devintf(OVE) ghash_clmulni_intel mlx4_core(OVE) aesni_intel kvm_intel(O) igb(OVE) lrw gf128mul i2c_algo_bit glue_helper [1548779.125577] iTCO_wdt ablk_helper kvm(O) iTCO_vendor_support sg cryptd compat(OVE) dca sb_edac i2c_i801 kbox_pci(OVE) i2c_core edac_core pcspkr lpc_ich shpchp mfd_core ipmi_si(OVE) ipmi_msghandler(OVE) nf_conntrack_ipv4 nf_defrag_ipv4 vhost_net(O) tun(O) vhost(O) macvtap macvlan vfio_pci irqbypass vfio_iommu_type1 vfio xt_sctp nf_conntrack_proto_sctp nf_nat_proto_sctp nf_nat nf_conntrack sctp libcrc32c ip_tables ext3 mbcache jbd sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul crct10dif_common libahci mpt2sas(OVE) libata raid_class scsi_transport_sas dm_mod [last unloaded: kbox] [1548779.178704] CPU: 35 PID: 33769 Comm: migration Tainted: G OE V--- 3.10.0-327.36.58.10_2.x86_64 #1 [1548779.189855] Hardware name: HUAWEI TECHNOLOGIES CO.,LTD. CH80GPUB8/CH80GPUB8, BIOS GPUBV201 06/18/2015 [1548779.199585] task: 880123da ti: 8801a8068000 task.ti: 8801a8068000 [1548779.207596] RIP: 0010:[] [] __rmap_clear_dirty+0x4a/0xf0 [kvm] [1548779.217209] RSP: 0018:8801a806bba0 EFLAGS: 00010206 [1548779.222879] RAX: RBX: RCX:
Re: [Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Hi, Chris Friesen, did you solve the problem? On 2017/2/9 22:37, Herongguang (Stephen) wrote: Hi. I had a problem when I just repeatedly live migrate a vm between two compute nodes. The phenomenon was that the KVM module was crashed and then the host rebooted. However I cannot reliably trigger this BUG. The backtrace is the same as http://www.spinics.net/lists/kvm/msg138475.html. The crash is triggered when function __rmap_clear_dirty is invoked and an invalid page(0x7f183000) is visited. The value %rdi is 0x7f183000, which is obviously not a valid kernel pointer for x86_64. The assembly of __rmap_clear_dirty is: 0xa04d9ac0 <__rmap_clear_dirty>:callq 0x816543d0 0xa04d9ac5 <__rmap_clear_dirty+5>: push %rbp 0xa04d9ac6 <__rmap_clear_dirty+6>: mov %rsp,%rbp 0xa04d9ac9 <__rmap_clear_dirty+9>: push %rbx 0xa04d9aca <__rmap_clear_dirty+10>: sub $0x18,%rsp 0xa04d9ace <__rmap_clear_dirty+14>: mov (%rsi),%rdi 0xa04d9ad1 <__rmap_clear_dirty+17>: mov %gs:0x28,%rax 0xa04d9ada <__rmap_clear_dirty+26>: mov %rax,-0x10(%rbp) 0xa04d9ade <__rmap_clear_dirty+30>: xor %eax,%eax 0xa04d9ae0 <__rmap_clear_dirty+32>: test %rdi,%rdi 0xa04d9ae3 <__rmap_clear_dirty+35>: je 0xa04d9b78 0xa04d9ae9 <__rmap_clear_dirty+41>: test $0x1,%dil 0xa04d9aed <__rmap_clear_dirty+45>: je 0xa04d9b98 0xa04d9af3 <__rmap_clear_dirty+51>: and $0xfffe,%rdi 0xa04d9af7 <__rmap_clear_dirty+55>: movl $0x0,-0x18(%rbp) 0xa04d9afe <__rmap_clear_dirty+62>: mov %rdi,-0x20(%rbp) 0xa04d9b02 <__rmap_clear_dirty+66>: mov (%rdi),%rdi 0xa04d9b05 <__rmap_clear_dirty+69>: test %rdi,%rdi ... The details of the environment is: Host info: x86_64 CentOS 7 kernel (3.10.0-327.36.58.10_2.x86_64, slightly modified). The CPU is Broadwell Intel(R) Xeon(R) CPU E5-2618L v4 @ 2.20GHz. Kmod info: version is 4.4.36 What I know is that the parameter PML(Page Modification Logging) is enabled by default. # cat /sys/module/kvm_intel/parameters/pml # Y Below is the crash message: [1548777.924180] kvm: zapping shadow pages for mmio generation wraparound [1548777.947844] HTB: quantum of class 10001 is big. Consider r2q change. [1548778.185389] kvm: zapping shadow pages for mmio generation wraparound [1548778.994801] BUG: unable to handle kernel paging request at 7f183000 [1548779.002135] IP: [] __rmap_clear_dirty+0x4a/0xf0 [kvm] [1548779.009151] PGD 1f8452067 PUD 0 [1548779.012774] Thread overran stack, or stack corrupted [1548779.018076] Oops: [#1] SMP [1548779.027050] collected_len = 1048570, LOG_BUF_LEN_LOCAL = 1048576 [1548779.042039] kbox: no notify die func register. no need to notify [1548779.048392] do nothing after die! [1548779.052071] Modules linked in: kboxdriver(O) kbox(O) sch_htb ip_set_hash_net ip6table_filter ip6_tables iptable_filter igb_uio(OE) uio bridge dm_service_time dm_multipath iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 8021q garp stp mrp llc vfat fat isofs ext4 jbd2 xfs sha512_generic dev_connlimit(O) bum(O) ip_set nfnetlink prio(O) nat(O) vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre ib_uverbs(OVE) hotpatch(OE) sigma_serial(O) pmcint(O) guest_kbox_ram(O) signo_catch(O) mlx4_ib(OVE) mlx4_en(OVE) vxlan ip6_udp_tunnel udp_tunnel ib_sa(OVE) ib_mad(OVE) ptp ib_core(OVE) ib_addr(OVE) pps_core ib_netlink(OVE) intel_powerclamp coretemp intel_rapl crc32_pclmul crc32c_intel ipmi_devintf(OVE) ghash_clmulni_intel mlx4_core(OVE) aesni_intel kvm_intel(O) igb(OVE) lrw gf128mul i2c_algo_bit glue_helper [1548779.125577] iTCO_wdt ablk_helper kvm(O) iTCO_vendor_support sg cryptd compat(OVE) dca sb_edac i2c_i801 kbox_pci(OVE) i2c_core edac_core pcspkr lpc_ich shpchp mfd_core ipmi_si(OVE) ipmi_msghandler(OVE) nf_conntrack_ipv4 nf_defrag_ipv4 vhost_net(O) tun(O) vhost(O) macvtap macvlan vfio_pci irqbypass vfio_iommu_type1 vfio xt_sctp nf_conntrack_proto_sctp nf_nat_proto_sctp nf_nat nf_conntrack sctp libcrc32c ip_tables ext3 mbcache jbd sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul crct10dif_common libahci mpt2sas(OVE) libata raid_class scsi_transport_sas dm_mod [last unloaded: kbox] [1548779.178704] CPU: 35 PID: 33769 Comm: migration Tainted: G OE V--- 3.10.0-327.36.58.10_2.x86_64 #1 [1548779.189855] Hardware name: HUAWEI TECHNOLOGIES CO.,LTD. CH80GPUB8/CH80GPUB8, BIOS GPUBV201 06/18/2015 [1548779.199585] task: 880123da ti: 8801a8068000 task.ti: 8801a8068000 [1548779.207596] RIP: 0010:[] [] __rmap_clear_dirty+0x4a/0xf0 [kvm] [1548779.217209] RSP: 0018:8801a806bba0 EFLAGS: 00010206 [1548779.222879] RAX: RBX: RCX: [1548779.230543] RDX:
[Qemu-devel] kvm bug in __rmap_clear_dirty during live migration
Hi. I had a problem when I just repeatedly live migrate a vm between two compute nodes. The phenomenon was that the KVM module was crashed and then the host rebooted. However I cannot reliably trigger this BUG. The backtrace is the same as http://www.spinics.net/lists/kvm/msg138475.html. The crash is triggered when function __rmap_clear_dirty is invoked and an invalid page(0x7f183000) is visited. The value %rdi is 0x7f183000, which is obviously not a valid kernel pointer for x86_64. The assembly of __rmap_clear_dirty is: 0xa04d9ac0 <__rmap_clear_dirty>:callq 0x816543d0 0xa04d9ac5 <__rmap_clear_dirty+5>: push %rbp 0xa04d9ac6 <__rmap_clear_dirty+6>: mov %rsp,%rbp 0xa04d9ac9 <__rmap_clear_dirty+9>: push %rbx 0xa04d9aca <__rmap_clear_dirty+10>: sub $0x18,%rsp 0xa04d9ace <__rmap_clear_dirty+14>: mov (%rsi),%rdi 0xa04d9ad1 <__rmap_clear_dirty+17>: mov %gs:0x28,%rax 0xa04d9ada <__rmap_clear_dirty+26>: mov %rax,-0x10(%rbp) 0xa04d9ade <__rmap_clear_dirty+30>: xor %eax,%eax 0xa04d9ae0 <__rmap_clear_dirty+32>: test %rdi,%rdi 0xa04d9ae3 <__rmap_clear_dirty+35>: je 0xa04d9b78 0xa04d9ae9 <__rmap_clear_dirty+41>: test $0x1,%dil 0xa04d9aed <__rmap_clear_dirty+45>: je 0xa04d9b98 0xa04d9af3 <__rmap_clear_dirty+51>: and $0xfffe,%rdi 0xa04d9af7 <__rmap_clear_dirty+55>: movl $0x0,-0x18(%rbp) 0xa04d9afe <__rmap_clear_dirty+62>: mov %rdi,-0x20(%rbp) 0xa04d9b02 <__rmap_clear_dirty+66>: mov (%rdi),%rdi 0xa04d9b05 <__rmap_clear_dirty+69>: test %rdi,%rdi ... The details of the environment is: Host info: x86_64 CentOS 7 kernel (3.10.0-327.36.58.10_2.x86_64, slightly modified). The CPU is Broadwell Intel(R) Xeon(R) CPU E5-2618L v4 @ 2.20GHz. Kmod info: version is 4.4.36 What I know is that the parameter PML(Page Modification Logging) is enabled by default. # cat /sys/module/kvm_intel/parameters/pml # Y Below is the crash message: [1548777.924180] kvm: zapping shadow pages for mmio generation wraparound [1548777.947844] HTB: quantum of class 10001 is big. Consider r2q change. [1548778.185389] kvm: zapping shadow pages for mmio generation wraparound [1548778.994801] BUG: unable to handle kernel paging request at 7f183000 [1548779.002135] IP: [] __rmap_clear_dirty+0x4a/0xf0 [kvm] [1548779.009151] PGD 1f8452067 PUD 0 [1548779.012774] Thread overran stack, or stack corrupted [1548779.018076] Oops: [#1] SMP [1548779.027050] collected_len = 1048570, LOG_BUF_LEN_LOCAL = 1048576 [1548779.042039] kbox: no notify die func register. no need to notify [1548779.048392] do nothing after die! [1548779.052071] Modules linked in: kboxdriver(O) kbox(O) sch_htb ip_set_hash_net ip6table_filter ip6_tables iptable_filter igb_uio(OE) uio bridge dm_service_time dm_multipath iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 8021q garp stp mrp llc vfat fat isofs ext4 jbd2 xfs sha512_generic dev_connlimit(O) bum(O) ip_set nfnetlink prio(O) nat(O) vport_vxlan(O) openvswitch(O) nf_defrag_ipv6 gre ib_uverbs(OVE) hotpatch(OE) sigma_serial(O) pmcint(O) guest_kbox_ram(O) signo_catch(O) mlx4_ib(OVE) mlx4_en(OVE) vxlan ip6_udp_tunnel udp_tunnel ib_sa(OVE) ib_mad(OVE) ptp ib_core(OVE) ib_addr(OVE) pps_core ib_netlink(OVE) intel_powerclamp coretemp intel_rapl crc32_pclmul crc32c_intel ipmi_devintf(OVE) ghash_clmulni_intel mlx4_core(OVE) aesni_intel kvm_intel(O) igb(OVE) lrw gf128mul i2c_algo_bit glue_helper [1548779.125577] iTCO_wdt ablk_helper kvm(O) iTCO_vendor_support sg cryptd compat(OVE) dca sb_edac i2c_i801 kbox_pci(OVE) i2c_core edac_core pcspkr lpc_ich shpchp mfd_core ipmi_si(OVE) ipmi_msghandler(OVE) nf_conntrack_ipv4 nf_defrag_ipv4 vhost_net(O) tun(O) vhost(O) macvtap macvlan vfio_pci irqbypass vfio_iommu_type1 vfio xt_sctp nf_conntrack_proto_sctp nf_nat_proto_sctp nf_nat nf_conntrack sctp libcrc32c ip_tables ext3 mbcache jbd sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul crct10dif_common libahci mpt2sas(OVE) libata raid_class scsi_transport_sas dm_mod [last unloaded: kbox] [1548779.178704] CPU: 35 PID: 33769 Comm: migration Tainted: G OE V--- 3.10.0-327.36.58.10_2.x86_64 #1 [1548779.189855] Hardware name: HUAWEI TECHNOLOGIES CO.,LTD. CH80GPUB8/CH80GPUB8, BIOS GPUBV201 06/18/2015 [1548779.199585] task: 880123da ti: 8801a8068000 task.ti: 8801a8068000 [1548779.207596] RIP: 0010:[] [] __rmap_clear_dirty+0x4a/0xf0 [kvm] [1548779.217209] RSP: 0018:8801a806bba0 EFLAGS: 00010206 [1548779.222879] RAX: RBX: RCX: [1548779.230543] RDX: c90059d64000 RSI: c90059f2efd0 RDI: 7f183000 [1548779.238211] RBP: 8801a806bbc0
