I'll give your suggestions I try. I need to move to a server that I can forcibly reboot remotely (to recover), so it will be a while.
david Avi Kivity wrote: > david ahern wrote: >> As a quick test I added a printk to the loop, right after the while(): >> >> while (atomic_read(&completed) != needed) { >> printk("kvm_flush_remote_tlbs: completed = %d, needed = %d\n", >> atomic_read(&completed), needed); >> cpu_relax(); >> barrier(); >> } >> >> >> This is the output right before a lockup: >> >> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 2, >> needed = 2 >> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 2, >> needed = 2 >> Oct 24 16:03:47 bldr-ccm20 kernel: kvm_flush_remote_tlbs: completed = 1, >> needed = 2 >> Oct 24 16:03:57 bldr-ccm20 last message repeated 105738 times >> Oct 24 16:03:57 bldr-ccm20 kernel: BUG: soft lockup detected on CPU#0! >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c044a0b7>] softlockup_tick+0x98/0xa6 >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c042cc98>] >> update_process_times+0x39/0x5c >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c04176ec>] >> smp_apic_timer_interrupt+0x5c/0x64 >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c04049bf>] >> apic_timer_interrupt+0x1f/0x24 >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c0424130>] vprintk+0x288/0x2bc >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c0459db7>] follow_page+0x168/0x1b6 >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c04d8067>] >> cfq_slice_async_store+0x5/0x38 >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c0459db7>] follow_page+0x168/0x1b6 >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c0406406>] do_IRQ+0xa5/0xae >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c040492e>] common_interrupt+0x1a/0x20 >> Oct 24 16:03:57 bldr-ccm20 kernel: [<c042417c>] printk+0x18/0x8e >> Oct 24 16:03:57 bldr-ccm20 kernel: [<f89a9812>] >> kvm_flush_remote_tlbs+0xe0/0xf2 [kvm] >> ... >> >> >> I'd like to get a solution for RHEL5, so I am attempting to backport >> smp_call_function_mask(). I'm open to other suggestions if you think it is >> corruption or the problem is somewhere else. >> > > No, it looks like the problem is indeed in kvm_flush_remote_tlbs(), and > not a corruption elsewhere. > > Things to check: > > - whether cpus_weight(mask) == needed > - whether wrapping the whole thing in preempt_disable()/preempt_enable() > helps > > hey! I see a bug! > >> continue; >> cpu = vcpu->cpu; >> if (cpu != -1 && cpu != raw_smp_processor_id()) >> if (!cpu_isset(cpu, cpus)) { >> cpu_set(cpu, cpus); >> ++needed; >> } >> } >> > > vcpu->cpu can change during execution if this snippet due to a vcpu > being migrated concurrently with this being executed. Since the > compiler is free to reload 'cpu' from 'vcpu->cpu', the code can operate > on corrupted data. > > A 'barrier();' after 'cpu = vcpu->cpu;' should fix it, if this is indeed > the bug. > > ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/ _______________________________________________ kvm-devel mailing list kvm-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/kvm-devel