Re: [Xen-devel] [PATCH v4] xen: avoid crash in disable_hotplug_cpu
On 9/7/18 10:31 AM, Olaf Hering wrote: > The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0: > > BUG: unable to handle kernel NULL pointer dereference at 02d8 > PGD 0 P4D 0 > Oops: [#1] PREEMPT SMP NOPTI > CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 > openSUSE Tumbleweed (unreleased) > Hardware name: Intel Corporation S5520UR/S5520UR, BIOS > S5500.86B.01.00.0050.050620101605 05/06/2010 > RIP: e030:device_offline+0x9/0xb0 > Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff > 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 87 d8 02 00 00 01 > 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6 > RSP: e02b:c90040f27e80 EFLAGS: 00010203 > RAX: RBX: RCX: > RDX: 8801f380 RSI: c90040f27e70 RDI: > RBP: R08: 820e47b3 R09: > R10: 7ff0 R11: R12: 822e6d30 > R13: dead0200 R14: dead0100 R15: 8158b4e0 > FS: 7ffa595158c0() GS:8801f39c() knlGS: > CS: e033 DS: ES: CR0: 80050033 > CR2: 02d8 CR3: 0001d9602000 CR4: 2660 > Call Trace: > handle_vcpu_hotplug_event+0xb5/0xc0 > xenwatch_thread+0x80/0x140 > ? wait_woken+0x80/0x80 > kthread+0x112/0x130 > ? kthread_create_worker_on_cpu+0x40/0x40 > ret_from_fork+0x3a/0x50 > > This happens because handle_vcpu_hotplug_event is called twice. In the > first iteration cpu_present is still true, in the second iteration > cpu_present is false which causes get_cpu_device to return NULL. > In case of cpu#0, cpu_online is apparently always true. > > Fix this crash by checking if the cpu can be hotplugged, which is false > for a cpu that was just removed. > > Also check if the cpu was actually offlined by device_remove, otherwise > leave the cpu_present state as it is. > > Rearrange to code to do all work with device_hotplug_lock held. > > Signed-off-by: Olaf Hering Applied to for-linus-4.19b -boris ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4] xen: avoid crash in disable_hotplug_cpu
On 11/09/18 12:48, Olaf Hering wrote: > Am Tue, 11 Sep 2018 09:52:58 +0200 > schrieb Olaf Hering : > >> As Jürgen suggested in another mail, Xen should probably disable hotplugging >> for cpu#0 in the generic setup code. Then cpu_is_hotpluggable(cpu) would >> do the right thing. > > The relevant code is all private to arch/x86/kernel/topology.c. > There is no way to set .hotpluggable prior to register_cpu. > I guess you need to work that out with the x86 maintainers. Okay, I'll do that. Juergen ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4] xen: avoid crash in disable_hotplug_cpu
Am Tue, 11 Sep 2018 09:52:58 +0200 schrieb Olaf Hering : > As Jürgen suggested in another mail, Xen should probably disable hotplugging > for cpu#0 in the generic setup code. Then cpu_is_hotpluggable(cpu) would > do the right thing. The relevant code is all private to arch/x86/kernel/topology.c. There is no way to set .hotpluggable prior to register_cpu. I guess you need to work that out with the x86 maintainers. Olaf pgp0HN0c9KZcE.pgp Description: Digitale Signatur von OpenPGP ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4] xen: avoid crash in disable_hotplug_cpu
On 11/09/18 09:52, Olaf Hering wrote: > Am Fri, 7 Sep 2018 12:56:37 -0400 > schrieb Boris Ostrovsky : > >> I was hoping you'd respond to my question about warning. > > It looks like CONFIG_BOOTPARAM_HOTPLUG_CPU0=y is the reason for the warning. > As Jürgen suggested in another mail, Xen should probably disable hotplugging > for cpu#0 in the generic setup code. Then cpu_is_hotpluggable(cpu) would > do the right thing. > > Do you want me to merge that extra change into this patch? I think an extra patch would be better, as it is a different issue. Juergen ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4] xen: avoid crash in disable_hotplug_cpu
Am Fri, 7 Sep 2018 12:56:37 -0400 schrieb Boris Ostrovsky : > I was hoping you'd respond to my question about warning. It looks like CONFIG_BOOTPARAM_HOTPLUG_CPU0=y is the reason for the warning. As Jürgen suggested in another mail, Xen should probably disable hotplugging for cpu#0 in the generic setup code. Then cpu_is_hotpluggable(cpu) would do the right thing. Do you want me to merge that extra change into this patch? Olaf pgpCm3C0Q1jVM.pgp Description: Digitale Signatur von OpenPGP ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4] xen: avoid crash in disable_hotplug_cpu
Am Fri, 7 Sep 2018 12:56:37 -0400 schrieb Boris Ostrovsky : > I was hoping you'd respond to my question about warning. > > root@haswell> xl vcpu-set 3 0 > and in the guest > > [root@vm-0238 ~]# [ 32.866955] [ cut here ] > [ 32.866963] spinlock on CPU0 exists on IRQ1! > [ 32.866984] WARNING: CPU: 0 PID: 14 at arch/x86/xen/spinlock.c:90 > xen_init_lock_cpu+0xbf/0xd0 This happens to work for me, on X5550. Please send your .config. Olaf pgpofLFY4MBHD.pgp Description: Digitale Signatur von OpenPGP ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4] xen: avoid crash in disable_hotplug_cpu
On 09/07/2018 10:31 AM, Olaf Hering wrote: > The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0: > > BUG: unable to handle kernel NULL pointer dereference at 02d8 > PGD 0 P4D 0 > Oops: [#1] PREEMPT SMP NOPTI > CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 > openSUSE Tumbleweed (unreleased) > Hardware name: Intel Corporation S5520UR/S5520UR, BIOS > S5500.86B.01.00.0050.050620101605 05/06/2010 > RIP: e030:device_offline+0x9/0xb0 > Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff > 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 87 d8 02 00 00 01 > 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6 > RSP: e02b:c90040f27e80 EFLAGS: 00010203 > RAX: RBX: RCX: > RDX: 8801f380 RSI: c90040f27e70 RDI: > RBP: R08: 820e47b3 R09: > R10: 7ff0 R11: R12: 822e6d30 > R13: dead0200 R14: dead0100 R15: 8158b4e0 > FS: 7ffa595158c0() GS:8801f39c() knlGS: > CS: e033 DS: ES: CR0: 80050033 > CR2: 02d8 CR3: 0001d9602000 CR4: 2660 > Call Trace: > handle_vcpu_hotplug_event+0xb5/0xc0 > xenwatch_thread+0x80/0x140 > ? wait_woken+0x80/0x80 > kthread+0x112/0x130 > ? kthread_create_worker_on_cpu+0x40/0x40 > ret_from_fork+0x3a/0x50 > > This happens because handle_vcpu_hotplug_event is called twice. In the > first iteration cpu_present is still true, in the second iteration > cpu_present is false which causes get_cpu_device to return NULL. > In case of cpu#0, cpu_online is apparently always true. > > Fix this crash by checking if the cpu can be hotplugged, which is false > for a cpu that was just removed. > > Also check if the cpu was actually offlined by device_remove, otherwise > leave the cpu_present state as it is. > > Rearrange to code to do all work with device_hotplug_lock held. > > Signed-off-by: Olaf Hering > --- > drivers/xen/cpu_hotplug.c | 15 --- > 1 file changed, 8 insertions(+), 7 deletions(-) > > diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c > index d4265c8ebb22..b1357aa4bc55 100644 > --- a/drivers/xen/cpu_hotplug.c > +++ b/drivers/xen/cpu_hotplug.c > @@ -19,15 +19,16 @@ static void enable_hotplug_cpu(int cpu) > > static void disable_hotplug_cpu(int cpu) > { > - if (cpu_online(cpu)) { > - lock_device_hotplug(); > + if (!cpu_is_hotpluggable(cpu)) > + return; > + lock_device_hotplug(); > + if (cpu_online(cpu)) > device_offline(get_cpu_device(cpu)); > - unlock_device_hotplug(); > - } > - if (cpu_present(cpu)) > + if (!cpu_online(cpu) && cpu_present(cpu)) { > xen_arch_unregister_cpu(cpu); > - > - set_cpu_present(cpu, false); > + set_cpu_present(cpu, false); > + } > + unlock_device_hotplug(); > } > > static int vcpu_online(unsigned int cpu) I was hoping you'd respond to my question about warning. root@haswell> xl vcpu-set 3 0 and in the guest [root@vm-0238 ~]# [ 32.866955] [ cut here ] [ 32.866963] spinlock on CPU0 exists on IRQ1! [ 32.866984] WARNING: CPU: 0 PID: 14 at arch/x86/xen/spinlock.c:90 xen_init_lock_cpu+0xbf/0xd0 [ 32.866990] Modules linked in: [ 32.866995] CPU: 0 PID: 14 Comm: cpuhp/0 Not tainted 4.19.0-rc2 #31 [ 32.867001] RIP: e030:xen_init_lock_cpu+0xbf/0xd0 [ 32.867005] Code: 4a 8b 0c e5 00 c7 14 82 48 c7 c2 90 4f 01 00 4c 89 2c 11 e9 85 00 00 00 8b 14 02 44 89 e6 48 c7 c7 a0 0f 08 82 e8 ab e3 05 00 <0f> 0b e9 7a ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 80 3d 59 02 20 [ 32.867015] RSP: e02b:c900401ffe40 EFLAGS: 00010286 [ 32.867019] RAX: RBX: RCX: 0006 [ 32.867024] RDX: 0007 RSI: 0001 RDI: 88003d8168b0 [ 32.867039] RBP: 00014f98 R08: 81eb04a0 R09: 7f9b [ 32.867045] R10: 0065 R11: 82a9b7cd R12: [ 32.867050] R13: 8101a820 R14: 88003d401280 R15: 810aec10 [ 32.867061] FS: () GS:88003d80() knlGS: [ 32.867066] CS: e033 DS: ES: CR0: 80050033 [ 32.867081] CR2: 5569b64e72b8 CR3: 2e902000 CR4: 00042660 [ 32.867089] Call Trace: [ 32.867096] ? cstate_cleanup+0x47/0x47 [ 32.867101] xen_cpu_up_online+0xa/0x10 [ 32.867107] cpuhp_invoke_callback+0x8d/0x500 [ 32.867113] ? sort_range+0x20/0x20 [ 32.867117] cpuhp_thread_fun+0xb0/0x110 [ 32.867121] smpboot_thread_fn+0xc5/0x160 [ 32.867126] kthread+0x112/0x130 [ 32.867131] ? kthread_bind+0x30/0x30 [ 32.867136] ret_from_fork+0x35/0x40 [ 32.867141] ---[ end trace 15d4d7112a1b1cea ]--- [ 32.867148] genirq: Flags mismatch irq 1.
Re: [Xen-devel] [PATCH v4] xen: avoid crash in disable_hotplug_cpu
On 07/09/18 16:31, Olaf Hering wrote: > The command 'xl vcpu-set 0 0', issued in dom0, will crash dom0: > > BUG: unable to handle kernel NULL pointer dereference at 02d8 > PGD 0 P4D 0 > Oops: [#1] PREEMPT SMP NOPTI > CPU: 7 PID: 65 Comm: xenwatch Not tainted 4.19.0-rc2-1.ga9462db-default #1 > openSUSE Tumbleweed (unreleased) > Hardware name: Intel Corporation S5520UR/S5520UR, BIOS > S5500.86B.01.00.0050.050620101605 05/06/2010 > RIP: e030:device_offline+0x9/0xb0 > Code: 77 24 00 e9 ce fe ff ff 48 8b 13 e9 68 ff ff ff 48 8b 13 e9 29 ff ff ff > 48 8b 13 e9 ea fe ff ff 90 66 66 66 66 90 41 54 55 53 87 d8 02 00 00 01 > 0f 85 88 00 00 00 48 c7 c2 20 09 60 81 31 f6 > RSP: e02b:c90040f27e80 EFLAGS: 00010203 > RAX: RBX: RCX: > RDX: 8801f380 RSI: c90040f27e70 RDI: > RBP: R08: 820e47b3 R09: > R10: 7ff0 R11: R12: 822e6d30 > R13: dead0200 R14: dead0100 R15: 8158b4e0 > FS: 7ffa595158c0() GS:8801f39c() knlGS: > CS: e033 DS: ES: CR0: 80050033 > CR2: 02d8 CR3: 0001d9602000 CR4: 2660 > Call Trace: > handle_vcpu_hotplug_event+0xb5/0xc0 > xenwatch_thread+0x80/0x140 > ? wait_woken+0x80/0x80 > kthread+0x112/0x130 > ? kthread_create_worker_on_cpu+0x40/0x40 > ret_from_fork+0x3a/0x50 > > This happens because handle_vcpu_hotplug_event is called twice. In the > first iteration cpu_present is still true, in the second iteration > cpu_present is false which causes get_cpu_device to return NULL. > In case of cpu#0, cpu_online is apparently always true. > > Fix this crash by checking if the cpu can be hotplugged, which is false > for a cpu that was just removed. > > Also check if the cpu was actually offlined by device_remove, otherwise > leave the cpu_present state as it is. > > Rearrange to code to do all work with device_hotplug_lock held. > > Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross Juergen ___ Xen-devel mailing list Xen-devel@lists.xenproject.org https://lists.xenproject.org/mailman/listinfo/xen-devel