Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Lan Tianyu]

On 2016年09月23日 13:26, Peter Xu wrote:
> On Thu, Sep 22, 2016 at 12:34:36PM +0800, Chao Gao wrote:
>> Hi, we had 3 problems left here.
>> 1. IRQremapping can't work with x2apic_cluster mode.
>> 2. apic_id > 255 can't receive devices interrupts.
>> 3. windows crash when present IRQremapping capability to it.
> 

Hi Peter:

Thanks for your information. We just want to make sure these issues
aren't ignored.

> For (3), I don't know whether it's urgent or not - I've put it into my
> todo list (assuming this is for HPC and mostly we are using Linux
> guests?).
> 
> For (1-2), again we may need to wait for Radim's patches.

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Peter Xu]

On Thu, Sep 22, 2016 at 12:34:36PM +0800, Chao Gao wrote:
> Hi, we had 3 problems left here.
> 1. IRQremapping can't work with x2apic_cluster mode.
> 2. apic_id > 255 can't receive devices interrupts.
> 3. windows crash when present IRQremapping capability to it.

For (3), I don't know whether it's urgent or not - I've put it into my
todo list (assuming this is for HPC and mostly we are using Linux
guests?).

For (1-2), again we may need to wait for Radim's patches.

Thanks.

-- peterx

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Chao Gao]

Hi, we had 3 problems left here.
1. IRQremapping can't work with x2apic_cluster mode.
2. apic_id > 255 can't receive devices interrupts.
3. windows crash when present IRQremapping capability to it.

I test with latest kernel v4.8-rc7 and find all of them are unsolved.
Also in the qemu and kernel code bases, I couldn't find some patches to 
solve them. Will you fix these problems or leave them to communities? 

Thanks,
-Chao

On Tue, Aug 09, 2016 at 02:51:16PM +0200, Radim Krčmář wrote:
>2016-08-09 16:19+0800, Chao Gao:
>> On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote:
>>>I think the problem is with x2apic. Currently, x2apic is enabled in
>>>vIOMMU when kernel irqchip is used. This is problematic, since
>>>actually we are throughing away dest_id[31:8] without Radim's patches,
>>>meanwhile I see that by default x2apic is using cluster mode.
>>>
>>>In cluster mode, 8 bits will possibly not suffice (I think the reason
>>>why >17 vcpus will bring trouble is that each cluster has 16 vcpus,
>>>we'll have trouble if we have more than one cluster).
>>>
>>>To temporarily solve your issue, you should not only need "-global
>>>ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys"
>>>to you guest kernel boot parameter, to force guest boot with x2apic
>>>physical mode (not cluster mode). Though this can only work for <255
>>>vcpus. IMHO we may still need to wait for Radim's patches to test >255
>>>case.
>> 
>> ok, after adding "x2apic_phys" to guest kernel boot parameter, I 
>> boot up a 288(yes, 288) vcpus guest successfully with command
>> qemu-system-x86_64 -boot c -m 4096 -sdl --enable-kvm \
>> -M kernel-irqchip=split -bios bios.bin -smp cpus=288 -hda vdisk.img \
>> -device intel-iommu,intremap=on -machine q35
>> Also, I can see interrupts on those cpu with inital apicid>255 from 
>> /proc/cpuinfo and /proc/interrupts. 
>
>Great, thanks for testing!
>Only IPIs will be correctly delivered to apic_id > 255 without few more
>patches on the QEMU side, though.
>

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Igor Mammedov]

On Thu, 11 Aug 2016 13:10:57 +0800
Peter Xu  wrote:

> On Wed, Aug 10, 2016 at 10:51:51AM +0200, Igor Mammedov wrote:
> 
> [...]
> 
> > > > Upstream guest kernel 4.7.0+ (d52bd54db) crashes when booting with irq 
> > > > remapping on:
> > > > 
> > > > ./qemu-system-x86_64 -enable-kvm -smp 
> > > > 1,sockets=9,cores=32,threads=1,maxcpus=288 -device 
> > > > qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 -bios 
> > > > x2apic_bios.bin  -m 1G -nographic -device intel-iommu,intremap=on 
> > > > -machine q35,kernel-irqchip=split -snapshot -global ioapic.version=0x20 
> > > > /dev/rhel72
> > > > 
> > > > 
> > > > [0.350669] smpboot: Max logical packages: 9
> > > > [0.351853] smpboot: APIC(0) Converting physical 0 to logical 
> > > > package 0
> > > > [0.353160] smpboot: APIC(11e) Converting physical 8 to logical 
> > > > package 1
> > > > [0.354627] DMAR: Host address width 39
> > > > [0.355621] DMAR: DRHD base: 0x00fed9 flags: 0x1
> > > > [0.356785] DMAR: dmar0: reg_base_addr fed9 ver 1:0 cap 
> > > > 12008c22260206 ecap f00f1a
> > > > [0.358721] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed9 IOMMU 0
> > > > [0.360029] DMAR-IR: Queued invalidation will be enabled to support 
> > > > x2apic and Intr-remapping.
> > > > [0.364605] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > > > [0.365805] BUG: unable to handle kernel NULL pointer dereference at 
> > > >   (null)
> > > > [0.367965] IP: [] x2apic_cluster_probe+0x35/0x70
> > > > [0.369373] PGD 0 
> > > > [0.370258] Oops: 0002 [#1] SMP
> > > > [0.371140] Modules linked in:
> > > > [0.372150] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.7.0+ #647
> > > > [0.373485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> > > > rel-1.9.0-143-gbac87e4 04/01/2014
> > > > [0.375622] task: 880039ad task.stack: 880039ad8000
> > > > [0.376875] RIP: 0010:[]  [] 
> > > > x2apic_cluster_probe+0x35/0x70
> > > > [0.379066] RSP: :880039adbe28  EFLAGS: 00010202
> > > > [0.380299] RAX:  RBX: 81f388a8 RCX: 
> > > > 880039e0
> > > > [0.381677] RDX:  RSI: 0002 RDI: 
> > > > 0246
> > > > [0.383096] RBP: 880039adbe28 R08: 00c6 R09: 
> > > > 880b9b80
> > > > [0.384579] R10: 00a0 R11: 0050 R12: 
> > > > 2000
> > > > [0.385990] R13: a118 R14: 011f R15: 
> > > > 0120
> > > > [0.387448] FS:  () GS:880039e0() 
> > > > knlGS:
> > > > [0.389454] CS:  0010 DS:  ES:  CR0: 80050033
> > > > [0.390697] CR2:  CR3: 01c06000 CR4: 
> > > > 06f0
> > > > [0.392114] Stack:
> > > > [0.392889]  880039adbe40 81da277c a110 
> > > > 880039adbe78
> > > > [0.395135]  81d9c055 81f14c60 880039ad0a58 
> > > > 81c95ac0
> > > > [0.397469]  818232c0 880039ad 880039adbf38 
> > > > 81d86293
> > > > [0.399695] Call Trace:
> > > > [0.400529]  [] 
> > > > default_setup_apic_routing+0x28/0x69
> > > > [0.401881]  [] native_smp_prepare_cpus+0x223/0x2d2
> > > > [0.403260]  [] kernel_init_freeable+0xd8/0x249
> > > > [0.404525]  [] kernel_init+0xe/0x110
> > > > [0.405703]  [] ret_from_fork+0x1f/0x40
> > > > [0.406966]  [] ? rest_init+0x80/0x80
> > > > [0.408165] Code: 00 31 c0 65 8b 15 2c f1 fa 7e 85 c9 75 01 c3 48 63 
> > > > ca 55 48 c7 c0 10 d7 00 00 48 8b 0c cd 20 8d d4 81 89 d2 48 89 e5 48 8b 
> > > > 04 08  48 0f ab 10 49 c7 c0 60 b0 05 81 48 c7 c1 a0 ae 05 81 ba 01 
> > > > [0.417107] RIP  [] x2apic_cluster_probe+0x35/0x70
> > > > [0.418516]  RSP 
> > > > [0.419461] CR2: 
> > > > [0.420386] ---[ end trace f68728a0d3053b52 ]---
> 
> I failed to reproduce this panic on my machine with parameter:
> 
> bin=x86_64-softmmu/qemu-system-x86_64
> $bin -M q35,kernel-irqchip=split -enable-kvm -m 2048 \
>  -monitor stdio -smp 4 \
>  -device intel-iommu,intremap=on \
>  -netdev user,id=net0,hostfwd=tcp::-:22 \
>  -device e1000,netdev=net0 \
>  -kernel /root/git/linux/arch/x86/boot/bzImage \
>  -append root=/dev/sda3 \
>  /root/images/rhel-7.2.qcow2
> 
> Guest kernel version is exactly 4.7.0+ (d52bd54db). In the guest, I
> see x2apic enabled. Did I miss anything special?
> 
you missed presence of x2apic CPU which this series enables,
add/change CLI as following:

-smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
-device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0

+ add x2apic_phys to kernel's command line

PS:
the last kernel I've tried is:
   v4.8-rc1-53-ga0cba21 + fix from Luiz

> [...]
> 
> > adding x2apic_phys to kernel's command line makes it crash but at another 
> > place:
> > 
> > [ 

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Peter Xu]

On Wed, Aug 10, 2016 at 10:51:51AM +0200, Igor Mammedov wrote:

[...]

> > > Upstream guest kernel 4.7.0+ (d52bd54db) crashes when booting with irq 
> > > remapping on:
> > > 
> > > ./qemu-system-x86_64 -enable-kvm -smp 
> > > 1,sockets=9,cores=32,threads=1,maxcpus=288 -device 
> > > qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 -bios 
> > > x2apic_bios.bin  -m 1G -nographic -device intel-iommu,intremap=on 
> > > -machine q35,kernel-irqchip=split -snapshot -global ioapic.version=0x20 
> > > /dev/rhel72
> > > 
> > > 
> > > [0.350669] smpboot: Max logical packages: 9
> > > [0.351853] smpboot: APIC(0) Converting physical 0 to logical package 0
> > > [0.353160] smpboot: APIC(11e) Converting physical 8 to logical 
> > > package 1
> > > [0.354627] DMAR: Host address width 39
> > > [0.355621] DMAR: DRHD base: 0x00fed9 flags: 0x1
> > > [0.356785] DMAR: dmar0: reg_base_addr fed9 ver 1:0 cap 
> > > 12008c22260206 ecap f00f1a
> > > [0.358721] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed9 IOMMU 0
> > > [0.360029] DMAR-IR: Queued invalidation will be enabled to support 
> > > x2apic and Intr-remapping.
> > > [0.364605] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > > [0.365805] BUG: unable to handle kernel NULL pointer dereference at   
> > > (null)
> > > [0.367965] IP: [] x2apic_cluster_probe+0x35/0x70
> > > [0.369373] PGD 0 
> > > [0.370258] Oops: 0002 [#1] SMP
> > > [0.371140] Modules linked in:
> > > [0.372150] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.7.0+ #647
> > > [0.373485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> > > rel-1.9.0-143-gbac87e4 04/01/2014
> > > [0.375622] task: 880039ad task.stack: 880039ad8000
> > > [0.376875] RIP: 0010:[]  [] 
> > > x2apic_cluster_probe+0x35/0x70
> > > [0.379066] RSP: :880039adbe28  EFLAGS: 00010202
> > > [0.380299] RAX:  RBX: 81f388a8 RCX: 
> > > 880039e0
> > > [0.381677] RDX:  RSI: 0002 RDI: 
> > > 0246
> > > [0.383096] RBP: 880039adbe28 R08: 00c6 R09: 
> > > 880b9b80
> > > [0.384579] R10: 00a0 R11: 0050 R12: 
> > > 2000
> > > [0.385990] R13: a118 R14: 011f R15: 
> > > 0120
> > > [0.387448] FS:  () GS:880039e0() 
> > > knlGS:
> > > [0.389454] CS:  0010 DS:  ES:  CR0: 80050033
> > > [0.390697] CR2:  CR3: 01c06000 CR4: 
> > > 06f0
> > > [0.392114] Stack:
> > > [0.392889]  880039adbe40 81da277c a110 
> > > 880039adbe78
> > > [0.395135]  81d9c055 81f14c60 880039ad0a58 
> > > 81c95ac0
> > > [0.397469]  818232c0 880039ad 880039adbf38 
> > > 81d86293
> > > [0.399695] Call Trace:
> > > [0.400529]  [] default_setup_apic_routing+0x28/0x69
> > > [0.401881]  [] native_smp_prepare_cpus+0x223/0x2d2
> > > [0.403260]  [] kernel_init_freeable+0xd8/0x249
> > > [0.404525]  [] kernel_init+0xe/0x110
> > > [0.405703]  [] ret_from_fork+0x1f/0x40
> > > [0.406966]  [] ? rest_init+0x80/0x80
> > > [0.408165] Code: 00 31 c0 65 8b 15 2c f1 fa 7e 85 c9 75 01 c3 48 63 
> > > ca 55 48 c7 c0 10 d7 00 00 48 8b 0c cd 20 8d d4 81 89 d2 48 89 e5 48 8b 
> > > 04 08  48 0f ab 10 49 c7 c0 60 b0 05 81 48 c7 c1 a0 ae 05 81 ba 01 
> > > [0.417107] RIP  [] x2apic_cluster_probe+0x35/0x70
> > > [0.418516]  RSP 
> > > [0.419461] CR2: 
> > > [0.420386] ---[ end trace f68728a0d3053b52 ]---  

I failed to reproduce this panic on my machine with parameter:

bin=x86_64-softmmu/qemu-system-x86_64
$bin -M q35,kernel-irqchip=split -enable-kvm -m 2048 \
 -monitor stdio -smp 4 \
 -device intel-iommu,intremap=on \
 -netdev user,id=net0,hostfwd=tcp::-:22 \
 -device e1000,netdev=net0 \
 -kernel /root/git/linux/arch/x86/boot/bzImage \
 -append root=/dev/sda3 \
 /root/images/rhel-7.2.qcow2

Guest kernel version is exactly 4.7.0+ (d52bd54db). In the guest, I
see x2apic enabled. Did I miss anything special?

[...]

> adding x2apic_phys to kernel's command line makes it crash but at another 
> place:
> 
> [0.364909] smpboot: Max logical packages: 9
> [0.365838] smpboot: APIC(0) Converting physical 0 to logical package 0
> [0.367183] smpboot: APIC(11e) Converting physical 8 to logical package 1
> [0.370291] x2apic: IRQ remapping doesn't support X2APIC mode
> [0.371901] x2apic disabled

Failed to understand why x2apic_phys will affect the system if x2apic
is disabled after all.

Thanks!

-- peterx

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Igor Mammedov]

On Tue, 9 Aug 2016 21:35:04 +0800
Peter Xu  wrote:

> On Tue, Aug 09, 2016 at 10:28:41AM +0200, Igor Mammedov wrote:
> > On Mon, 8 Aug 2016 16:57:14 +0800
> > Peter Xu  wrote:
> >   
> > > On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:  
> > > > HI, everyone.
> > > > 
> > > > We have done some tests after merging this patch set into the lastest 
> > > > qemu
> > > > master. In kvm aspect, we use the lastest kvm linux-next branch. Here 
> > > > are
> > > > some problems we have met.
> > > > 
> > > > 1. We can't boot up a 288 vcpus linux guest with CLI:
> > > > qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> > > > -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> > > > -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> > > > The problem exists, even after we only assign 32 vcpus to the linux 
> > > > guest.
> > > > Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" 
> > > > is a clue.
> > > > The output of qemu and kernel is in attachments. Do you have any idea
> > > > about the problem and how to solve it?
> > > 
> > > IIUC, we need to wait for Radim's QEMU patches to finally enable 288
> > > vcpus?
> > > 
> > > Btw, could you please try adding this to the QEMU cmdline when testing
> > > with 32 vcpus:
> > > 
> > >   -global ioapic.version=0x20
> > > 
> > > I see that you were running RHEL 7.2 guest with a default e1000. In
> > > that case, we may need to boost ioapic version to 0x20.
> > > 
> > > Thanks,
> > > 
> > > -- peterx  
> > 
> > Peter,
> > 
> > Upstream guest kernel 4.7.0+ (d52bd54db) crashes when booting with irq 
> > remapping on:
> > 
> > ./qemu-system-x86_64 -enable-kvm -smp 
> > 1,sockets=9,cores=32,threads=1,maxcpus=288 -device 
> > qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 -bios x2apic_bios.bin  
> > -m 1G -nographic -device intel-iommu,intremap=on -machine 
> > q35,kernel-irqchip=split -snapshot -global ioapic.version=0x20 /dev/rhel72
> > 
> > 
> > [0.350669] smpboot: Max logical packages: 9
> > [0.351853] smpboot: APIC(0) Converting physical 0 to logical package 0
> > [0.353160] smpboot: APIC(11e) Converting physical 8 to logical package 1
> > [0.354627] DMAR: Host address width 39
> > [0.355621] DMAR: DRHD base: 0x00fed9 flags: 0x1
> > [0.356785] DMAR: dmar0: reg_base_addr fed9 ver 1:0 cap 
> > 12008c22260206 ecap f00f1a
> > [0.358721] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed9 IOMMU 0
> > [0.360029] DMAR-IR: Queued invalidation will be enabled to support 
> > x2apic and Intr-remapping.
> > [0.364605] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > [0.365805] BUG: unable to handle kernel NULL pointer dereference at 
> >   (null)
> > [0.367965] IP: [] x2apic_cluster_probe+0x35/0x70
> > [0.369373] PGD 0 
> > [0.370258] Oops: 0002 [#1] SMP
> > [0.371140] Modules linked in:
> > [0.372150] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.7.0+ #647
> > [0.373485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> > rel-1.9.0-143-gbac87e4 04/01/2014
> > [0.375622] task: 880039ad task.stack: 880039ad8000
> > [0.376875] RIP: 0010:[]  [] 
> > x2apic_cluster_probe+0x35/0x70
> > [0.379066] RSP: :880039adbe28  EFLAGS: 00010202
> > [0.380299] RAX:  RBX: 81f388a8 RCX: 
> > 880039e0
> > [0.381677] RDX:  RSI: 0002 RDI: 
> > 0246
> > [0.383096] RBP: 880039adbe28 R08: 00c6 R09: 
> > 880b9b80
> > [0.384579] R10: 00a0 R11: 0050 R12: 
> > 2000
> > [0.385990] R13: a118 R14: 011f R15: 
> > 0120
> > [0.387448] FS:  () GS:880039e0() 
> > knlGS:
> > [0.389454] CS:  0010 DS:  ES:  CR0: 80050033
> > [0.390697] CR2:  CR3: 01c06000 CR4: 
> > 06f0
> > [0.392114] Stack:
> > [0.392889]  880039adbe40 81da277c a110 
> > 880039adbe78
> > [0.395135]  81d9c055 81f14c60 880039ad0a58 
> > 81c95ac0
> > [0.397469]  818232c0 880039ad 880039adbf38 
> > 81d86293
> > [0.399695] Call Trace:
> > [0.400529]  [] default_setup_apic_routing+0x28/0x69
> > [0.401881]  [] native_smp_prepare_cpus+0x223/0x2d2
> > [0.403260]  [] kernel_init_freeable+0xd8/0x249
> > [0.404525]  [] kernel_init+0xe/0x110
> > [0.405703]  [] ret_from_fork+0x1f/0x40
> > [0.406966]  [] ? rest_init+0x80/0x80
> > [0.408165] Code: 00 31 c0 65 8b 15 2c f1 fa 7e 85 c9 75 01 c3 48 63 ca 
> > 55 48 c7 c0 10 d7 00 00 48 8b 0c cd 20 8d d4 81 89 d2 48 89 e5 48 8b 04 08 
> >  48 0f ab 10 49 c7 c0 60 b0 05 81 48 c7 c1 a0 ae 05 81 ba 01 
> > [0.417107] RIP  [] x2apic_cluster_probe+0x35/0x70
> 

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Luiz Capitulino]

On Tue, 9 Aug 2016 21:35:04 +0800
Peter Xu  wrote:

> On Tue, Aug 09, 2016 at 10:28:41AM +0200, Igor Mammedov wrote:
> > On Mon, 8 Aug 2016 16:57:14 +0800
> > Peter Xu  wrote:
> >   
> > > On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:  
> > > > HI, everyone.
> > > > 
> > > > We have done some tests after merging this patch set into the lastest 
> > > > qemu
> > > > master. In kvm aspect, we use the lastest kvm linux-next branch. Here 
> > > > are
> > > > some problems we have met.
> > > > 
> > > > 1. We can't boot up a 288 vcpus linux guest with CLI:
> > > > qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> > > > -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> > > > -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> > > > The problem exists, even after we only assign 32 vcpus to the linux 
> > > > guest.
> > > > Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" 
> > > > is a clue.
> > > > The output of qemu and kernel is in attachments. Do you have any idea
> > > > about the problem and how to solve it?
> > > 
> > > IIUC, we need to wait for Radim's QEMU patches to finally enable 288
> > > vcpus?
> > > 
> > > Btw, could you please try adding this to the QEMU cmdline when testing
> > > with 32 vcpus:
> > > 
> > >   -global ioapic.version=0x20
> > > 
> > > I see that you were running RHEL 7.2 guest with a default e1000. In
> > > that case, we may need to boost ioapic version to 0x20.
> > > 
> > > Thanks,
> > > 
> > > -- peterx  
> > 
> > Peter,
> > 
> > Upstream guest kernel 4.7.0+ (d52bd54db) crashes when booting with irq 
> > remapping on:
> > 
> > ./qemu-system-x86_64 -enable-kvm -smp 
> > 1,sockets=9,cores=32,threads=1,maxcpus=288 -device 
> > qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 -bios x2apic_bios.bin  
> > -m 1G -nographic -device intel-iommu,intremap=on -machine 
> > q35,kernel-irqchip=split -snapshot -global ioapic.version=0x20 /dev/rhel72
> > 
> > 
> > [0.350669] smpboot: Max logical packages: 9
> > [0.351853] smpboot: APIC(0) Converting physical 0 to logical package 0
> > [0.353160] smpboot: APIC(11e) Converting physical 8 to logical package 1
> > [0.354627] DMAR: Host address width 39
> > [0.355621] DMAR: DRHD base: 0x00fed9 flags: 0x1
> > [0.356785] DMAR: dmar0: reg_base_addr fed9 ver 1:0 cap 
> > 12008c22260206 ecap f00f1a
> > [0.358721] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed9 IOMMU 0
> > [0.360029] DMAR-IR: Queued invalidation will be enabled to support 
> > x2apic and Intr-remapping.
> > [0.364605] DMAR-IR: Enabled IRQ remapping in x2apic mode
> > [0.365805] BUG: unable to handle kernel NULL pointer dereference at 
> >   (null)
> > [0.367965] IP: [] x2apic_cluster_probe+0x35/0x70
> > [0.369373] PGD 0 
> > [0.370258] Oops: 0002 [#1] SMP
> > [0.371140] Modules linked in:
> > [0.372150] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.7.0+ #647
> > [0.373485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> > rel-1.9.0-143-gbac87e4 04/01/2014
> > [0.375622] task: 880039ad task.stack: 880039ad8000
> > [0.376875] RIP: 0010:[]  [] 
> > x2apic_cluster_probe+0x35/0x70
> > [0.379066] RSP: :880039adbe28  EFLAGS: 00010202
> > [0.380299] RAX:  RBX: 81f388a8 RCX: 
> > 880039e0
> > [0.381677] RDX:  RSI: 0002 RDI: 
> > 0246
> > [0.383096] RBP: 880039adbe28 R08: 00c6 R09: 
> > 880b9b80
> > [0.384579] R10: 00a0 R11: 0050 R12: 
> > 2000
> > [0.385990] R13: a118 R14: 011f R15: 
> > 0120
> > [0.387448] FS:  () GS:880039e0() 
> > knlGS:
> > [0.389454] CS:  0010 DS:  ES:  CR0: 80050033
> > [0.390697] CR2:  CR3: 01c06000 CR4: 
> > 06f0
> > [0.392114] Stack:
> > [0.392889]  880039adbe40 81da277c a110 
> > 880039adbe78
> > [0.395135]  81d9c055 81f14c60 880039ad0a58 
> > 81c95ac0
> > [0.397469]  818232c0 880039ad 880039adbf38 
> > 81d86293
> > [0.399695] Call Trace:
> > [0.400529]  [] default_setup_apic_routing+0x28/0x69
> > [0.401881]  [] native_smp_prepare_cpus+0x223/0x2d2
> > [0.403260]  [] kernel_init_freeable+0xd8/0x249
> > [0.404525]  [] kernel_init+0xe/0x110
> > [0.405703]  [] ret_from_fork+0x1f/0x40
> > [0.406966]  [] ? rest_init+0x80/0x80
> > [0.408165] Code: 00 31 c0 65 8b 15 2c f1 fa 7e 85 c9 75 01 c3 48 63 ca 
> > 55 48 c7 c0 10 d7 00 00 48 8b 0c cd 20 8d d4 81 89 d2 48 89 e5 48 8b 04 08 
> >  48 0f ab 10 49 c7 c0 60 b0 05 81 48 c7 c1 a0 ae 05 81 ba 01 
> > [0.417107] RIP  [] x2apic_cluster_probe+0x35/0x70
> 

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Peter Xu]

On Tue, Aug 09, 2016 at 10:28:41AM +0200, Igor Mammedov wrote:
> On Mon, 8 Aug 2016 16:57:14 +0800
> Peter Xu  wrote:
> 
> > On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
> > > HI, everyone.
> > > 
> > > We have done some tests after merging this patch set into the lastest qemu
> > > master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> > > some problems we have met.
> > > 
> > > 1. We can't boot up a 288 vcpus linux guest with CLI:
> > > qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> > > -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> > > -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> > > The problem exists, even after we only assign 32 vcpus to the linux guest.
> > > Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is 
> > > a clue.
> > > The output of qemu and kernel is in attachments. Do you have any idea
> > > about the problem and how to solve it?  
> > 
> > IIUC, we need to wait for Radim's QEMU patches to finally enable 288
> > vcpus?
> > 
> > Btw, could you please try adding this to the QEMU cmdline when testing
> > with 32 vcpus:
> > 
> >   -global ioapic.version=0x20
> > 
> > I see that you were running RHEL 7.2 guest with a default e1000. In
> > that case, we may need to boost ioapic version to 0x20.
> > 
> > Thanks,
> > 
> > -- peterx
> 
> Peter,
> 
> Upstream guest kernel 4.7.0+ (d52bd54db) crashes when booting with irq 
> remapping on:
> 
> ./qemu-system-x86_64 -enable-kvm -smp 
> 1,sockets=9,cores=32,threads=1,maxcpus=288 -device 
> qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 -bios x2apic_bios.bin  
> -m 1G -nographic -device intel-iommu,intremap=on -machine 
> q35,kernel-irqchip=split -snapshot -global ioapic.version=0x20 /dev/rhel72
> 
> 
> [0.350669] smpboot: Max logical packages: 9
> [0.351853] smpboot: APIC(0) Converting physical 0 to logical package 0
> [0.353160] smpboot: APIC(11e) Converting physical 8 to logical package 1
> [0.354627] DMAR: Host address width 39
> [0.355621] DMAR: DRHD base: 0x00fed9 flags: 0x1
> [0.356785] DMAR: dmar0: reg_base_addr fed9 ver 1:0 cap 12008c22260206 
> ecap f00f1a
> [0.358721] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed9 IOMMU 0
> [0.360029] DMAR-IR: Queued invalidation will be enabled to support x2apic 
> and Intr-remapping.
> [0.364605] DMAR-IR: Enabled IRQ remapping in x2apic mode
> [0.365805] BUG: unable to handle kernel NULL pointer dereference at   
> (null)
> [0.367965] IP: [] x2apic_cluster_probe+0x35/0x70
> [0.369373] PGD 0 
> [0.370258] Oops: 0002 [#1] SMP
> [0.371140] Modules linked in:
> [0.372150] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.7.0+ #647
> [0.373485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
> rel-1.9.0-143-gbac87e4 04/01/2014
> [0.375622] task: 880039ad task.stack: 880039ad8000
> [0.376875] RIP: 0010:[]  [] 
> x2apic_cluster_probe+0x35/0x70
> [0.379066] RSP: :880039adbe28  EFLAGS: 00010202
> [0.380299] RAX:  RBX: 81f388a8 RCX: 
> 880039e0
> [0.381677] RDX:  RSI: 0002 RDI: 
> 0246
> [0.383096] RBP: 880039adbe28 R08: 00c6 R09: 
> 880b9b80
> [0.384579] R10: 00a0 R11: 0050 R12: 
> 2000
> [0.385990] R13: a118 R14: 011f R15: 
> 0120
> [0.387448] FS:  () GS:880039e0() 
> knlGS:
> [0.389454] CS:  0010 DS:  ES:  CR0: 80050033
> [0.390697] CR2:  CR3: 01c06000 CR4: 
> 06f0
> [0.392114] Stack:
> [0.392889]  880039adbe40 81da277c a110 
> 880039adbe78
> [0.395135]  81d9c055 81f14c60 880039ad0a58 
> 81c95ac0
> [0.397469]  818232c0 880039ad 880039adbf38 
> 81d86293
> [0.399695] Call Trace:
> [0.400529]  [] default_setup_apic_routing+0x28/0x69
> [0.401881]  [] native_smp_prepare_cpus+0x223/0x2d2
> [0.403260]  [] kernel_init_freeable+0xd8/0x249
> [0.404525]  [] kernel_init+0xe/0x110
> [0.405703]  [] ret_from_fork+0x1f/0x40
> [0.406966]  [] ? rest_init+0x80/0x80
> [0.408165] Code: 00 31 c0 65 8b 15 2c f1 fa 7e 85 c9 75 01 c3 48 63 ca 55 
> 48 c7 c0 10 d7 00 00 48 8b 0c cd 20 8d d4 81 89 d2 48 89 e5 48 8b 04 08  
> 48 0f ab 10 49 c7 c0 60 b0 05 81 48 c7 c1 a0 ae 05 81 ba 01 
> [0.417107] RIP  [] x2apic_cluster_probe+0x35/0x70
> [0.418516]  RSP 
> [0.419461] CR2: 
> [0.420386] ---[ end trace f68728a0d3053b52 ]---

Hi, Igor,

Thanks for the feedback!

Not sure whether this patch can fix it (since they looks alike):

  https://lkml.org/lkml/2016/8/4/345

CC Luiz.

Thanks,

-- peterx

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Radim Krčmář]

2016-08-09 16:19+0800, Chao Gao:
> On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote:
>>On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote:
>>> On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote:
>>> >On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
>>> >> HI, everyone.
>>> >> 
>>> >> We have done some tests after merging this patch set into the lastest 
>>> >> qemu
>>> >> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
>>> >> some problems we have met.
>>> >> 
>>> >> 1. We can't boot up a 288 vcpus linux guest with CLI:
>>> >> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
>>> >> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
>>> >> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
>>> >> The problem exists, even after we only assign 32 vcpus to the linux 
>>> >> guest.
>>> >> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is 
>>> >> a clue.
>>> >> The output of qemu and kernel is in attachments. Do you have any idea
>>> >> about the problem and how to solve it?
>>> >
>>> >IIUC, we need to wait for Radim's QEMU patches to finally enable 288
>>> >vcpus?
>>> >
>>> >Btw, could you please try adding this to the QEMU cmdline when testing
>>> >with 32 vcpus:
>>> >
>>> >  -global ioapic.version=0x20
>>> >
>>> >I see that you were running RHEL 7.2 guest with a default e1000. In
>>> >that case, we may need to boost ioapic version to 0x20.
>>> 
>>> It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus to 
>>> the guest
>>> or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe when 
>>> irqchip
>>> is in kernel, intremap can only handle situations that vcpus number is less 
>>> than 
>>> physical cpus'. Do you think it's right? 
>>
>>I don't think so. Vcpu number should not be related to host cpu
>>numbers.
>>
>>I think the problem is with x2apic. Currently, x2apic is enabled in
>>vIOMMU when kernel irqchip is used. This is problematic, since
>>actually we are throughing away dest_id[31:8] without Radim's patches,
>>meanwhile I see that by default x2apic is using cluster mode.
>>
>>In cluster mode, 8 bits will possibly not suffice (I think the reason
>>why >17 vcpus will bring trouble is that each cluster has 16 vcpus,
>>we'll have trouble if we have more than one cluster).
>>
>>To temporarily solve your issue, you should not only need "-global
>>ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys"
>>to you guest kernel boot parameter, to force guest boot with x2apic
>>physical mode (not cluster mode). Though this can only work for <255
>>vcpus. IMHO we may still need to wait for Radim's patches to test >255
>>case.
> 
> ok, after adding "x2apic_phys" to guest kernel boot parameter, I 
> boot up a 288(yes, 288) vcpus guest successfully with command
> qemu-system-x86_64 -boot c -m 4096 -sdl --enable-kvm \
> -M kernel-irqchip=split -bios bios.bin -smp cpus=288 -hda vdisk.img \
> -device intel-iommu,intremap=on -machine q35
> Also, I can see interrupts on those cpu with inital apicid>255 from 
> /proc/cpuinfo and /proc/interrupts. 

Great, thanks for testing!
Only IPIs will be correctly delivered to apic_id > 255 without few more
patches on the QEMU side, though.

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Radim Krčmář]

2016-08-09 15:09+0800, Peter Xu:
> On Tue, Aug 09, 2016 at 08:33:13AM +0200, Jan Kiszka wrote:
>> On 2016-08-09 08:24, Peter Xu wrote:
>> > On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote:
>> >> On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote:
>> >>> On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote:
>>  On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
>> > HI, everyone.
>> >
>> > We have done some tests after merging this patch set into the lastest 
>> > qemu
>> > master. In kvm aspect, we use the lastest kvm linux-next branch. Here 
>> > are
>> > some problems we have met.
>> >
>> > 1. We can't boot up a 288 vcpus linux guest with CLI:
>> > qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
>> > -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
>> > -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
>> > The problem exists, even after we only assign 32 vcpus to the linux 
>> > guest.
>> > Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" 
>> > is a clue.
>> > The output of qemu and kernel is in attachments. Do you have any idea
>> > about the problem and how to solve it?
>> 
>>  IIUC, we need to wait for Radim's QEMU patches to finally enable 288
>>  vcpus?
>> 
>>  Btw, could you please try adding this to the QEMU cmdline when testing
>>  with 32 vcpus:
>> 
>>   -global ioapic.version=0x20
>> 
>>  I see that you were running RHEL 7.2 guest with a default e1000. In
>>  that case, we may need to boost ioapic version to 0x20.
>> >>>
>> >>> It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus 
>> >>> to the guest
>> >>> or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe 
>> >>> when irqchip
>> >>> is in kernel, intremap can only handle situations that vcpus number is 
>> >>> less than 
>> >>> physical cpus'. Do you think it's right? 
>> >>
>> >> I don't think so. Vcpu number should not be related to host cpu
>> >> numbers.
>> >>
>> >> I think the problem is with x2apic. Currently, x2apic is enabled in
>> >> vIOMMU when kernel irqchip is used. This is problematic, since
>> >> actually we are throughing away dest_id[31:8] without Radim's patches,
>> >> meanwhile I see that by default x2apic is using cluster mode.
>> >>
>> >> In cluster mode, 8 bits will possibly not suffice (I think the reason
>> >> why >17 vcpus will bring trouble is that each cluster has 16 vcpus,
>> >> we'll have trouble if we have more than one cluster).
>> >>
>> >> To temporarily solve your issue, you should not only need "-global
>> >> ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys"
>> >> to you guest kernel boot parameter, to force guest boot with x2apic
>> >> physical mode (not cluster mode). Though this can only work for <255
>> >> vcpus. IMHO we may still need to wait for Radim's patches to test >255
>> >> case.
>> > 
>> > Not sure whether we should temporarily disable EIM by default for now
>> > (provide an extra flag to optionally enable it)? Since it might break
>> > guests with >17 vcpus.
>> > 
>> > CC Jan as well.
>> 
>> A switch for EIM would be fine for me if it helps.
>> 
>> To my understanding, the issue will be gone with an enhance KVM
>> interface that we can then also detect via some cap (to flip the default
>> again)?
> 
> Would you help explain how to do it?
> 
> Btw, if we have that switch, the default can go back to EIM mode along
> with Radim's future patches.

I will post patches today as the feature made it upstream.

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Igor Mammedov]

On Mon, 8 Aug 2016 16:57:14 +0800
Peter Xu  wrote:

> On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
> > HI, everyone.
> > 
> > We have done some tests after merging this patch set into the lastest qemu
> > master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> > some problems we have met.
> > 
> > 1. We can't boot up a 288 vcpus linux guest with CLI:
> > qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> > -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> > -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> > The problem exists, even after we only assign 32 vcpus to the linux guest.
> > Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a 
> > clue.
> > The output of qemu and kernel is in attachments. Do you have any idea
> > about the problem and how to solve it?  
> 
> IIUC, we need to wait for Radim's QEMU patches to finally enable 288
> vcpus?
> 
> Btw, could you please try adding this to the QEMU cmdline when testing
> with 32 vcpus:
> 
>   -global ioapic.version=0x20
> 
> I see that you were running RHEL 7.2 guest with a default e1000. In
> that case, we may need to boost ioapic version to 0x20.
> 
> Thanks,
> 
> -- peterx

Peter,

Upstream guest kernel 4.7.0+ (d52bd54db) crashes when booting with irq 
remapping on:

./qemu-system-x86_64 -enable-kvm -smp 
1,sockets=9,cores=32,threads=1,maxcpus=288 -device 
qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 -bios x2apic_bios.bin  -m 
1G -nographic -device intel-iommu,intremap=on -machine q35,kernel-irqchip=split 
-snapshot -global ioapic.version=0x20 /dev/rhel72


[0.350669] smpboot: Max logical packages: 9
[0.351853] smpboot: APIC(0) Converting physical 0 to logical package 0
[0.353160] smpboot: APIC(11e) Converting physical 8 to logical package 1
[0.354627] DMAR: Host address width 39
[0.355621] DMAR: DRHD base: 0x00fed9 flags: 0x1
[0.356785] DMAR: dmar0: reg_base_addr fed9 ver 1:0 cap 12008c22260206 
ecap f00f1a
[0.358721] DMAR-IR: IOAPIC id 0 under DRHD base  0xfed9 IOMMU 0
[0.360029] DMAR-IR: Queued invalidation will be enabled to support x2apic 
and Intr-remapping.
[0.364605] DMAR-IR: Enabled IRQ remapping in x2apic mode
[0.365805] BUG: unable to handle kernel NULL pointer dereference at 
  (null)
[0.367965] IP: [] x2apic_cluster_probe+0x35/0x70
[0.369373] PGD 0 
[0.370258] Oops: 0002 [#1] SMP
[0.371140] Modules linked in:
[0.372150] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.7.0+ #647
[0.373485] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 
rel-1.9.0-143-gbac87e4 04/01/2014
[0.375622] task: 880039ad task.stack: 880039ad8000
[0.376875] RIP: 0010:[]  [] 
x2apic_cluster_probe+0x35/0x70
[0.379066] RSP: :880039adbe28  EFLAGS: 00010202
[0.380299] RAX:  RBX: 81f388a8 RCX: 880039e0
[0.381677] RDX:  RSI: 0002 RDI: 0246
[0.383096] RBP: 880039adbe28 R08: 00c6 R09: 880b9b80
[0.384579] R10: 00a0 R11: 0050 R12: 2000
[0.385990] R13: a118 R14: 011f R15: 0120
[0.387448] FS:  () GS:880039e0() 
knlGS:
[0.389454] CS:  0010 DS:  ES:  CR0: 80050033
[0.390697] CR2:  CR3: 01c06000 CR4: 06f0
[0.392114] Stack:
[0.392889]  880039adbe40 81da277c a110 
880039adbe78
[0.395135]  81d9c055 81f14c60 880039ad0a58 
81c95ac0
[0.397469]  818232c0 880039ad 880039adbf38 
81d86293
[0.399695] Call Trace:
[0.400529]  [] default_setup_apic_routing+0x28/0x69
[0.401881]  [] native_smp_prepare_cpus+0x223/0x2d2
[0.403260]  [] kernel_init_freeable+0xd8/0x249
[0.404525]  [] kernel_init+0xe/0x110
[0.405703]  [] ret_from_fork+0x1f/0x40
[0.406966]  [] ? rest_init+0x80/0x80
[0.408165] Code: 00 31 c0 65 8b 15 2c f1 fa 7e 85 c9 75 01 c3 48 63 ca 55 
48 c7 c0 10 d7 00 00 48 8b 0c cd 20 8d d4 81 89 d2 48 89 e5 48 8b 04 08  48 
0f ab 10 49 c7 c0 60 b0 05 81 48 c7 c1 a0 ae 05 81 ba 01 
[0.417107] RIP  [] x2apic_cluster_probe+0x35/0x70
[0.418516]  RSP 
[0.419461] CR2: 
[0.420386] ---[ end trace f68728a0d3053b52 ]---

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Igor Mammedov]

On Tue, 9 Aug 2016 11:23:58 +0800
Chao Gao  wrote:

> On Mon, Aug 08, 2016 at 11:18:20AM +0200, Igor Mammedov wrote:
> >On Mon, 8 Aug 2016 15:41:23 +0800
> >Chao Gao  wrote:
> >  
> >> HI, everyone.
> >> 
> >> We have done some tests after merging this patch set into the lastest qemu
> >> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> >> some problems we have met.
> >> 
> >> 1. We can't boot up a 288 vcpus linux guest with CLI:
> >> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> >> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> >> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> >> The problem exists, even after we only assign 32 vcpus to the linux guest.
> >> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a 
> >> clue.
> >> The output of qemu and kernel is in attachments. Do you have any idea
> >> about the problem and how to solve it?  
> >I don't think we ever looked at "kernel-irqchip=split" only in kernel 
> >variant's
> >been targeted so far.
> >Radim probably knows better whether it should work or not.
> >
> >Have you tried with smaller amount of CPUs but with APIC IDs above 254,
> >like in test below?
> >
> >[...]
> >  
> >> >Tested with following CLI:
> >> > QEMU -M q35 -enable-kvm -smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
> >> >  -device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0   \
> >> >  -bios x2apic_bios.bin  
> 
> I test with CLI:
> qemu-system-x86_64 -M q35 \
> -enable-kvm -smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
> -device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 \
> -bios bios.bin -hda vdisk.img -serial stdio -m 4096 2>>qemu_and_guest.log >&2
> 
> But, I think there should have a cpu with initial apicid >255 
> in /proc/cpuinfo. The log(in attachments) shows that the guest kernel 
> treats the other cpu as a bad one. What do you think cause the problem?
Bad cpu happens because of following:
[0.319911] x2apic: IRQ remapping doesn't support X2APIC mode
[0.321427] x2apic disabled

if I recall correctly Radim gave me a hack/patch to workaround it,
and that's what I've been testing with.

Anyway from log it looks like CPU with x2apic is there and seabios
initialized it correctly. Missing part is working irq remapping.



> 
> # cat /proc/interrupts
> localhost login: CPU0   
>0:125   IO-APIC-edge  timer
>1:117   IO-APIC-edge  i8042
>4:382   IO-APIC-edge  serial
>7:  0   IO-APIC-edge  parport0
>8:  1   IO-APIC-edge  rtc0
>9:  0   IO-APIC-fasteoi   acpi
>   12:   1661   IO-APIC-edge  i8042
>   16:  0   IO-APIC-fasteoi   i801_smbus
>   22: 27   IO-APIC-fasteoi   enp0s2
>   24:   7310   PCI-MSI-edge  :00:1f.2
>  NMI:  0   Non-maskable interrupts
>  LOC:   6401   Local timer interrupts
>  SPU:  0   Spurious interrupts
>  PMI:  0   Performance monitoring interrupts
>  IWI:   3870   IRQ work interrupts
>  RTR:  0   APIC ICR read retries
>  RES:  0   Rescheduling interrupts
>  CAL:  0   Function call interrupts
>  TLB:  0   TLB shootdowns
>  TRM:  0   Thermal event interrupts
>  THR:  0   Threshold APIC interrupts
>  MCE:  0   Machine check exceptions
>  MCP:  1   Machine check polls
>  ERR:  0
>  MIS:  0
> 
> # cat /proc/cpuinfo
> processor   : 0
> vendor_id   : GenuineIntel
> cpu family  : 6
> model   : 6
> model name  : QEMU Virtual CPU version 2.5+
> stepping: 3
> microcode   : 0x1
> cpu MHz : 3591.682
> cache size  : 4096 KB
> physical id : 0
> siblings: 1
> core id : 0
> cpu cores   : 1
> apicid  : 0
> initial apicid  : 0
> fpu : yes
> fpu_exception   : yes
> cpuid level : 13
> wp  : yes
> flags   : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
> pse36 clflush mmx fxsr sse sse2 ht syscall nx lm rep_good nopl xtopology pni 
> cx16 x2apic hypervisor lahf_lm
> bogomips: 7183.36
> clflush size: 64
> cache_alignment : 64
> address sizes   : 40 bits physical, 48 bits virtual
> power management:

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Chao Gao]

On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote:
>On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote:
>> On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote:
>> >On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
>> >> HI, everyone.
>> >> 
>> >> We have done some tests after merging this patch set into the lastest qemu
>> >> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
>> >> some problems we have met.
>> >> 
>> >> 1. We can't boot up a 288 vcpus linux guest with CLI:
>> >> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
>> >> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
>> >> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
>> >> The problem exists, even after we only assign 32 vcpus to the linux guest.
>> >> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is 
>> >> a clue.
>> >> The output of qemu and kernel is in attachments. Do you have any idea
>> >> about the problem and how to solve it?
>> >
>> >IIUC, we need to wait for Radim's QEMU patches to finally enable 288
>> >vcpus?
>> >
>> >Btw, could you please try adding this to the QEMU cmdline when testing
>> >with 32 vcpus:
>> >
>> >  -global ioapic.version=0x20
>> >
>> >I see that you were running RHEL 7.2 guest with a default e1000. In
>> >that case, we may need to boost ioapic version to 0x20.
>> 
>> It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus to 
>> the guest
>> or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe when 
>> irqchip
>> is in kernel, intremap can only handle situations that vcpus number is less 
>> than 
>> physical cpus'. Do you think it's right? 
>
>I don't think so. Vcpu number should not be related to host cpu
>numbers.
>
>I think the problem is with x2apic. Currently, x2apic is enabled in
>vIOMMU when kernel irqchip is used. This is problematic, since
>actually we are throughing away dest_id[31:8] without Radim's patches,
>meanwhile I see that by default x2apic is using cluster mode.
>
>In cluster mode, 8 bits will possibly not suffice (I think the reason
>why >17 vcpus will bring trouble is that each cluster has 16 vcpus,
>we'll have trouble if we have more than one cluster).
>
>To temporarily solve your issue, you should not only need "-global
>ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys"
>to you guest kernel boot parameter, to force guest boot with x2apic
>physical mode (not cluster mode). Though this can only work for <255
>vcpus. IMHO we may still need to wait for Radim's patches to test >255
>case.

ok, after adding "x2apic_phys" to guest kernel boot parameter, I 
boot up a 288(yes, 288) vcpus guest successfully with command
qemu-system-x86_64 -boot c -m 4096 -sdl --enable-kvm \
-M kernel-irqchip=split -bios bios.bin -smp cpus=288 -hda vdisk.img \
-device intel-iommu,intremap=on -machine q35
Also, I can see interrupts on those cpu with inital apicid>255 from 
/proc/cpuinfo and /proc/interrupts. 

Thanks.
-Chao

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Peter Xu]

On Tue, Aug 09, 2016 at 08:33:13AM +0200, Jan Kiszka wrote:
> On 2016-08-09 08:24, Peter Xu wrote:
> > On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote:
> >> On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote:
> >>> On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote:
>  On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
> > HI, everyone.
> >
> > We have done some tests after merging this patch set into the lastest 
> > qemu
> > master. In kvm aspect, we use the lastest kvm linux-next branch. Here 
> > are
> > some problems we have met.
> >
> > 1. We can't boot up a 288 vcpus linux guest with CLI:
> > qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> > -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> > -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> > The problem exists, even after we only assign 32 vcpus to the linux 
> > guest.
> > Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" 
> > is a clue.
> > The output of qemu and kernel is in attachments. Do you have any idea
> > about the problem and how to solve it?
> 
>  IIUC, we need to wait for Radim's QEMU patches to finally enable 288
>  vcpus?
> 
>  Btw, could you please try adding this to the QEMU cmdline when testing
>  with 32 vcpus:
> 
>   -global ioapic.version=0x20
> 
>  I see that you were running RHEL 7.2 guest with a default e1000. In
>  that case, we may need to boost ioapic version to 0x20.
> >>>
> >>> It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus 
> >>> to the guest
> >>> or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe 
> >>> when irqchip
> >>> is in kernel, intremap can only handle situations that vcpus number is 
> >>> less than 
> >>> physical cpus'. Do you think it's right? 
> >>
> >> I don't think so. Vcpu number should not be related to host cpu
> >> numbers.
> >>
> >> I think the problem is with x2apic. Currently, x2apic is enabled in
> >> vIOMMU when kernel irqchip is used. This is problematic, since
> >> actually we are throughing away dest_id[31:8] without Radim's patches,
> >> meanwhile I see that by default x2apic is using cluster mode.
> >>
> >> In cluster mode, 8 bits will possibly not suffice (I think the reason
> >> why >17 vcpus will bring trouble is that each cluster has 16 vcpus,
> >> we'll have trouble if we have more than one cluster).
> >>
> >> To temporarily solve your issue, you should not only need "-global
> >> ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys"
> >> to you guest kernel boot parameter, to force guest boot with x2apic
> >> physical mode (not cluster mode). Though this can only work for <255
> >> vcpus. IMHO we may still need to wait for Radim's patches to test >255
> >> case.
> > 
> > Not sure whether we should temporarily disable EIM by default for now
> > (provide an extra flag to optionally enable it)? Since it might break
> > guests with >17 vcpus.
> > 
> > CC Jan as well.
> 
> A switch for EIM would be fine for me if it helps.
> 
> To my understanding, the issue will be gone with an enhance KVM
> interface that we can then also detect via some cap (to flip the default
> again)?

Would you help explain how to do it?

Btw, if we have that switch, the default can go back to EIM mode along
with Radim's future patches.

Thanks,

-- peterx

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Jan Kiszka]

On 2016-08-09 08:24, Peter Xu wrote:
> On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote:
>> On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote:
>>> On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote:
 On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
> HI, everyone.
>
> We have done some tests after merging this patch set into the lastest qemu
> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> some problems we have met.
>
> 1. We can't boot up a 288 vcpus linux guest with CLI:
> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> The problem exists, even after we only assign 32 vcpus to the linux guest.
> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is 
> a clue.
> The output of qemu and kernel is in attachments. Do you have any idea
> about the problem and how to solve it?

 IIUC, we need to wait for Radim's QEMU patches to finally enable 288
 vcpus?

 Btw, could you please try adding this to the QEMU cmdline when testing
 with 32 vcpus:

  -global ioapic.version=0x20

 I see that you were running RHEL 7.2 guest with a default e1000. In
 that case, we may need to boost ioapic version to 0x20.
>>>
>>> It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus to 
>>> the guest
>>> or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe when 
>>> irqchip
>>> is in kernel, intremap can only handle situations that vcpus number is less 
>>> than 
>>> physical cpus'. Do you think it's right? 
>>
>> I don't think so. Vcpu number should not be related to host cpu
>> numbers.
>>
>> I think the problem is with x2apic. Currently, x2apic is enabled in
>> vIOMMU when kernel irqchip is used. This is problematic, since
>> actually we are throughing away dest_id[31:8] without Radim's patches,
>> meanwhile I see that by default x2apic is using cluster mode.
>>
>> In cluster mode, 8 bits will possibly not suffice (I think the reason
>> why >17 vcpus will bring trouble is that each cluster has 16 vcpus,
>> we'll have trouble if we have more than one cluster).
>>
>> To temporarily solve your issue, you should not only need "-global
>> ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys"
>> to you guest kernel boot parameter, to force guest boot with x2apic
>> physical mode (not cluster mode). Though this can only work for <255
>> vcpus. IMHO we may still need to wait for Radim's patches to test >255
>> case.
> 
> Not sure whether we should temporarily disable EIM by default for now
> (provide an extra flag to optionally enable it)? Since it might break
> guests with >17 vcpus.
> 
> CC Jan as well.

A switch for EIM would be fine for me if it helps.

To my understanding, the issue will be gone with an enhance KVM
interface that we can then also detect via some cap (to flip the default
again)?

Jan



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Peter Xu]

On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote:
> On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote:
> > On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote:
> > >On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
> > >> HI, everyone.
> > >> 
> > >> We have done some tests after merging this patch set into the lastest 
> > >> qemu
> > >> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> > >> some problems we have met.
> > >> 
> > >> 1. We can't boot up a 288 vcpus linux guest with CLI:
> > >> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> > >> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> > >> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> > >> The problem exists, even after we only assign 32 vcpus to the linux 
> > >> guest.
> > >> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is 
> > >> a clue.
> > >> The output of qemu and kernel is in attachments. Do you have any idea
> > >> about the problem and how to solve it?
> > >
> > >IIUC, we need to wait for Radim's QEMU patches to finally enable 288
> > >vcpus?
> > >
> > >Btw, could you please try adding this to the QEMU cmdline when testing
> > >with 32 vcpus:
> > >
> > >  -global ioapic.version=0x20
> > >
> > >I see that you were running RHEL 7.2 guest with a default e1000. In
> > >that case, we may need to boost ioapic version to 0x20.
> > 
> > It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus to 
> > the guest
> > or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe when 
> > irqchip
> > is in kernel, intremap can only handle situations that vcpus number is less 
> > than 
> > physical cpus'. Do you think it's right? 
> 
> I don't think so. Vcpu number should not be related to host cpu
> numbers.
> 
> I think the problem is with x2apic. Currently, x2apic is enabled in
> vIOMMU when kernel irqchip is used. This is problematic, since
> actually we are throughing away dest_id[31:8] without Radim's patches,
> meanwhile I see that by default x2apic is using cluster mode.
> 
> In cluster mode, 8 bits will possibly not suffice (I think the reason
> why >17 vcpus will bring trouble is that each cluster has 16 vcpus,
> we'll have trouble if we have more than one cluster).
> 
> To temporarily solve your issue, you should not only need "-global
> ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys"
> to you guest kernel boot parameter, to force guest boot with x2apic
> physical mode (not cluster mode). Though this can only work for <255
> vcpus. IMHO we may still need to wait for Radim's patches to test >255
> case.

Not sure whether we should temporarily disable EIM by default for now
(provide an extra flag to optionally enable it)? Since it might break
guests with >17 vcpus.

CC Jan as well.

-- peterx

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Peter Xu]

On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote:
> On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote:
> >On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
> >> HI, everyone.
> >> 
> >> We have done some tests after merging this patch set into the lastest qemu
> >> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> >> some problems we have met.
> >> 
> >> 1. We can't boot up a 288 vcpus linux guest with CLI:
> >> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> >> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> >> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> >> The problem exists, even after we only assign 32 vcpus to the linux guest.
> >> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a 
> >> clue.
> >> The output of qemu and kernel is in attachments. Do you have any idea
> >> about the problem and how to solve it?
> >
> >IIUC, we need to wait for Radim's QEMU patches to finally enable 288
> >vcpus?
> >
> >Btw, could you please try adding this to the QEMU cmdline when testing
> >with 32 vcpus:
> >
> >  -global ioapic.version=0x20
> >
> >I see that you were running RHEL 7.2 guest with a default e1000. In
> >that case, we may need to boost ioapic version to 0x20.
> 
> It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus to 
> the guest
> or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe when 
> irqchip
> is in kernel, intremap can only handle situations that vcpus number is less 
> than 
> physical cpus'. Do you think it's right? 

I don't think so. Vcpu number should not be related to host cpu
numbers.

I think the problem is with x2apic. Currently, x2apic is enabled in
vIOMMU when kernel irqchip is used. This is problematic, since
actually we are throughing away dest_id[31:8] without Radim's patches,
meanwhile I see that by default x2apic is using cluster mode.

In cluster mode, 8 bits will possibly not suffice (I think the reason
why >17 vcpus will bring trouble is that each cluster has 16 vcpus,
we'll have trouble if we have more than one cluster).

To temporarily solve your issue, you should not only need "-global
ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys"
to you guest kernel boot parameter, to force guest boot with x2apic
physical mode (not cluster mode). Though this can only work for <255
vcpus. IMHO we may still need to wait for Radim's patches to test >255
case.

Thanks.

-- peterx

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Chao Gao]

On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote:
>On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
>> HI, everyone.
>> 
>> We have done some tests after merging this patch set into the lastest qemu
>> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
>> some problems we have met.
>> 
>> 1. We can't boot up a 288 vcpus linux guest with CLI:
>> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
>> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
>> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
>> The problem exists, even after we only assign 32 vcpus to the linux guest.
>> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a 
>> clue.
>> The output of qemu and kernel is in attachments. Do you have any idea
>> about the problem and how to solve it?
>
>IIUC, we need to wait for Radim's QEMU patches to finally enable 288
>vcpus?
>
>Btw, could you please try adding this to the QEMU cmdline when testing
>with 32 vcpus:
>
>  -global ioapic.version=0x20
>
>I see that you were running RHEL 7.2 guest with a default e1000. In
>that case, we may need to boost ioapic version to 0x20.

It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus to the 
guest
or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe when 
irqchip
is in kernel, intremap can only handle situations that vcpus number is less 
than 
physical cpus'. Do you think it's right? 

Thanks,
-Chao

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Chao Gao]

On Mon, Aug 08, 2016 at 11:18:20AM +0200, Igor Mammedov wrote:
>On Mon, 8 Aug 2016 15:41:23 +0800
>Chao Gao  wrote:
>
>> HI, everyone.
>> 
>> We have done some tests after merging this patch set into the lastest qemu
>> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
>> some problems we have met.
>> 
>> 1. We can't boot up a 288 vcpus linux guest with CLI:
>> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
>> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
>> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
>> The problem exists, even after we only assign 32 vcpus to the linux guest.
>> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a 
>> clue.
>> The output of qemu and kernel is in attachments. Do you have any idea
>> about the problem and how to solve it?
>I don't think we ever looked at "kernel-irqchip=split" only in kernel variant's
>been targeted so far.
>Radim probably knows better whether it should work or not.
>
>Have you tried with smaller amount of CPUs but with APIC IDs above 254,
>like in test below?
>
>[...]
>
>> >Tested with following CLI:
>> > QEMU -M q35 -enable-kvm -smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
>> >  -device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0   \
>> >  -bios x2apic_bios.bin

I test with CLI:
qemu-system-x86_64 -M q35 \
-enable-kvm -smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
-device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0 \
-bios bios.bin -hda vdisk.img -serial stdio -m 4096 2>>qemu_and_guest.log >&2

But, I think there should have a cpu with initial apicid >255 
in /proc/cpuinfo. The log(in attachments) shows that the guest kernel 
treats the other cpu as a bad one. What do you think cause the problem?

# cat /proc/interrupts
localhost login: CPU0   
   0:125   IO-APIC-edge  timer
   1:117   IO-APIC-edge  i8042
   4:382   IO-APIC-edge  serial
   7:  0   IO-APIC-edge  parport0
   8:  1   IO-APIC-edge  rtc0
   9:  0   IO-APIC-fasteoi   acpi
  12:   1661   IO-APIC-edge  i8042
  16:  0   IO-APIC-fasteoi   i801_smbus
  22: 27   IO-APIC-fasteoi   enp0s2
  24:   7310   PCI-MSI-edge  :00:1f.2
 NMI:  0   Non-maskable interrupts
 LOC:   6401   Local timer interrupts
 SPU:  0   Spurious interrupts
 PMI:  0   Performance monitoring interrupts
 IWI:   3870   IRQ work interrupts
 RTR:  0   APIC ICR read retries
 RES:  0   Rescheduling interrupts
 CAL:  0   Function call interrupts
 TLB:  0   TLB shootdowns
 TRM:  0   Thermal event interrupts
 THR:  0   Threshold APIC interrupts
 MCE:  0   Machine check exceptions
 MCP:  1   Machine check polls
 ERR:  0
 MIS:  0

# cat /proc/cpuinfo
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model   : 6
model name  : QEMU Virtual CPU version 2.5+
stepping: 3
microcode   : 0x1
cpu MHz : 3591.682
cache size  : 4096 KB
physical id : 0
siblings: 1
core id : 0
cpu cores   : 1
apicid  : 0
initial apicid  : 0
fpu : yes
fpu_exception   : yes
cpuid level : 13
wp  : yes
flags   : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pse36 clflush mmx fxsr sse sse2 ht syscall nx lm rep_good nopl xtopology pni 
cx16 x2apic hypervisor lahf_lm
bogomips: 7183.36
clflush size: 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:
Warning: Number of hotpluggable cpus requested (288) exceeds the recommended 
cpus supported by KVM (240)
Changing serial settings was 0/0 now 3/0
SeaBIOS (version ?-20160808_104017-g.c)
BUILD: gcc: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-4) binutils: version 
2.23.52.0.1-55.el7 20130226
No Xen hypervisor found.
vendor 8086 device 29c0
Running on QEMU (q35)
Running on KVM
RamSize: 0x8000 [cmos]
Relocating init from 0x000da4e0 to 0x7ffac9d0 (size 79264)
Found QEMU fw_cfg
QEMU fw_cfg DMA interface supported
RamBlock: addr 0x len 0x8000 [e820]
RamBlock: addr 0x0001 len 0x8000 [e820]
Moving pm_base to 0x600
=== PCI bus & bridge init ===
PCI: pci_bios_init_bus_rec bus = 0x0
=== PCI device probing ===
Found 6 PCI devices (max PCI bus is 00)
=== PCI new allocation pass #1 ===
PCI: check devices
=== PCI new allocation pass #2 ===
PCI: IO: c000 - c09f
PCI: 32: c000 - fec0
PCI: map device bdf=00:02.0  bar 1, addr c000, size 0040 [io]
PCI: map device bdf=00:1f.3  bar 4, addr c040, size 0040 [io]
PCI: map device bdf=00:1f.2  bar 4, addr c080, size 0020 [io]
PCI: map device bdf=00:02.0  bar 6, addr feb8, size 0004 [mem]
PCI: map device bdf=00:02.0  bar 0, addr 

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Igor Mammedov]

On Mon, 8 Aug 2016 15:41:23 +0800
Chao Gao  wrote:

> HI, everyone.
> 
> We have done some tests after merging this patch set into the lastest qemu
> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> some problems we have met.
> 
> 1. We can't boot up a 288 vcpus linux guest with CLI:
> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> The problem exists, even after we only assign 32 vcpus to the linux guest.
> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a 
> clue.
> The output of qemu and kernel is in attachments. Do you have any idea
> about the problem and how to solve it?
I don't think we ever looked at "kernel-irqchip=split" only in kernel variant's
been targeted so far.
Radim probably knows better whether it should work or not.

Have you tried with smaller amount of CPUs but with APIC IDs above 254,
like in test below?

[...]

> >Tested with following CLI:
> > QEMU -M q35 -enable-kvm -smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
> >  -device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0   \
> >  -bios x2apic_bios.bin

Re: [Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Peter Xu]

On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote:
> HI, everyone.
> 
> We have done some tests after merging this patch set into the lastest qemu
> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are
> some problems we have met.
> 
> 1. We can't boot up a 288 vcpus linux guest with CLI:
> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \
> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \
> -hda vdisk.img -device intel-iommu,intremap=on -machine q35.
> The problem exists, even after we only assign 32 vcpus to the linux guest.
> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is a 
> clue.
> The output of qemu and kernel is in attachments. Do you have any idea
> about the problem and how to solve it?

IIUC, we need to wait for Radim's QEMU patches to finally enable 288
vcpus?

Btw, could you please try adding this to the QEMU cmdline when testing
with 32 vcpus:

  -global ioapic.version=0x20

I see that you were running RHEL 7.2 guest with a default e1000. In
that case, we may need to boost ioapic version to 0x20.

Thanks,

-- peterx

[Qemu-devel] [PATCH for-2.8 00/18] pc: q35: x2APIC support in kvm_apic mode

[Igor Mammedov]

Changes since RFC:
  - use new KVM_CAP_X2APIC_API to detect x2APIC IDs support
  - rebase on top of 2.7-rc1, since many deps were merged
  - fix etc/boot-cpus to account for -device provided cpus
  - include not yet merged _PXM fix as prereq
  - add 2.8 machine type and bump up maxcpus count since it

Series extends current CPU/kvm_apic/generic pc machine
code to support x2APIC and upto 288 VCPUs when QEMU
is used with KVM's lapic.

Due to FW_CFG_MAX_CPUS (which is actually apic_id_limit)
being limited to uint16_t, the max possible APIC ID is
limitted to 2^16 with this series but that should
be sufficient for bumping VCPUs number for quite a while.

Tested with following CLI:
 QEMU -M q35 -enable-kvm -smp 1,sockets=9,cores=32,threads=1,maxcpus=288 \
  -device qemu64-x86_64-cpu,socket-id=8,core-id=30,thread-id=0   \
  -bios x2apic_bios.bin


git gree for testing:
https://github.com/imammedo/qemu.git x2apic_v1

To play with the feature, one would also need x2apic enabled
seabios counterpart:
https://github.com/imammedo/seabios.git x2apic_v3

PS:
As kernel deps it needs 4.8 kernel on host side and it
doesn't include irq remapping/iommu fixes that Radim
has WIP branch, that should be posted separately/on top of this

But even without above kernel boots in x2APIC mode

Igor Mammedov (17):
  numa: reduce code duplication by adding helper numa_get_node_for_cpu()
  acpi: provide _PXM method for CPU devices if QEMU is started numa
enabled
  tests: acpi: extend cphp testcase with numa check
  pc: acpi: x2APIC support for MADT table
  pc: acpi: x2APIC support for SRAT table
  acpi: cphp: support x2APIC entry in cpu._MAT
  acpi: cphp: force switch to modern cpu hotplug if APIC ID > 254
  pc: leave max apic_id_limit only in legacy cpu hotplug code
  pc: apic_common: extend APIC ID property to 32bit
  pc: apic_common: restore APIC ID to initial ID on reset
  pc: apic_common: reset APIC ID to initial ID when switching into
x2APIC mode
  pc: kvm_apic: pass APIC ID depending on xAPIC/x2APIC mode
  pc: clarify FW_CFG_MAX_CPUS usage comment
  increase MAX_CPUMASK_BITS from 255 to 288
  pc: add 'etc/boot-cpus' fw_cfg file for machine with more than 255
CPUs
  pc: add 2.8 machine
  pc: q35: bump max_cpus to 288

root (1):
  linux-headers: update to v4.8-rc1

 include/hw/acpi/acpi-defs.h|  29 +
 include/hw/compat.h|   2 +
 include/hw/i386/apic_internal.h|   3 +-
 include/hw/i386/pc.h   |   4 ++
 include/sysemu/numa.h  |   3 +
 include/sysemu/sysemu.h|   2 +-
 linux-headers/linux/kvm.h  |  13 -
 target-i386/cpu.h  |   1 +
 target-i386/kvm_i386.h |   1 +
 hw/acpi/cpu.c  |  17 ++
 hw/acpi/cpu_hotplug.c  |  17 --
 hw/arm/virt-acpi-build.c   |   6 +-
 hw/arm/virt.c  |   9 ++-
 hw/i386/acpi-build.c   | 117 +
 hw/i386/kvm/apic.c |  13 -
 hw/i386/pc.c   |  47 ---
 hw/i386/pc_piix.c  |  17 --
 hw/i386/pc_q35.c   |  14 -
 hw/intc/apic_common.c  |  46 ++-
 hw/ppc/spapr.c |   2 +-
 hw/ppc/spapr_cpu_core.c|   6 +-
 numa.c |  12 
 target-i386/cpu.c  |   2 +-
 target-i386/kvm.c  |  14 +
 tests/acpi-test-data/pc/SRAT.cphp  | Bin 0 -> 304 bytes
 tests/acpi-test-data/q35/SRAT.cphp | Bin 0 -> 304 bytes
 tests/bios-tables-test.c   |   6 +-
 27 files changed, 307 insertions(+), 96 deletions(-)
 create mode 100644 tests/acpi-test-data/pc/SRAT.cphp
 create mode 100644 tests/acpi-test-data/q35/SRAT.cphp

-- 
2.7.4