Re: [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

Hi,

Thank you for your reply. I've checked that this commit fixed my issue.

вт, 11 февр. 2020 г. в 17:50, Thomas Huth <1661...@bugs.launchpad.net>:
>
> There was a fix for this assertion message wrt PMU registers in December 2017 
> already:
> https://git.qemu.org/?p=qemu.git;a=commitdiff;h=0b368a10c71af96f6cf
> Thus, can you still reproduce this issue with the latest version of QEMU, or 
> is the problem gone now?
>
> ** Changed in: qemu
>Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1661386
>
> Title:
>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>
> Status in QEMU:
>   Incomplete
>
> Bug description:
>   Hello,
>
>
>   I see the following when try to run qemu from master as the following:
>
>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>   loglevel=7' -m 1024 -serial stdio
>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
>   First broken commit has been bisected:
>
>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>   Author: Paolo Bonzini 
>   Date:   Wed Mar 30 22:55:29 2016 +0200
>
>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>
>   This would have caught the bug in the previous patch.
>
>   Signed-off-by: Paolo Bonzini 
>
>   My cpuinfo is the following:
>
>   processor   : 0
>   vendor_id   : GenuineIntel
>   cpu family  : 6
>   model   : 44
>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>   stepping: 2
>   microcode   : 0x14
>   cpu MHz : 3066.775
>   cache size  : 12288 KB
>   physical id : 0
>   siblings: 2
>   core id : 0
>   cpu cores   : 2
>   apicid  : 0
>   initial apicid  : 0
>   fpu : yes
>   fpu_exception   : yes
>   cpuid level : 11
>   wp  : yes
>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>   bugs:
>   bogomips: 6133.55
>   clflush size    : 64
>   cache_alignment : 64
>   address sizes   : 40 bits physical, 48 bits virtual
>   power management:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions


-- 
With best regards,
Matwey V. Kornilov

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  Incomplete

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini 
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
  Signed-off-by: Paolo Bonzini 

  My cpuinfo is the following:

  processor   : 0
  vendor_id   : GenuineIntel
  cpu family  : 6
  model   : 44
  model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
  stepping: 2
  microcode   : 0x14
  cpu MHz : 3066.775
  cache size  : 12288 KB
  physical id : 0
  siblings: 2
  core id : 0
  cpu cores   : 2
  apicid 

Re: [Qemu-devel] [Bug 1810433] Re: aarch64-linux-user master: inconsistent pwrite behaviour

[Matwey V. Kornilov]

Thank you!

пт, 18 янв. 2019 г. в 19:36, Peter Maydell :
>
> Commit now in QEMU master as 2bd3f8998e1e7dcd9afc29fab25. This will be
> in the next release: QEMU 4.0.
>
>
> ** Changed in: qemu
>Status: In Progress => Fix Committed
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1810433
>
> Title:
>   aarch64-linux-user master: inconsistent pwrite behaviour
>
> Status in QEMU:
>   Fix Committed
>
> Bug description:
>   Hello,
>
>   I am running aarch64-linux-user from master, commit
>   20d6c7312f1b812bb9c750f4087f69ac8485cc90
>
>   And I've found the following inconsistent emulation of pwrite() call when 
> buf==NULL and len=0.
>   Minimal reproducible sample is the following:
>
>   #define _GNU_SOURCE
>   #include 
>   #include 
>   #include 
>   #include 
>   #include 
>   #include 
>   #include 
>
>   /*
>System  | Result
>   -+
>Native x86_64 4.12.14   | pwrite ret = 0
>Native aarch64 4.4.159  | pwrite ret = 0
>qemu-aarch64 at x86_64  | pwrite ret = -1
>  ( 20d6c7312f1b8 ) |
>   */
>
>   int main(int argc, char** argv) {
>int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
>if (fd < 0) {
> perror("open");
> return 1;
>}
>
>int ret = fallocate(fd, 0, 0, 1000);
>if (ret < 0) {
> perror("fallocate");
> return 1;
>}
>
>ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
>printf("pwrite ret = %ld\n", ret_pwrite);
>
>close(fd);
>
>return 0;
>   }
>
>
>   Please note, that the same binary executable prints different output at 
> native aarch64 platform and under aarch64-linux-user
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1810433/+subscriptions


-- 
With best regards,
Matwey V. Kornilov

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1810433

Title:
  aarch64-linux-user master: inconsistent pwrite behaviour

Status in QEMU:
  Fix Committed

Bug description:
  Hello,

  I am running aarch64-linux-user from master, commit
  20d6c7312f1b812bb9c750f4087f69ac8485cc90

  And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
  Minimal reproducible sample is the following:

  #define _GNU_SOURCE
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 

  /*
   System  | Result
  -+
   Native x86_64 4.12.14   | pwrite ret = 0
   Native aarch64 4.4.159  | pwrite ret = 0
   qemu-aarch64 at x86_64  | pwrite ret = -1
     ( 20d6c7312f1b8 ) |
  */

  int main(int argc, char** argv) {
   int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
   if (fd < 0) {
    perror("open");
    return 1;
   }

   int ret = fallocate(fd, 0, 0, 1000);
   if (ret < 0) {
    perror("fallocate");
    return 1;
   }

   ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
   printf("pwrite ret = %ld\n", ret_pwrite);

   close(fd);

   return 0;
  }

  
  Please note, that the same binary executable prints different output at 
native aarch64 platform and under aarch64-linux-user

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1810433/+subscriptions

Re: [Qemu-devel] qemu-user-linux: how could I measure performance for aarch64 and arm?

[Matwey V. Kornilov]

пн, 14 янв. 2019 г. в 13:24, Peter Maydell :
>
> On Sun, 13 Jan 2019 at 07:31, Matwey V. Kornilov
>  wrote:
> >
> > пт, 11 янв. 2019 г. в 22:24, Matwey V. Kornilov :
> > > Indeed, qemu-arm from master runs for 4 minutes where 2.11 runs for 2
> > > hours for me. It is impressive improvement.
> >
> > I've managed to bisected the first good (fast) commit:
> >
> > commit 2a53535af471f4bee9d6cb5b363746b8d5ed21dd
> > Author: Luke Shumaker 
> > Date:   Thu Dec 28 13:08:13 2017 -0500
> >
> > linux-user: init_guest_space: Try to make ARM space+commpage continuous
> >
> > Though I am not sure, how does it help.
>
> Oh, right, you were running into that bug:
> https://bugs.launchpad.net/qemu/+bug/1740219
>
> The problem was that we were not putting things in the right
> place in memory for 32-bit Arm guest binaries in particular,
> which could mean that we spent a long time trying a lot of
> placements for memory mappings that failed instead of getting
> an arrangement that worked first time. This meant startup
> time for the guest binary was pretty slow. I guess your
> application does a lot of exec()ing of new processes?

Indeed. Thank you.

>
> thanks
> -- PMM



-- 
With best regards,
Matwey V. Kornilov

Re: [Qemu-devel] qemu-user-linux: how could I measure performance for aarch64 and arm?

[Matwey V. Kornilov]

пт, 11 янв. 2019 г. в 22:24, Matwey V. Kornilov :
>
> пт, 11 янв. 2019 г. в 12:52, Peter Maydell :
> >
> > On Thu, 10 Jan 2019 at 19:33, Matwey V. Kornilov
> >  wrote:
> > > I am running the same application compiled for aarch64 and armv7l on
> > > x86_64 platform using qemu-user-linux tools.
> > >
> > > I see dramatic performance difference (30 times) between emulated
> > > architectures: aarch64 runs for ~4 minutes, armv7l runs for ~2 hours.
> > > I do understand that CPU architecture emulation is inherently slow
> > > thing, but my question is about the difference.
> > >
> > > How could I debug to understand what is the reason for such a big
> > > difference? I've already tried to run stress-ng compiled for this two
> > > architectures, but it leads to the same performance per second.
> > >
> > > I am running qemu 2.11, should I try other version?
> >
> > Yes, do try 3.1 -- we have done some overall TCG performance
> > improvements.
>
> Indeed, qemu-arm from master runs for 4 minutes where 2.11 runs for 2
> hours for me. It is impressive improvement.

I've managed to bisected the first good (fast) commit:

commit 2a53535af471f4bee9d6cb5b363746b8d5ed21dd
Author: Luke Shumaker 
Date:   Thu Dec 28 13:08:13 2017 -0500

linux-user: init_guest_space: Try to make ARM space+commpage continuous

Though I am not sure, how does it help.

>
> >
> > For a big difference between target architectures like that,
> > I would try starting by using some host performance tools on
> > the two runs to see where all the time is being taken in
> > the armv7l guest run -- is it all in translated guest code,
> > or is there more time (proportionally) spent in particular
> > parts of the QEMU C code? Does the armv7l version do
> > many more or different syscalls (check with the QEMU -strace
> > option) ?
> >
> > Also you should check performance on h/w 32 bit vs
> > 64-bit Arm if you can, to confirm that it's not just
> > that the guest application runs much slower there.
> > (If you don't have the arm hardware you could at least
> > check x86 32-bit vs 64-bit.)
> >
> > thanks
> > -- PMM
>
>
>
> --
> With best regards,
> Matwey V. Kornilov



-- 
With best regards,
Matwey V. Kornilov

Re: [Qemu-devel] qemu-user-linux: how could I measure performance for aarch64 and arm?

[Matwey V. Kornilov]

пт, 11 янв. 2019 г. в 12:52, Peter Maydell :
>
> On Thu, 10 Jan 2019 at 19:33, Matwey V. Kornilov
>  wrote:
> > I am running the same application compiled for aarch64 and armv7l on
> > x86_64 platform using qemu-user-linux tools.
> >
> > I see dramatic performance difference (30 times) between emulated
> > architectures: aarch64 runs for ~4 minutes, armv7l runs for ~2 hours.
> > I do understand that CPU architecture emulation is inherently slow
> > thing, but my question is about the difference.
> >
> > How could I debug to understand what is the reason for such a big
> > difference? I've already tried to run stress-ng compiled for this two
> > architectures, but it leads to the same performance per second.
> >
> > I am running qemu 2.11, should I try other version?
>
> Yes, do try 3.1 -- we have done some overall TCG performance
> improvements.

Indeed, qemu-arm from master runs for 4 minutes where 2.11 runs for 2
hours for me. It is impressive improvement.

>
> For a big difference between target architectures like that,
> I would try starting by using some host performance tools on
> the two runs to see where all the time is being taken in
> the armv7l guest run -- is it all in translated guest code,
> or is there more time (proportionally) spent in particular
> parts of the QEMU C code? Does the armv7l version do
> many more or different syscalls (check with the QEMU -strace
> option) ?
>
> Also you should check performance on h/w 32 bit vs
> 64-bit Arm if you can, to confirm that it's not just
> that the guest application runs much slower there.
> (If you don't have the arm hardware you could at least
> check x86 32-bit vs 64-bit.)
>
> thanks
> -- PMM



-- 
With best regards,
Matwey V. Kornilov

[Qemu-devel] qemu-user-linux: how could I measure performance for aarch64 and arm?

[Matwey V. Kornilov]

Hello,

I am running the same application compiled for aarch64 and armv7l on
x86_64 platform using qemu-user-linux tools.

I see dramatic performance difference (30 times) between emulated
architectures: aarch64 runs for ~4 minutes, armv7l runs for ~2 hours.
I do understand that CPU architecture emulation is inherently slow
thing, but my question is about the difference.

How could I debug to understand what is the reason for such a big
difference? I've already tried to run stress-ng compiled for this two
architectures, but it leads to the same performance per second.

I am running qemu 2.11, should I try other version?

[Qemu-devel] [Bug 1810433] Re: aarch64-linux-user master: inconsistent pwrite behaviour

[Matwey V. Kornilov]

I've also check qemu-arm with the same test code. Surprisingly, I see
correct result:

pwrite ret = 0

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1810433

Title:
  aarch64-linux-user master: inconsistent pwrite behaviour

Status in QEMU:
  New

Bug description:
  Hello,

  I am running aarch64-linux-user from master, commit
  20d6c7312f1b812bb9c750f4087f69ac8485cc90

  And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
  Minimal reproducible sample is the following:

  #define _GNU_SOURCE
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 

  /*
   System  | Result
  -+
   Native x86_64 4.12.14   | pwrite ret = 0
   Native aarch64 4.4.159  | pwrite ret = 0
   qemu-aarch64 at x86_64  | pwrite ret = -1
     ( 20d6c7312f1b8 ) |
  */

  int main(int argc, char** argv) {
   int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
   if (fd < 0) {
    perror("open");
    return 1;
   }

   int ret = fallocate(fd, 0, 0, 1000);
   if (ret < 0) {
    perror("fallocate");
    return 1;
   }

   ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
   printf("pwrite ret = %ld\n", ret_pwrite);

   close(fd);

   return 0;
  }

  
  Please note, that the same binary executable prints different output at 
native aarch64 platform and under aarch64-linux-user

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1810433/+subscriptions

[Qemu-devel] [Bug 1810433] Re: aarch64-linux-user master: inconsistent pwrite behaviour

[Matwey V. Kornilov]

It would be great if you could fix it.

Also, there are probably should exist POSIX conformance test suites
around the world. As far as I understand, this particular issue could be
found by running such a test under qemu-linux-user. I mean what if there
are other similar issues?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1810433

Title:
  aarch64-linux-user master: inconsistent pwrite behaviour

Status in QEMU:
  New

Bug description:
  Hello,

  I am running aarch64-linux-user from master, commit
  20d6c7312f1b812bb9c750f4087f69ac8485cc90

  And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
  Minimal reproducible sample is the following:

  #define _GNU_SOURCE
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 

  /*
   System  | Result
  -+
   Native x86_64 4.12.14   | pwrite ret = 0
   Native aarch64 4.4.159  | pwrite ret = 0
   qemu-aarch64 at x86_64  | pwrite ret = -1
     ( 20d6c7312f1b8 ) |
  */

  int main(int argc, char** argv) {
   int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
   if (fd < 0) {
    perror("open");
    return 1;
   }

   int ret = fallocate(fd, 0, 0, 1000);
   if (ret < 0) {
    perror("fallocate");
    return 1;
   }

   ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
   printf("pwrite ret = %ld\n", ret_pwrite);

   close(fd);

   return 0;
  }

  
  Please note, that the same binary executable prints different output at 
native aarch64 platform and under aarch64-linux-user

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1810433/+subscriptions

[Qemu-devel] [Bug 1810433] Re: aarch64-linux-user master: inconsistent pwrite behaviour

[Matwey V. Kornilov]

Do you know how to fix it?

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1810433

Title:
  aarch64-linux-user master: inconsistent pwrite behaviour

Status in QEMU:
  New

Bug description:
  Hello,

  I am running aarch64-linux-user from master, commit
  20d6c7312f1b812bb9c750f4087f69ac8485cc90

  And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
  Minimal reproducible sample is the following:

  #define _GNU_SOURCE
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 

  /*
   System  | Result
  -+
   Native x86_64 4.12.14   | pwrite ret = 0
   Native aarch64 4.4.159  | pwrite ret = 0
   qemu-aarch64 at x86_64  | pwrite ret = -1
     ( 20d6c7312f1b8 ) |
  */

  int main(int argc, char** argv) {
   int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
   if (fd < 0) {
    perror("open");
    return 1;
   }

   int ret = fallocate(fd, 0, 0, 1000);
   if (ret < 0) {
    perror("fallocate");
    return 1;
   }

   ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
   printf("pwrite ret = %ld\n", ret_pwrite);

   close(fd);

   return 0;
  }

  
  Please note, that the same binary executable prints different output at 
native aarch64 platform and under aarch64-linux-user

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1810433/+subscriptions

[Qemu-devel] [Bug 1750229] Re: virtio-blk-pci regression: softlock in guest kernel at module loading

[Matwey V. Kornilov]

This has been fixed in the kernel


** Changed in: qemu
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1750229

Title:
  virtio-blk-pci regression: softlock in guest kernel at module loading

Status in QEMU:
  Fix Released

Bug description:
  Hello,

  I am running qemu from master git branch on x86_64 host with kernel is
  4.4.114. I've found that commit

  9a4c0e220d8a "hw/virtio-pci: fix virtio behaviour"

  introduces an regression with the following command:

  qemu-system-x86_64 -enable-kvm -nodefaults -no-reboot -nographic
  -vga none -runas qemu -kernel .build.kernel.kvm -initrd
  .build.initrd.kvm -append 'panic=1 softlockup_panic=1 no-kvmclock
  nmi_watchdog=0 console=ttyS0 root=/dev/disk/by-id/virtio-0' -m 2048
  -drive file=./root,format=raw,if=none,id=disk,serial=0,cache=unsafe
  -device virtio-blk-pci,drive=disk -serial stdio -smp 2

  Starting from this commit to master the following happens with a wide
  variety of guest kernels (4.4 to 4.15):

  [   62.428107] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=-20 
stuck for 59s!
  [   62.437426] Showing busy workqueues and worker pools:
  [   62.443117] workqueue events: flags=0x0
  [   62.447512]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
  [   62.448161] pending: check_corruption
  [   62.458570] workqueue kblockd: flags=0x18
  [   62.463082]   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=3/256
  [   62.463082] in-flight: 4:blk_mq_run_work_fn
  [   62.463082] pending: blk_mq_run_work_fn, blk_mq_timeout_work
  [   62.474831] pool 1: cpus=0 node=0 flags=0x0 nice=-20 hung=59s workers=2 
idle: 214
  [   62.492121] INFO: rcu_preempt detected stalls on CPUs/tasks:
  [   62.492121]  Tasks blocked on level-0 rcu_node (CPUs 0-1): P4
  [   62.492121]  (detected by 0, t=15002 jiffies, g=-130, c=-131, q=32)
  [   62.492121] kworker/0:0HR  running task0 4  2 
0x8000
  [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
  [   62.492121] Call Trace:
  [   62.492121]  
  [   62.492121]  sched_show_task+0xdf/0x100
  [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
  [   62.492121]  rcu_check_callbacks+0x93d/0x9d0
  [   62.492121]  ? tick_sched_do_timer+0x40/0x40
  [   62.492121]  update_process_times+0x28/0x50
  [   62.492121]  tick_sched_handle+0x22/0x70
  [   62.492121]  tick_sched_timer+0x34/0x70
  [   62.492121]  __hrtimer_run_queues+0xcc/0x250
  [   62.492121]  hrtimer_interrupt+0xab/0x1f0
  [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
  [   62.492121]  apic_timer_interrupt+0xa2/0xb0
  [   62.492121]  
  [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
  [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
ff11
  [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
0001
  [   62.492121] RDX: a477c0371000 RSI: a477c0371000 RDI: 

  [   62.492121] RBP: 0001 R08:  R09: 
01080020
  [   62.492121] R10: dc7cc1e4fc00 R11:  R12: 

  [   62.492121] R13:  R14: 92a1f93f R15: 
92a1f8e1aa80
  [   62.492121]  ? vp_synchronize_vectors+0x60/0x60
  [   62.492121]  vp_notify+0x12/0x20
  [   62.492121]  virtqueue_notify+0x18/0x30
  [   62.492121]  virtio_queue_rq+0x2f5/0x300 [virtio_blk]
  [   62.492121]  blk_mq_dispatch_rq_list+0x7e/0x4a0
  [   62.492121]  blk_mq_do_dispatch_sched+0x4a/0xd0
  [   62.492121]  blk_mq_sched_dispatch_requests+0x106/0x170
  [   62.492121]  __blk_mq_run_hw_queue+0x80/0x90
  [   62.492121]  process_one_work+0x1e3/0x420
  [   62.492121]  worker_thread+0x2b/0x3d0
  [   62.492121]  ? process_one_work+0x420/0x420
  [   62.492121]  kthread+0x113/0x130
  [   62.492121]  ? kthread_create_worker_on_cpu+0x50/0x50
  [   62.492121]  ret_from_fork+0x3a/0x50
  [   62.492121] kworker/0:0HR  running task0 4  2 
0x8000
  [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
  [   62.492121] Call Trace:
  [   62.492121]  
  [   62.492121]  sched_show_task+0xdf/0x100
  [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
  [   62.492121]  rcu_check_callbacks+0x972/0x9d0
  [   62.492121]  ? tick_sched_do_timer+0x40/0x40
  [   62.492121]  update_process_times+0x28/0x50
  [   62.492121]  tick_sched_handle+0x22/0x70
  [   62.492121]  tick_sched_timer+0x34/0x70
  [   62.492121]  __hrtimer_run_queues+0xcc/0x250
  [   62.492121]  hrtimer_interrupt+0xab/0x1f0
  [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
  [   62.492121]  apic_timer_interrupt+0xa2/0xb0
  [   62.492121]  
  [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
  [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
ff11
  [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
0001
  [   62.492121] RDX: a477c0371000 RSI: a477c0371000 

[Qemu-devel] [Bug 1810433] Re: aarch64-linux-user master: inconsistent pwrite behaviour

[Matwey V. Kornilov]

** Attachment added: "Test case source file"
   
https://bugs.launchpad.net/qemu/+bug/1810433/+attachment/5226714/+files/main.c

** Description changed:

  Hello,
  
  I am running aarch64-linux-user from master, commit
  20d6c7312f1b812bb9c750f4087f69ac8485cc90
  
  And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
  Minimal reproducible sample is the following:
  
  #define _GNU_SOURCE
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  
  /*
-  System  | Result
+  System  | Result
  -+
-  Native x86_64 4.12.14   | pwrite ret = 0
-  Native aarch64 4.4.159  | pwrite ret = 0
-  qemu-aarch64 at x86_64  | pwrite ret = -1
-( 20d6c7312f1b8 ) |
+  Native x86_64 4.12.14   | pwrite ret = 0
+  Native aarch64 4.4.159  | pwrite ret = 0
+  qemu-aarch64 at x86_64  | pwrite ret = -1
+    ( 20d6c7312f1b8 ) |
  */
  
  int main(int argc, char** argv) {
-   int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
-   if (fd < 0) {
-   perror("open");
-   return 1;
-   }
+  int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
+  if (fd < 0) {
+   perror("open");
+   return 1;
+  }
  
-   int ret = fallocate(fd, 0, 0, 1000);
-   if (ret < 0) {
-   perror("fallocate");
-   return 1;
-   }
+  int ret = fallocate(fd, 0, 0, 1000);
+  if (ret < 0) {
+   perror("fallocate");
+   return 1;
+  }
  
-   ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
-   printf("pwrite ret = %ld\n", ret_pwrite);
+  ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
+  printf("pwrite ret = %ld\n", ret_pwrite);
  
- close(fd);
+  close(fd);
  
-   return 0;
+  return 0;
  }
+ 
+ 
+ Please note, that the same binary executable prints different output at 
native aarch64 platform and under aarch64-linux-user

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1810433

Title:
  aarch64-linux-user master: inconsistent pwrite behaviour

Status in QEMU:
  New

Bug description:
  Hello,

  I am running aarch64-linux-user from master, commit
  20d6c7312f1b812bb9c750f4087f69ac8485cc90

  And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
  Minimal reproducible sample is the following:

  #define _GNU_SOURCE
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 

  /*
   System  | Result
  -+
   Native x86_64 4.12.14   | pwrite ret = 0
   Native aarch64 4.4.159  | pwrite ret = 0
   qemu-aarch64 at x86_64  | pwrite ret = -1
     ( 20d6c7312f1b8 ) |
  */

  int main(int argc, char** argv) {
   int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
   if (fd < 0) {
    perror("open");
    return 1;
   }

   int ret = fallocate(fd, 0, 0, 1000);
   if (ret < 0) {
    perror("fallocate");
    return 1;
   }

   ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
   printf("pwrite ret = %ld\n", ret_pwrite);

   close(fd);

   return 0;
  }

  
  Please note, that the same binary executable prints different output at 
native aarch64 platform and under aarch64-linux-user

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1810433/+subscriptions

[Qemu-devel] [Bug 1810433] [NEW] aarch64-linux-user master: inconsistent pwrite behaviour

[Matwey V. Kornilov]

Public bug reported:

Hello,

I am running aarch64-linux-user from master, commit
20d6c7312f1b812bb9c750f4087f69ac8485cc90

And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
Minimal reproducible sample is the following:

#define _GNU_SOURCE
#include 
#include 
#include 
#include 
#include 
#include 
#include 

/*
 System  | Result
-+
 Native x86_64 4.12.14   | pwrite ret = 0
 Native aarch64 4.4.159  | pwrite ret = 0
 qemu-aarch64 at x86_64  | pwrite ret = -1
   ( 20d6c7312f1b8 ) |
*/

int main(int argc, char** argv) {
 int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
 if (fd < 0) {
  perror("open");
  return 1;
 }

 int ret = fallocate(fd, 0, 0, 1000);
 if (ret < 0) {
  perror("fallocate");
  return 1;
 }

 ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
 printf("pwrite ret = %ld\n", ret_pwrite);

 close(fd);

 return 0;
}


Please note, that the same binary executable prints different output at native 
aarch64 platform and under aarch64-linux-user

** Affects: qemu
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1810433

Title:
  aarch64-linux-user master: inconsistent pwrite behaviour

Status in QEMU:
  New

Bug description:
  Hello,

  I am running aarch64-linux-user from master, commit
  20d6c7312f1b812bb9c750f4087f69ac8485cc90

  And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
  Minimal reproducible sample is the following:

  #define _GNU_SOURCE
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 

  /*
   System  | Result
  -+
   Native x86_64 4.12.14   | pwrite ret = 0
   Native aarch64 4.4.159  | pwrite ret = 0
   qemu-aarch64 at x86_64  | pwrite ret = -1
     ( 20d6c7312f1b8 ) |
  */

  int main(int argc, char** argv) {
   int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
   if (fd < 0) {
    perror("open");
    return 1;
   }

   int ret = fallocate(fd, 0, 0, 1000);
   if (ret < 0) {
    perror("fallocate");
    return 1;
   }

   ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
   printf("pwrite ret = %ld\n", ret_pwrite);

   close(fd);

   return 0;
  }

  
  Please note, that the same binary executable prints different output at 
native aarch64 platform and under aarch64-linux-user

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1810433/+subscriptions

[Qemu-devel] [Bug 1810433] Re: aarch64-linux-user master: inconsistent pwrite behaviour

[Matwey V. Kornilov]

** Attachment added: "Test binary statically compiled for aarch64"
   
https://bugs.launchpad.net/qemu/+bug/1810433/+attachment/5226715/+files/pwrite_test.aarch64

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1810433

Title:
  aarch64-linux-user master: inconsistent pwrite behaviour

Status in QEMU:
  New

Bug description:
  Hello,

  I am running aarch64-linux-user from master, commit
  20d6c7312f1b812bb9c750f4087f69ac8485cc90

  And I've found the following inconsistent emulation of pwrite() call when 
buf==NULL and len=0.
  Minimal reproducible sample is the following:

  #define _GNU_SOURCE
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 
  #include 

  /*
   System  | Result
  -+
   Native x86_64 4.12.14   | pwrite ret = 0
   Native aarch64 4.4.159  | pwrite ret = 0
   qemu-aarch64 at x86_64  | pwrite ret = -1
     ( 20d6c7312f1b8 ) |
  */

  int main(int argc, char** argv) {
   int fd = open("test.dat", O_CREAT | O_RDWR, 0644);
   if (fd < 0) {
    perror("open");
    return 1;
   }

   int ret = fallocate(fd, 0, 0, 1000);
   if (ret < 0) {
    perror("fallocate");
    return 1;
   }

   ssize_t ret_pwrite = pwrite(fd, NULL, 0, 0);
   printf("pwrite ret = %ld\n", ret_pwrite);

   close(fd);

   return 0;
  }

  
  Please note, that the same binary executable prints different output at 
native aarch64 platform and under aarch64-linux-user

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1810433/+subscriptions

[Qemu-devel] VM memory caching model

[Matwey V. Kornilov]

Hi,

Sorry in advance, if wrong maillist.

Where may I find comprehensive description how virtualization CPU
extensions (like VMX) interact with CPU data caches as well as
corresponding qemu implementation details? I've looked through
memory_ldst.inc.c with little success.

I am trying to debug virtio_blk issues found on Xeon X5675. I run nested
guest under qemu-kvm from ESXi 5.5 hosted guest, this is quite odd setup
but still. Currently (master qemu, host and guest - 4.15 kernel), virtio
is broken in this setup due to some kind of memory synchronization
issues. Let me please recall, that main virtio communication abstraction
is a queue. The queue is consisted from three parts: descriptor table,
avail ring, used ring. This structures are supposed to be shared memory
between the guest and the hypervisor. All of them are structures
allocated by the guest device driver, pointers in guest physical address
space are transferred to the hypervisor through PCI MMIO configuration
BAR. This is so-called modern PCI virtio.

When the guest wants to notify the hypervisor for update, the guest
device driver writes to PCI BAR. At the hypervisor side, notification is
implemented through eventfd and KVM_IOEVENTFD ioctl. It works as
expected, since I see that qemu receives a lot of notification for the
queue. However, qemu sees no updates in avail ring where guest driver
increments so called "avail index" at each transfer. At the hypervisor
side avail ring index is always 1, so as virtio_queue_empty_rcu() always
says that incoming queue is empty one.

Re: [Qemu-devel] [Bug 1750229] Re: virtio-blk-pci regression: softlock in guest kernel at module loading

[Matwey V. Kornilov]

The same story with 4.15.4 host kernel.

2018-02-21 11:58 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
> Well, last_avail_idx equals to shadow_avail_idx and both of them are 1
> at the qemu side. So, only one request is transferred.
> I wonder why, probably something is badly cached, but new avail_idx
> (which is supposed to become 2) is never shown up.
>
> 2018-02-20 15:49 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
>> virtqueue_pop() returns NULL due to virtio_queue_empty_rcu() returns
>> true all the time.
>>
>> 2018-02-20 14:47 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
>>> Well, I've found that on qemu side VirtQueue for virtio_blk device
>>> infinitely calls virtio_blk_handle_vq() where virtio_blk_get_request()
>>> (virtqueue_pop() in essens) always returns NULL.
>>>
>>> 2018-02-18 15:26 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
>>>> ** Attachment added: ".build.kernel.kvm"
>>>>
>>>> https://bugs.launchpad.net/qemu/+bug/1750229/+attachment/5057653/+files/.build.kernel.kvm
>>>>
>>>> --
>>>> You received this bug notification because you are subscribed to the bug
>>>> report.
>>>> https://bugs.launchpad.net/bugs/1750229
>>>>
>>>> Title:
>>>>   virtio-blk-pci regression: softlock in guest kernel at module loading
>>>>
>>>> Status in QEMU:
>>>>   New
>>>>
>>>> Bug description:
>>>>   Hello,
>>>>
>>>>   I am running qemu from master git branch on x86_64 host with kernel is
>>>>   4.4.114. I've found that commit
>>>>
>>>>   9a4c0e220d8a "hw/virtio-pci: fix virtio behaviour"
>>>>
>>>>   introduces an regression with the following command:
>>>>
>>>>   qemu-system-x86_64 -enable-kvm -nodefaults -no-reboot -nographic
>>>>   -vga none -runas qemu -kernel .build.kernel.kvm -initrd
>>>>   .build.initrd.kvm -append 'panic=1 softlockup_panic=1 no-kvmclock
>>>>   nmi_watchdog=0 console=ttyS0 root=/dev/disk/by-id/virtio-0' -m 2048
>>>>   -drive file=./root,format=raw,if=none,id=disk,serial=0,cache=unsafe
>>>>   -device virtio-blk-pci,drive=disk -serial stdio -smp 2
>>>>
>>>>   Starting from this commit to master the following happens with a wide
>>>>   variety of guest kernels (4.4 to 4.15):
>>>>
>>>>   [   62.428107] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 
>>>> nice=-20 stuck for 59s!
>>>>   [   62.437426] Showing busy workqueues and worker pools:
>>>>   [   62.443117] workqueue events: flags=0x0
>>>>   [   62.447512]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
>>>>   [   62.448161] pending: check_corruption
>>>>   [   62.458570] workqueue kblockd: flags=0x18
>>>>   [   62.463082]   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=3/256
>>>>   [   62.463082] in-flight: 4:blk_mq_run_work_fn
>>>>   [   62.463082] pending: blk_mq_run_work_fn, blk_mq_timeout_work
>>>>   [   62.474831] pool 1: cpus=0 node=0 flags=0x0 nice=-20 hung=59s 
>>>> workers=2 idle: 214
>>>>   [   62.492121] INFO: rcu_preempt detected stalls on CPUs/tasks:
>>>>   [   62.492121]  Tasks blocked on level-0 rcu_node (CPUs 0-1): P4
>>>>   [   62.492121]  (detected by 0, t=15002 jiffies, g=-130, c=-131, q=32)
>>>>   [   62.492121] kworker/0:0HR  running task0 4  2 
>>>> 0x8000
>>>>   [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
>>>>   [   62.492121] Call Trace:
>>>>   [   62.492121]  
>>>>   [   62.492121]  sched_show_task+0xdf/0x100
>>>>   [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
>>>>   [   62.492121]  rcu_check_callbacks+0x93d/0x9d0
>>>>   [   62.492121]  ? tick_sched_do_timer+0x40/0x40
>>>>   [   62.492121]  update_process_times+0x28/0x50
>>>>   [   62.492121]  tick_sched_handle+0x22/0x70
>>>>   [   62.492121]  tick_sched_timer+0x34/0x70
>>>>   [   62.492121]  __hrtimer_run_queues+0xcc/0x250
>>>>   [   62.492121]  hrtimer_interrupt+0xab/0x1f0
>>>>   [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
>>>>   [   62.492121]  apic_timer_interrupt+0xa2/0xb0
>>>>   [   62.492121]  
>>>>   [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
>>>>   [   62.492121] RSP: 

Re: [Qemu-devel] [Bug 1750229] Re: virtio-blk-pci regression: softlock in guest kernel at module loading

[Matwey V. Kornilov]

Well, last_avail_idx equals to shadow_avail_idx and both of them are 1
at the qemu side. So, only one request is transferred.
I wonder why, probably something is badly cached, but new avail_idx
(which is supposed to become 2) is never shown up.

2018-02-20 15:49 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
> virtqueue_pop() returns NULL due to virtio_queue_empty_rcu() returns
> true all the time.
>
> 2018-02-20 14:47 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
>> Well, I've found that on qemu side VirtQueue for virtio_blk device
>> infinitely calls virtio_blk_handle_vq() where virtio_blk_get_request()
>> (virtqueue_pop() in essens) always returns NULL.
>>
>> 2018-02-18 15:26 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
>>> ** Attachment added: ".build.kernel.kvm"
>>>
>>> https://bugs.launchpad.net/qemu/+bug/1750229/+attachment/5057653/+files/.build.kernel.kvm
>>>
>>> --
>>> You received this bug notification because you are subscribed to the bug
>>> report.
>>> https://bugs.launchpad.net/bugs/1750229
>>>
>>> Title:
>>>   virtio-blk-pci regression: softlock in guest kernel at module loading
>>>
>>> Status in QEMU:
>>>   New
>>>
>>> Bug description:
>>>   Hello,
>>>
>>>   I am running qemu from master git branch on x86_64 host with kernel is
>>>   4.4.114. I've found that commit
>>>
>>>   9a4c0e220d8a "hw/virtio-pci: fix virtio behaviour"
>>>
>>>   introduces an regression with the following command:
>>>
>>>   qemu-system-x86_64 -enable-kvm -nodefaults -no-reboot -nographic
>>>   -vga none -runas qemu -kernel .build.kernel.kvm -initrd
>>>   .build.initrd.kvm -append 'panic=1 softlockup_panic=1 no-kvmclock
>>>   nmi_watchdog=0 console=ttyS0 root=/dev/disk/by-id/virtio-0' -m 2048
>>>   -drive file=./root,format=raw,if=none,id=disk,serial=0,cache=unsafe
>>>   -device virtio-blk-pci,drive=disk -serial stdio -smp 2
>>>
>>>   Starting from this commit to master the following happens with a wide
>>>   variety of guest kernels (4.4 to 4.15):
>>>
>>>   [   62.428107] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 
>>> nice=-20 stuck for 59s!
>>>   [   62.437426] Showing busy workqueues and worker pools:
>>>   [   62.443117] workqueue events: flags=0x0
>>>   [   62.447512]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
>>>   [   62.448161] pending: check_corruption
>>>   [   62.458570] workqueue kblockd: flags=0x18
>>>   [   62.463082]   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=3/256
>>>   [   62.463082] in-flight: 4:blk_mq_run_work_fn
>>>   [   62.463082] pending: blk_mq_run_work_fn, blk_mq_timeout_work
>>>   [   62.474831] pool 1: cpus=0 node=0 flags=0x0 nice=-20 hung=59s 
>>> workers=2 idle: 214
>>>   [   62.492121] INFO: rcu_preempt detected stalls on CPUs/tasks:
>>>   [   62.492121]  Tasks blocked on level-0 rcu_node (CPUs 0-1): P4
>>>   [   62.492121]  (detected by 0, t=15002 jiffies, g=-130, c=-131, q=32)
>>>   [   62.492121] kworker/0:0HR  running task0 4  2 
>>> 0x8000
>>>   [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
>>>   [   62.492121] Call Trace:
>>>   [   62.492121]  
>>>   [   62.492121]  sched_show_task+0xdf/0x100
>>>   [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
>>>   [   62.492121]  rcu_check_callbacks+0x93d/0x9d0
>>>   [   62.492121]  ? tick_sched_do_timer+0x40/0x40
>>>   [   62.492121]  update_process_times+0x28/0x50
>>>   [   62.492121]  tick_sched_handle+0x22/0x70
>>>   [   62.492121]  tick_sched_timer+0x34/0x70
>>>   [   62.492121]  __hrtimer_run_queues+0xcc/0x250
>>>   [   62.492121]  hrtimer_interrupt+0xab/0x1f0
>>>   [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
>>>   [   62.492121]  apic_timer_interrupt+0xa2/0xb0
>>>   [   62.492121]  
>>>   [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
>>>   [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
>>> ff11
>>>   [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
>>> 0001
>>>   [   62.492121] RDX: a477c0371000 RSI: a477c0371000 RDI: 
>>> 
>>>   [   62.492121] RBP: 0001 R08:  R09: 
>>> 01080020
>>>   [   62.492121] R10: dc7cc1e4fc00 R11:

Re: [Qemu-devel] [Bug 1750229] Re: virtio-blk-pci regression: softlock in guest kernel at module loading

[Matwey V. Kornilov]

virtqueue_pop() returns NULL due to virtio_queue_empty_rcu() returns
true all the time.

2018-02-20 14:47 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
> Well, I've found that on qemu side VirtQueue for virtio_blk device
> infinitely calls virtio_blk_handle_vq() where virtio_blk_get_request()
> (virtqueue_pop() in essens) always returns NULL.
>
> 2018-02-18 15:26 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
>> ** Attachment added: ".build.kernel.kvm"
>>
>> https://bugs.launchpad.net/qemu/+bug/1750229/+attachment/5057653/+files/.build.kernel.kvm
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1750229
>>
>> Title:
>>   virtio-blk-pci regression: softlock in guest kernel at module loading
>>
>> Status in QEMU:
>>   New
>>
>> Bug description:
>>   Hello,
>>
>>   I am running qemu from master git branch on x86_64 host with kernel is
>>   4.4.114. I've found that commit
>>
>>   9a4c0e220d8a "hw/virtio-pci: fix virtio behaviour"
>>
>>   introduces an regression with the following command:
>>
>>   qemu-system-x86_64 -enable-kvm -nodefaults -no-reboot -nographic
>>   -vga none -runas qemu -kernel .build.kernel.kvm -initrd
>>   .build.initrd.kvm -append 'panic=1 softlockup_panic=1 no-kvmclock
>>   nmi_watchdog=0 console=ttyS0 root=/dev/disk/by-id/virtio-0' -m 2048
>>   -drive file=./root,format=raw,if=none,id=disk,serial=0,cache=unsafe
>>   -device virtio-blk-pci,drive=disk -serial stdio -smp 2
>>
>>   Starting from this commit to master the following happens with a wide
>>   variety of guest kernels (4.4 to 4.15):
>>
>>   [   62.428107] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 
>> nice=-20 stuck for 59s!
>>   [   62.437426] Showing busy workqueues and worker pools:
>>   [   62.443117] workqueue events: flags=0x0
>>   [   62.447512]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
>>   [   62.448161] pending: check_corruption
>>   [   62.458570] workqueue kblockd: flags=0x18
>>   [   62.463082]   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=3/256
>>   [   62.463082] in-flight: 4:blk_mq_run_work_fn
>>   [   62.463082] pending: blk_mq_run_work_fn, blk_mq_timeout_work
>>   [   62.474831] pool 1: cpus=0 node=0 flags=0x0 nice=-20 hung=59s workers=2 
>> idle: 214
>>   [   62.492121] INFO: rcu_preempt detected stalls on CPUs/tasks:
>>   [   62.492121]  Tasks blocked on level-0 rcu_node (CPUs 0-1): P4
>>   [   62.492121]  (detected by 0, t=15002 jiffies, g=-130, c=-131, q=32)
>>   [   62.492121] kworker/0:0HR  running task0 4  2 
>> 0x8000
>>   [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
>>   [   62.492121] Call Trace:
>>   [   62.492121]  
>>   [   62.492121]  sched_show_task+0xdf/0x100
>>   [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
>>   [   62.492121]  rcu_check_callbacks+0x93d/0x9d0
>>   [   62.492121]  ? tick_sched_do_timer+0x40/0x40
>>   [   62.492121]  update_process_times+0x28/0x50
>>   [   62.492121]  tick_sched_handle+0x22/0x70
>>   [   62.492121]  tick_sched_timer+0x34/0x70
>>   [   62.492121]  __hrtimer_run_queues+0xcc/0x250
>>   [   62.492121]  hrtimer_interrupt+0xab/0x1f0
>>   [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
>>   [   62.492121]  apic_timer_interrupt+0xa2/0xb0
>>   [   62.492121]  
>>   [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
>>   [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
>> ff11
>>   [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
>> 0001
>>   [   62.492121] RDX: a477c0371000 RSI: a477c0371000 RDI: 
>> 
>>   [   62.492121] RBP: 0001 R08:  R09: 
>> 01080020
>>   [   62.492121] R10: dc7cc1e4fc00 R11:  R12: 
>> 
>>   [   62.492121] R13:  R14: 92a1f93f R15: 
>> 92a1f8e1aa80
>>   [   62.492121]  ? vp_synchronize_vectors+0x60/0x60
>>   [   62.492121]  vp_notify+0x12/0x20
>>   [   62.492121]  virtqueue_notify+0x18/0x30
>>   [   62.492121]  virtio_queue_rq+0x2f5/0x300 [virtio_blk]
>>   [   62.492121]  blk_mq_dispatch_rq_list+0x7e/0x4a0
>>   [   62.492121]  blk_mq_do_dispatch_sched+0x4a/0xd0
>>   [   62.492121]  blk_mq_sched_dispatch_requests+0x106/0x170
>>   [   62.492121]  __blk_mq_run_hw_queue+0x80/0x90
>>   [   62.492121]  proce

Re: [Qemu-devel] [Bug 1750229] Re: virtio-blk-pci regression: softlock in guest kernel at module loading

[Matwey V. Kornilov]

Well, I've found that on qemu side VirtQueue for virtio_blk device
infinitely calls virtio_blk_handle_vq() where virtio_blk_get_request()
(virtqueue_pop() in essens) always returns NULL.

2018-02-18 15:26 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
> ** Attachment added: ".build.kernel.kvm"
>
> https://bugs.launchpad.net/qemu/+bug/1750229/+attachment/5057653/+files/.build.kernel.kvm
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1750229
>
> Title:
>   virtio-blk-pci regression: softlock in guest kernel at module loading
>
> Status in QEMU:
>   New
>
> Bug description:
>   Hello,
>
>   I am running qemu from master git branch on x86_64 host with kernel is
>   4.4.114. I've found that commit
>
>   9a4c0e220d8a "hw/virtio-pci: fix virtio behaviour"
>
>   introduces an regression with the following command:
>
>   qemu-system-x86_64 -enable-kvm -nodefaults -no-reboot -nographic
>   -vga none -runas qemu -kernel .build.kernel.kvm -initrd
>   .build.initrd.kvm -append 'panic=1 softlockup_panic=1 no-kvmclock
>   nmi_watchdog=0 console=ttyS0 root=/dev/disk/by-id/virtio-0' -m 2048
>   -drive file=./root,format=raw,if=none,id=disk,serial=0,cache=unsafe
>   -device virtio-blk-pci,drive=disk -serial stdio -smp 2
>
>   Starting from this commit to master the following happens with a wide
>   variety of guest kernels (4.4 to 4.15):
>
>   [   62.428107] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 
> nice=-20 stuck for 59s!
>   [   62.437426] Showing busy workqueues and worker pools:
>   [   62.443117] workqueue events: flags=0x0
>   [   62.447512]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
>   [   62.448161] pending: check_corruption
>   [   62.458570] workqueue kblockd: flags=0x18
>   [   62.463082]   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=3/256
>   [   62.463082] in-flight: 4:blk_mq_run_work_fn
>   [   62.463082] pending: blk_mq_run_work_fn, blk_mq_timeout_work
>   [   62.474831] pool 1: cpus=0 node=0 flags=0x0 nice=-20 hung=59s workers=2 
> idle: 214
>   [   62.492121] INFO: rcu_preempt detected stalls on CPUs/tasks:
>   [   62.492121]  Tasks blocked on level-0 rcu_node (CPUs 0-1): P4
>   [   62.492121]  (detected by 0, t=15002 jiffies, g=-130, c=-131, q=32)
>   [   62.492121] kworker/0:0HR  running task0 4  2 
> 0x8000
>   [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
>   [   62.492121] Call Trace:
>   [   62.492121]  
>   [   62.492121]  sched_show_task+0xdf/0x100
>   [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
>   [   62.492121]  rcu_check_callbacks+0x93d/0x9d0
>   [   62.492121]  ? tick_sched_do_timer+0x40/0x40
>   [   62.492121]  update_process_times+0x28/0x50
>   [   62.492121]  tick_sched_handle+0x22/0x70
>   [   62.492121]  tick_sched_timer+0x34/0x70
>   [   62.492121]  __hrtimer_run_queues+0xcc/0x250
>   [   62.492121]  hrtimer_interrupt+0xab/0x1f0
>   [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
>   [   62.492121]  apic_timer_interrupt+0xa2/0xb0
>   [   62.492121]  
>   [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
>   [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
> ff11
>   [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
> 0001
>   [   62.492121] RDX: a477c0371000 RSI: a477c0371000 RDI: 
> 
>   [   62.492121] RBP: 0001 R08:  R09: 
> 01080020
>   [   62.492121] R10: dc7cc1e4fc00 R11:  R12: 
> 
>   [   62.492121] R13:  R14: 92a1f93f R15: 
> 92a1f8e1aa80
>   [   62.492121]  ? vp_synchronize_vectors+0x60/0x60
>   [   62.492121]  vp_notify+0x12/0x20
>   [   62.492121]  virtqueue_notify+0x18/0x30
>   [   62.492121]  virtio_queue_rq+0x2f5/0x300 [virtio_blk]
>   [   62.492121]  blk_mq_dispatch_rq_list+0x7e/0x4a0
>   [   62.492121]  blk_mq_do_dispatch_sched+0x4a/0xd0
>   [   62.492121]  blk_mq_sched_dispatch_requests+0x106/0x170
>   [   62.492121]  __blk_mq_run_hw_queue+0x80/0x90
>   [   62.492121]  process_one_work+0x1e3/0x420
>   [   62.492121]  worker_thread+0x2b/0x3d0
>   [   62.492121]  ? process_one_work+0x420/0x420
>   [   62.492121]  kthread+0x113/0x130
>   [   62.492121]  ? kthread_create_worker_on_cpu+0x50/0x50
>   [   62.492121]  ret_from_fork+0x3a/0x50
>   [   62.492121] kworker/0:0HR  running task0 4  2 
> 0x8000
>   [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
>   [   62.492121] Call Trace:
>   [   62.492121]  
>   [   62.492121]  sched_show_task+0xdf/0x100
>

[Qemu-devel] [Bug 1750229] Re: virtio-blk-pci regression: softlock in guest kernel at module loading

[Matwey V. Kornilov]

** Attachment added: ".build.initrd.kvm"
   
https://bugs.launchpad.net/qemu/+bug/1750229/+attachment/5057654/+files/.build.initrd.kvm

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1750229

Title:
  virtio-blk-pci regression: softlock in guest kernel at module loading

Status in QEMU:
  New

Bug description:
  Hello,

  I am running qemu from master git branch on x86_64 host with kernel is
  4.4.114. I've found that commit

  9a4c0e220d8a "hw/virtio-pci: fix virtio behaviour"

  introduces an regression with the following command:

  qemu-system-x86_64 -enable-kvm -nodefaults -no-reboot -nographic
  -vga none -runas qemu -kernel .build.kernel.kvm -initrd
  .build.initrd.kvm -append 'panic=1 softlockup_panic=1 no-kvmclock
  nmi_watchdog=0 console=ttyS0 root=/dev/disk/by-id/virtio-0' -m 2048
  -drive file=./root,format=raw,if=none,id=disk,serial=0,cache=unsafe
  -device virtio-blk-pci,drive=disk -serial stdio -smp 2

  Starting from this commit to master the following happens with a wide
  variety of guest kernels (4.4 to 4.15):

  [   62.428107] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=-20 
stuck for 59s!
  [   62.437426] Showing busy workqueues and worker pools:
  [   62.443117] workqueue events: flags=0x0
  [   62.447512]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
  [   62.448161] pending: check_corruption
  [   62.458570] workqueue kblockd: flags=0x18
  [   62.463082]   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=3/256
  [   62.463082] in-flight: 4:blk_mq_run_work_fn
  [   62.463082] pending: blk_mq_run_work_fn, blk_mq_timeout_work
  [   62.474831] pool 1: cpus=0 node=0 flags=0x0 nice=-20 hung=59s workers=2 
idle: 214
  [   62.492121] INFO: rcu_preempt detected stalls on CPUs/tasks:
  [   62.492121]  Tasks blocked on level-0 rcu_node (CPUs 0-1): P4
  [   62.492121]  (detected by 0, t=15002 jiffies, g=-130, c=-131, q=32)
  [   62.492121] kworker/0:0HR  running task0 4  2 
0x8000
  [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
  [   62.492121] Call Trace:
  [   62.492121]  
  [   62.492121]  sched_show_task+0xdf/0x100
  [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
  [   62.492121]  rcu_check_callbacks+0x93d/0x9d0
  [   62.492121]  ? tick_sched_do_timer+0x40/0x40
  [   62.492121]  update_process_times+0x28/0x50
  [   62.492121]  tick_sched_handle+0x22/0x70
  [   62.492121]  tick_sched_timer+0x34/0x70
  [   62.492121]  __hrtimer_run_queues+0xcc/0x250
  [   62.492121]  hrtimer_interrupt+0xab/0x1f0
  [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
  [   62.492121]  apic_timer_interrupt+0xa2/0xb0
  [   62.492121]  
  [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
  [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
ff11
  [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
0001
  [   62.492121] RDX: a477c0371000 RSI: a477c0371000 RDI: 

  [   62.492121] RBP: 0001 R08:  R09: 
01080020
  [   62.492121] R10: dc7cc1e4fc00 R11:  R12: 

  [   62.492121] R13:  R14: 92a1f93f R15: 
92a1f8e1aa80
  [   62.492121]  ? vp_synchronize_vectors+0x60/0x60
  [   62.492121]  vp_notify+0x12/0x20
  [   62.492121]  virtqueue_notify+0x18/0x30
  [   62.492121]  virtio_queue_rq+0x2f5/0x300 [virtio_blk]
  [   62.492121]  blk_mq_dispatch_rq_list+0x7e/0x4a0
  [   62.492121]  blk_mq_do_dispatch_sched+0x4a/0xd0
  [   62.492121]  blk_mq_sched_dispatch_requests+0x106/0x170
  [   62.492121]  __blk_mq_run_hw_queue+0x80/0x90
  [   62.492121]  process_one_work+0x1e3/0x420
  [   62.492121]  worker_thread+0x2b/0x3d0
  [   62.492121]  ? process_one_work+0x420/0x420
  [   62.492121]  kthread+0x113/0x130
  [   62.492121]  ? kthread_create_worker_on_cpu+0x50/0x50
  [   62.492121]  ret_from_fork+0x3a/0x50
  [   62.492121] kworker/0:0HR  running task0 4  2 
0x8000
  [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
  [   62.492121] Call Trace:
  [   62.492121]  
  [   62.492121]  sched_show_task+0xdf/0x100
  [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
  [   62.492121]  rcu_check_callbacks+0x972/0x9d0
  [   62.492121]  ? tick_sched_do_timer+0x40/0x40
  [   62.492121]  update_process_times+0x28/0x50
  [   62.492121]  tick_sched_handle+0x22/0x70
  [   62.492121]  tick_sched_timer+0x34/0x70
  [   62.492121]  __hrtimer_run_queues+0xcc/0x250
  [   62.492121]  hrtimer_interrupt+0xab/0x1f0
  [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
  [   62.492121]  apic_timer_interrupt+0xa2/0xb0
  [   62.492121]  
  [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
  [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
ff11
  [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
0001
  [   62.492121] RDX: 

[Qemu-devel] [Bug 1750229] Re: virtio-blk-pci regression: softlock in guest kernel at module loading

[Matwey V. Kornilov]

** Attachment added: ".build.kernel.kvm"
   
https://bugs.launchpad.net/qemu/+bug/1750229/+attachment/5057653/+files/.build.kernel.kvm

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1750229

Title:
  virtio-blk-pci regression: softlock in guest kernel at module loading

Status in QEMU:
  New

Bug description:
  Hello,

  I am running qemu from master git branch on x86_64 host with kernel is
  4.4.114. I've found that commit

  9a4c0e220d8a "hw/virtio-pci: fix virtio behaviour"

  introduces an regression with the following command:

  qemu-system-x86_64 -enable-kvm -nodefaults -no-reboot -nographic
  -vga none -runas qemu -kernel .build.kernel.kvm -initrd
  .build.initrd.kvm -append 'panic=1 softlockup_panic=1 no-kvmclock
  nmi_watchdog=0 console=ttyS0 root=/dev/disk/by-id/virtio-0' -m 2048
  -drive file=./root,format=raw,if=none,id=disk,serial=0,cache=unsafe
  -device virtio-blk-pci,drive=disk -serial stdio -smp 2

  Starting from this commit to master the following happens with a wide
  variety of guest kernels (4.4 to 4.15):

  [   62.428107] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=-20 
stuck for 59s!
  [   62.437426] Showing busy workqueues and worker pools:
  [   62.443117] workqueue events: flags=0x0
  [   62.447512]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
  [   62.448161] pending: check_corruption
  [   62.458570] workqueue kblockd: flags=0x18
  [   62.463082]   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=3/256
  [   62.463082] in-flight: 4:blk_mq_run_work_fn
  [   62.463082] pending: blk_mq_run_work_fn, blk_mq_timeout_work
  [   62.474831] pool 1: cpus=0 node=0 flags=0x0 nice=-20 hung=59s workers=2 
idle: 214
  [   62.492121] INFO: rcu_preempt detected stalls on CPUs/tasks:
  [   62.492121]  Tasks blocked on level-0 rcu_node (CPUs 0-1): P4
  [   62.492121]  (detected by 0, t=15002 jiffies, g=-130, c=-131, q=32)
  [   62.492121] kworker/0:0HR  running task0 4  2 
0x8000
  [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
  [   62.492121] Call Trace:
  [   62.492121]  
  [   62.492121]  sched_show_task+0xdf/0x100
  [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
  [   62.492121]  rcu_check_callbacks+0x93d/0x9d0
  [   62.492121]  ? tick_sched_do_timer+0x40/0x40
  [   62.492121]  update_process_times+0x28/0x50
  [   62.492121]  tick_sched_handle+0x22/0x70
  [   62.492121]  tick_sched_timer+0x34/0x70
  [   62.492121]  __hrtimer_run_queues+0xcc/0x250
  [   62.492121]  hrtimer_interrupt+0xab/0x1f0
  [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
  [   62.492121]  apic_timer_interrupt+0xa2/0xb0
  [   62.492121]  
  [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
  [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
ff11
  [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
0001
  [   62.492121] RDX: a477c0371000 RSI: a477c0371000 RDI: 

  [   62.492121] RBP: 0001 R08:  R09: 
01080020
  [   62.492121] R10: dc7cc1e4fc00 R11:  R12: 

  [   62.492121] R13:  R14: 92a1f93f R15: 
92a1f8e1aa80
  [   62.492121]  ? vp_synchronize_vectors+0x60/0x60
  [   62.492121]  vp_notify+0x12/0x20
  [   62.492121]  virtqueue_notify+0x18/0x30
  [   62.492121]  virtio_queue_rq+0x2f5/0x300 [virtio_blk]
  [   62.492121]  blk_mq_dispatch_rq_list+0x7e/0x4a0
  [   62.492121]  blk_mq_do_dispatch_sched+0x4a/0xd0
  [   62.492121]  blk_mq_sched_dispatch_requests+0x106/0x170
  [   62.492121]  __blk_mq_run_hw_queue+0x80/0x90
  [   62.492121]  process_one_work+0x1e3/0x420
  [   62.492121]  worker_thread+0x2b/0x3d0
  [   62.492121]  ? process_one_work+0x420/0x420
  [   62.492121]  kthread+0x113/0x130
  [   62.492121]  ? kthread_create_worker_on_cpu+0x50/0x50
  [   62.492121]  ret_from_fork+0x3a/0x50
  [   62.492121] kworker/0:0HR  running task0 4  2 
0x8000
  [   62.492121] Workqueue: kblockd blk_mq_run_work_fn
  [   62.492121] Call Trace:
  [   62.492121]  
  [   62.492121]  sched_show_task+0xdf/0x100
  [   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
  [   62.492121]  rcu_check_callbacks+0x972/0x9d0
  [   62.492121]  ? tick_sched_do_timer+0x40/0x40
  [   62.492121]  update_process_times+0x28/0x50
  [   62.492121]  tick_sched_handle+0x22/0x70
  [   62.492121]  tick_sched_timer+0x34/0x70
  [   62.492121]  __hrtimer_run_queues+0xcc/0x250
  [   62.492121]  hrtimer_interrupt+0xab/0x1f0
  [   62.492121]  smp_apic_timer_interrupt+0x62/0x150
  [   62.492121]  apic_timer_interrupt+0xa2/0xb0
  [   62.492121]  
  [   62.492121] RIP: 0010:iowrite16+0x1d/0x30
  [   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
ff11
  [   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 
0001
  [   62.492121] RDX: 

[Qemu-devel] [Bug 1750229] [NEW] virtio-blk-pci regression: softlock in guest kernel at module loading

[Matwey V. Kornilov]

Public bug reported:

Hello,

I am running qemu from master git branch on x86_64 host with kernel is
4.4.114. I've found that commit

9a4c0e220d8a "hw/virtio-pci: fix virtio behaviour"

introduces an regression with the following command:

qemu-system-x86_64 -enable-kvm -nodefaults -no-reboot -nographic
-vga none -runas qemu -kernel .build.kernel.kvm -initrd
.build.initrd.kvm -append 'panic=1 softlockup_panic=1 no-kvmclock
nmi_watchdog=0 console=ttyS0 root=/dev/disk/by-id/virtio-0' -m 2048
-drive file=./root,format=raw,if=none,id=disk,serial=0,cache=unsafe
-device virtio-blk-pci,drive=disk -serial stdio -smp 2

Starting from this commit to master the following happens with a wide
variety of guest kernels (4.4 to 4.15):

[   62.428107] BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=-20 
stuck for 59s!
[   62.437426] Showing busy workqueues and worker pools:
[   62.443117] workqueue events: flags=0x0
[   62.447512]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256
[   62.448161] pending: check_corruption
[   62.458570] workqueue kblockd: flags=0x18
[   62.463082]   pwq 1: cpus=0 node=0 flags=0x0 nice=-20 active=3/256
[   62.463082] in-flight: 4:blk_mq_run_work_fn
[   62.463082] pending: blk_mq_run_work_fn, blk_mq_timeout_work
[   62.474831] pool 1: cpus=0 node=0 flags=0x0 nice=-20 hung=59s workers=2 
idle: 214
[   62.492121] INFO: rcu_preempt detected stalls on CPUs/tasks:
[   62.492121]  Tasks blocked on level-0 rcu_node (CPUs 0-1): P4
[   62.492121]  (detected by 0, t=15002 jiffies, g=-130, c=-131, q=32)
[   62.492121] kworker/0:0HR  running task0 4  2 0x8000
[   62.492121] Workqueue: kblockd blk_mq_run_work_fn
[   62.492121] Call Trace:
[   62.492121]  
[   62.492121]  sched_show_task+0xdf/0x100
[   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
[   62.492121]  rcu_check_callbacks+0x93d/0x9d0
[   62.492121]  ? tick_sched_do_timer+0x40/0x40
[   62.492121]  update_process_times+0x28/0x50
[   62.492121]  tick_sched_handle+0x22/0x70
[   62.492121]  tick_sched_timer+0x34/0x70
[   62.492121]  __hrtimer_run_queues+0xcc/0x250
[   62.492121]  hrtimer_interrupt+0xab/0x1f0
[   62.492121]  smp_apic_timer_interrupt+0x62/0x150
[   62.492121]  apic_timer_interrupt+0xa2/0xb0
[   62.492121]  
[   62.492121] RIP: 0010:iowrite16+0x1d/0x30
[   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
ff11
[   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 0001
[   62.492121] RDX: a477c0371000 RSI: a477c0371000 RDI: 
[   62.492121] RBP: 0001 R08:  R09: 01080020
[   62.492121] R10: dc7cc1e4fc00 R11:  R12: 
[   62.492121] R13:  R14: 92a1f93f R15: 92a1f8e1aa80
[   62.492121]  ? vp_synchronize_vectors+0x60/0x60
[   62.492121]  vp_notify+0x12/0x20
[   62.492121]  virtqueue_notify+0x18/0x30
[   62.492121]  virtio_queue_rq+0x2f5/0x300 [virtio_blk]
[   62.492121]  blk_mq_dispatch_rq_list+0x7e/0x4a0
[   62.492121]  blk_mq_do_dispatch_sched+0x4a/0xd0
[   62.492121]  blk_mq_sched_dispatch_requests+0x106/0x170
[   62.492121]  __blk_mq_run_hw_queue+0x80/0x90
[   62.492121]  process_one_work+0x1e3/0x420
[   62.492121]  worker_thread+0x2b/0x3d0
[   62.492121]  ? process_one_work+0x420/0x420
[   62.492121]  kthread+0x113/0x130
[   62.492121]  ? kthread_create_worker_on_cpu+0x50/0x50
[   62.492121]  ret_from_fork+0x3a/0x50
[   62.492121] kworker/0:0HR  running task0 4  2 0x8000
[   62.492121] Workqueue: kblockd blk_mq_run_work_fn
[   62.492121] Call Trace:
[   62.492121]  
[   62.492121]  sched_show_task+0xdf/0x100
[   62.492121]  rcu_print_detail_task_stall_rnp+0x48/0x69
[   62.492121]  rcu_check_callbacks+0x972/0x9d0
[   62.492121]  ? tick_sched_do_timer+0x40/0x40
[   62.492121]  update_process_times+0x28/0x50
[   62.492121]  tick_sched_handle+0x22/0x70
[   62.492121]  tick_sched_timer+0x34/0x70
[   62.492121]  __hrtimer_run_queues+0xcc/0x250
[   62.492121]  hrtimer_interrupt+0xab/0x1f0
[   62.492121]  smp_apic_timer_interrupt+0x62/0x150
[   62.492121]  apic_timer_interrupt+0xa2/0xb0
[   62.492121]  
[   62.492121] RIP: 0010:iowrite16+0x1d/0x30
[   62.492121] RSP: 0018:a477c034fcc8 EFLAGS: 00010292 ORIG_RAX: 
ff11
[   62.492121] RAX: a24fbdb0 RBX: 92a1f8f82000 RCX: 0001
[   62.492121] RDX: a477c0371000 RSI: a477c0371000 RDI: 
[   62.492121] RBP: 0001 R08:  R09: 01080020
[   62.492121] R10: dc7cc1e4fc00 R11:  R12: 
[   62.492121] R13:  R14: 92a1f93f R15: 92a1f8e1aa80
[   62.492121]  ? vp_synchronize_vectors+0x60/0x60
[   62.492121]  vp_notify+0x12/0x20
[   62.492121]  virtqueue_notify+0x18/0x30
[   62.492121]  virtio_queue_rq+0x2f5/0x300 [virtio_blk]
[   62.492121]  blk_mq_dispatch_rq_list+0x7e/0x4a0
[   62.492121]  

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

2017-07-23 12:54 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
> 2017-02-08 11:49 GMT+03:00 Paolo Bonzini <bonz...@gnu.org>:
>>> Does qemu follow recommendations from section 4.3?
>>
>> All that QEMU does is initialize MSR values and QEMU is talking to KVM,
>> not to the processor; KVM in turn talks to the host kernel's perf
>> subsystem.
>>
>> It's the host kernel's perf subsystem that needs to follow Intel's
>> recommendation.  In particular, QEMU is setting CPUID to the values
>> retrieved by
>>
>> perf_get_x86_pmu_capability();
>
> I can not find this function mentioned in qemu master sources.
>

Ok, I found this place in kvm kernel module. But it doesn't do what
you expect it to do. It just reassembles 0xA EAX from previously
parsed data.
IA32_MISC_ENABLE is not accessed anywhere here.

> The only thing I see is that has_msr_architectural_pmu is set to be
> true in kvm_arch_init_vcpu() if 0xA EAX has non-zero version. This is
> not enough according to the Intel specs.
>
>>
>> so perhaps it's perf_get_x86_pmu_capability that misreads the
>> performance monitoring capabilities provided by ESX.  Please attach
>> dmesg logs from starting the host with loglevel=9, as well as "x86info
>> -a" output from the host, to see if perf misses some problematic
>> CPUID/MSR combination.
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1661386
>>
>> Title:
>>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>>
>> Status in QEMU:
>>   New
>>
>> Bug description:
>>   Hello,
>>
>>
>>   I see the following when try to run qemu from master as the following:
>>
>>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>>   loglevel=7' -m 1024 -serial stdio
>>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>>
>>   First broken commit has been bisected:
>>
>>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>>   Author: Paolo Bonzini <pbonz...@redhat.com>
>>   Date:   Wed Mar 30 22:55:29 2016 +0200
>>
>>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>>
>>   This would have caught the bug in the previous patch.
>>
>>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>>
>>   My cpuinfo is the following:
>>
>>   processor   : 0
>>   vendor_id   : GenuineIntel
>>   cpu family  : 6
>>   model   : 44
>>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>>   stepping: 2
>>   microcode   : 0x14
>>   cpu MHz : 3066.775
>>   cache size  : 12288 KB
>>   physical id : 0
>>   siblings: 2
>>   core id : 0
>>   cpu cores   : 2
>>   apicid  : 0
>>   initial apicid  : 0
>>   fpu : yes
>>   fpu_exception   : yes
>>   cpuid level : 11
>>   wp  : yes
>>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
>> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
>> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
>> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
>> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>>   bugs:
>>   bogomips: 6133.55
>>   clflush size: 64
>>   cache_alignment : 64
>>   address sizes   : 40 bits physical, 48 bits virtual
>>   power management:
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions
>
>
>
> --
> With best regards,
> Matwey V. Kornilov


-- 
With best regards,
Matwey V. Kornilov

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run 

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

2017-02-08 11:49 GMT+03:00 Paolo Bonzini <bonz...@gnu.org>:
>> Does qemu follow recommendations from section 4.3?
>
> All that QEMU does is initialize MSR values and QEMU is talking to KVM,
> not to the processor; KVM in turn talks to the host kernel's perf
> subsystem.
>
> It's the host kernel's perf subsystem that needs to follow Intel's
> recommendation.  In particular, QEMU is setting CPUID to the values
> retrieved by
>
> perf_get_x86_pmu_capability();

I can not find this function mentioned in qemu master sources.

The only thing I see is that has_msr_architectural_pmu is set to be
true in kvm_arch_init_vcpu() if 0xA EAX has non-zero version. This is
not enough according to the Intel specs.

>
> so perhaps it's perf_get_x86_pmu_capability that misreads the
> performance monitoring capabilities provided by ESX.  Please attach
> dmesg logs from starting the host with loglevel=9, as well as "x86info
> -a" output from the host, to see if perf misses some problematic
> CPUID/MSR combination.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1661386
>
> Title:
>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>
> Status in QEMU:
>   New
>
> Bug description:
>   Hello,
>
>
>   I see the following when try to run qemu from master as the following:
>
>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>   loglevel=7' -m 1024 -serial stdio
>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
>   First broken commit has been bisected:
>
>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>   Author: Paolo Bonzini <pbonz...@redhat.com>
>   Date:   Wed Mar 30 22:55:29 2016 +0200
>
>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>
>   This would have caught the bug in the previous patch.
>
>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>
>   My cpuinfo is the following:
>
>   processor   : 0
>   vendor_id   : GenuineIntel
>   cpu family  : 6
>   model   : 44
>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>   stepping: 2
>   microcode   : 0x14
>   cpu MHz : 3066.775
>   cache size  : 12288 KB
>   physical id : 0
>   siblings: 2
>   core id : 0
>   cpu cores   : 2
>   apicid  : 0
>   initial apicid  : 0
>   fpu : yes
>   fpu_exception   : yes
>   cpuid level : 11
>   wp  : yes
>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>   bugs:
>   bogomips    : 6133.55
>   clflush size: 64
>   cache_alignment : 64
>   address sizes   : 40 bits physical, 48 bits virtual
>   power management:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions


-- 
With best regards,
Matwey V. Kornilov

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini <pbonz...@redhat.com>
  Date:   W

[Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

Hi,

I've attached the files with logs you requested. Could you comment them
somehow?

x86info says that IA32_PERF is not enabled:

Performance MSRs:
  MSR_IA32_PERF_STATUS: 0x0
  MSR_IA32_MISC_ENABLE: 0x0 [Enabled: ]

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini 
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
  Signed-off-by: Paolo Bonzini 

  My cpuinfo is the following:

  processor   : 0
  vendor_id   : GenuineIntel
  cpu family  : 6
  model   : 44
  model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
  stepping: 2
  microcode   : 0x14
  cpu MHz : 3066.775
  cache size  : 12288 KB
  physical id : 0
  siblings: 2
  core id : 0
  cpu cores   : 2
  apicid  : 0
  initial apicid  : 0
  fpu : yes
  fpu_exception   : yes
  cpuid level : 11
  wp  : yes
  flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
  bugs:
  bogomips: 6133.55
  clflush size: 64
  cache_alignment : 64
  address sizes   : 40 bits physical, 48 bits virtual
  power management:

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions

[Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

** Attachment added: "x86info.txt"
   
https://bugs.launchpad.net/qemu/+bug/1661386/+attachment/4815388/+files/x86info.txt

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini 
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
  Signed-off-by: Paolo Bonzini 

  My cpuinfo is the following:

  processor   : 0
  vendor_id   : GenuineIntel
  cpu family  : 6
  model   : 44
  model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
  stepping: 2
  microcode   : 0x14
  cpu MHz : 3066.775
  cache size  : 12288 KB
  physical id : 0
  siblings: 2
  core id : 0
  cpu cores   : 2
  apicid  : 0
  initial apicid  : 0
  fpu : yes
  fpu_exception   : yes
  cpuid level : 11
  wp  : yes
  flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
  bugs:
  bogomips: 6133.55
  clflush size: 64
  cache_alignment : 64
  address sizes   : 40 bits physical, 48 bits virtual
  power management:

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions

[Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

** Attachment added: "dmesg-loglevel9"
   
https://bugs.launchpad.net/qemu/+bug/1661386/+attachment/4815387/+files/dmesg-loglevel9

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini 
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
  Signed-off-by: Paolo Bonzini 

  My cpuinfo is the following:

  processor   : 0
  vendor_id   : GenuineIntel
  cpu family  : 6
  model   : 44
  model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
  stepping: 2
  microcode   : 0x14
  cpu MHz : 3066.775
  cache size  : 12288 KB
  physical id : 0
  siblings: 2
  core id : 0
  cpu cores   : 2
  apicid  : 0
  initial apicid  : 0
  fpu : yes
  fpu_exception   : yes
  cpuid level : 11
  wp  : yes
  flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
  bugs:
  bogomips: 6133.55
  clflush size: 64
  cache_alignment : 64
  address sizes   : 40 bits physical, 48 bits virtual
  power management:

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

2017-02-06 22:38 GMT+03:00 Matwey V. Kornilov <matwey.korni...@gmail.com>:
> 2017-02-06 21:05 GMT+03:00 Dr. David Alan Gilbert <dgilb...@redhat.com>:
>>>> So you didn't mention this was running inside VMWare; it looks to me as if 
>>>> that's rejecting the PMU MSR accesses.
>>>> For reference which version of VMWare are you using?
>>
>>>ESXi 6.0.0 Build 2494585
>>
>>>I also find that enabling perf counters in VMWare configuration also
>> helps.
>>
>> OK, so that suggests the problem is that with PMU disabled in VMWare
>> config, it's not giving the right info to the guest to know it's
>> disabled.
>
> How should it provide info? Can we check it?
>

Hi,

I've found the following doc:

https://software.intel.com/sites/default/files/m/5/2/c/f/1/30320
-Nehalem-PMU-Programming-Guide-Core.pdf

I am not sure how up-to-date it is. Does qemu follow recommendations
from secton 4.3?
I use msr-tools package and rdmsr tool to check some registers from table 26...
IA32_MISC_ENABLE = 0
IA32_PERF_CAPABILITIES = 0
in my case.

>>
>>>But why did it just work before 48e1a45c3166 with perf counters
>> disabled?
>>
>> Before that bug it ignored the failure to write/read the PMU MSRs - but
>> also lost all the MSRs after the PMU access and we'd found that if we
>> ever had that happen we'd get lots of weird bugs related to the other
>> MSRs.
>>
>>>>
>>>> My colleague suggested that '-cpu host,pmu=off'  might work instead of
>>>> having to hack around with the source.
>>
>>> Indeed, this also helps.
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1661386
>>
>> Title:
>>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>>
>> Status in QEMU:
>>   New
>>
>> Bug description:
>>   Hello,
>>
>>
>>   I see the following when try to run qemu from master as the following:
>>
>>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>>   loglevel=7' -m 1024 -serial stdio
>>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>>
>>   First broken commit has been bisected:
>>
>>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>>   Author: Paolo Bonzini <pbonz...@redhat.com>
>>   Date:   Wed Mar 30 22:55:29 2016 +0200
>>
>>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>>
>>   This would have caught the bug in the previous patch.
>>
>>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>>
>>   My cpuinfo is the following:
>>
>>   processor   : 0
>>   vendor_id   : GenuineIntel
>>   cpu family  : 6
>>   model   : 44
>>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>>   stepping: 2
>>   microcode   : 0x14
>>   cpu MHz : 3066.775
>>   cache size  : 12288 KB
>>   physical id : 0
>>   siblings: 2
>>   core id : 0
>>   cpu cores   : 2
>>   apicid  : 0
>>   initial apicid  : 0
>>   fpu : yes
>>   fpu_exception   : yes
>>   cpuid level : 11
>>   wp  : yes
>>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
>> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
>> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
>> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
>> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>>   bugs:
>>   bogomips: 6133.55
>>   clflush size: 64
>>   cache_alignment : 64
>>   address sizes   : 40 bits physical, 48 bits virtual
>>   power management:
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions
>
>
>
> --
> With best regards,
> Matwey V. Kornilov
> http://blog.matwey.name
> xmpp://0x2...@jabber.ru


-- 
With best regards,
Matwey V. K

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

2017-02-06 21:05 GMT+03:00 Dr. David Alan Gilbert <dgilb...@redhat.com>:
>>> So you didn't mention this was running inside VMWare; it looks to me as if 
>>> that's rejecting the PMU MSR accesses.
>>> For reference which version of VMWare are you using?
>
>>ESXi 6.0.0 Build 2494585
>
>>I also find that enabling perf counters in VMWare configuration also
> helps.
>
> OK, so that suggests the problem is that with PMU disabled in VMWare
> config, it's not giving the right info to the guest to know it's
> disabled.

How should it provide info? Can we check it?

>
>>But why did it just work before 48e1a45c3166 with perf counters
> disabled?
>
> Before that bug it ignored the failure to write/read the PMU MSRs - but
> also lost all the MSRs after the PMU access and we'd found that if we
> ever had that happen we'd get lots of weird bugs related to the other
> MSRs.
>
>>>
>>> My colleague suggested that '-cpu host,pmu=off'  might work instead of
>>> having to hack around with the source.
>
>> Indeed, this also helps.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1661386
>
> Title:
>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>
> Status in QEMU:
>   New
>
> Bug description:
>   Hello,
>
>
>   I see the following when try to run qemu from master as the following:
>
>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>   loglevel=7' -m 1024 -serial stdio
>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
>   First broken commit has been bisected:
>
>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>   Author: Paolo Bonzini <pbonz...@redhat.com>
>   Date:   Wed Mar 30 22:55:29 2016 +0200
>
>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>
>   This would have caught the bug in the previous patch.
>
>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>
>   My cpuinfo is the following:
>
>   processor   : 0
>   vendor_id   : GenuineIntel
>   cpu family  : 6
>   model   : 44
>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>   stepping: 2
>   microcode   : 0x14
>   cpu MHz : 3066.775
>   cache size  : 12288 KB
>   physical id : 0
>   siblings: 2
>   core id : 0
>   cpu cores   : 2
>   apicid  : 0
>   initial apicid  : 0
>   fpu : yes
>   fpu_exception   : yes
>   cpuid level : 11
>   wp  : yes
>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>   bugs:
>   bogomips: 6133.55
>   clflush size: 64
>   cache_alignment : 64
>   address sizes   : 40 bits physical, 48 bits virtual
>   power management:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions


-- 
With best regards,
Matwey V. Kornilov
http://blog.matwey.name
xmpp://0x2...@jabber.ru

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit h

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

2017-02-06 20:11 GMT+03:00 Dr. David Alan Gilbert <dgilb...@redhat.com>:
> Ahha!
> [0.00] DMI: VMware, Inc. VMware Virtual Platform/440BX Desktop 
> Reference Platform, BIOS 6.00 09/30/2014
> [0.00] Hypervisor detected: VMware
>
> So you didn't mention this was running inside VMWare; it looks to me as if 
> that's rejecting the PMU MSR accesses.
> For reference which version of VMWare are you using?

ESXi 6.0.0 Build 2494585

I also find that enabling perf counters in VMWare configuration also helps.
But why did it just work before 48e1a45c3166 with perf counters disabled?

>
> My colleague suggested that '-cpu host,pmu=off'  might work instead of
> having to hack around with the source.

Indeed, this also helps.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1661386
>
> Title:
>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>
> Status in QEMU:
>   New
>
> Bug description:
>   Hello,
>
>
>   I see the following when try to run qemu from master as the following:
>
>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>   loglevel=7' -m 1024 -serial stdio
>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
>   First broken commit has been bisected:
>
>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>   Author: Paolo Bonzini <pbonz...@redhat.com>
>   Date:   Wed Mar 30 22:55:29 2016 +0200
>
>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>
>   This would have caught the bug in the previous patch.
>
>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>
>   My cpuinfo is the following:
>
>   processor   : 0
>   vendor_id   : GenuineIntel
>   cpu family  : 6
>   model   : 44
>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>   stepping: 2
>   microcode   : 0x14
>   cpu MHz : 3066.775
>   cache size  : 12288 KB
>   physical id : 0
>   siblings: 2
>   core id : 0
>   cpu cores   : 2
>   apicid  : 0
>   initial apicid  : 0
>   fpu : yes
>   fpu_exception   : yes
>   cpuid level : 11
>   wp  : yes
>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>   bugs:
>   bogomips: 6133.55
>   clflush size: 64
>   cache_alignment : 64
>   address sizes   : 40 bits physical, 48 bits virtual
>   power management:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions


-- 
With best regards,
Matwey V. Kornilov
http://blog.matwey.name
xmpp://0x2...@jabber.ru

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini <pbonz...@redhat.com>
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
  Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>

  My cpuinfo i

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

2017-02-06 13:02 GMT+03:00 Dr. David Alan Gilbert <dgilb...@redhat.com>:
> Hi Matwey,
>   1) Can you provide me with the output of the 'dmesg' command straight after 
> boot on your host.

I've attached dmesg. I had to do this from beginning.

>   2) If you look in target/i386/kvm.c in kvm_arch_init_vcpu around
line 871 is some code like:

kvm_arch_init_vcpu ver=7300402

Indeed, the guest kernel started.

>
> if ((ver & 0xff) > 0) {
> has_msr_architectural_pmu = true;
> num_architectural_pmu_counters = (ver & 0xff00) >> 8;
>
> /* Shouldn't be more than 32, since that's the number of bits
>  * available in EBX to tell us _which_ counters are available.
>  * Play it safe.
>  */
> if (num_architectural_pmu_counters > MAX_GP_COUNTERS) {
> num_architectural_pmu_counters = MAX_GP_COUNTERS;
> }
>
> change the start of that to :
> fprintf(stderr, "kvm_arch_init_vcpu ver=%x\n", ver);
> if (0) {
>
> I think that might make it work, but please tell us what it prints
> as ver=
>
> Dave
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1661386
>
> Title:
>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>
> Status in QEMU:
>   New
>
> Bug description:
>   Hello,
>
>
>   I see the following when try to run qemu from master as the following:
>
>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>   loglevel=7' -m 1024 -serial stdio
>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
>   First broken commit has been bisected:
>
>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>   Author: Paolo Bonzini <pbonz...@redhat.com>
>   Date:   Wed Mar 30 22:55:29 2016 +0200
>
>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>
>   This would have caught the bug in the previous patch.
>
>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>
>   My cpuinfo is the following:
>
>   processor   : 0
>   vendor_id   : GenuineIntel
>   cpu family  : 6
>   model   : 44
>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>   stepping: 2
>   microcode   : 0x14
>   cpu MHz : 3066.775
>   cache size  : 12288 KB
>   physical id : 0
>   siblings: 2
>   core id : 0
>   cpu cores   : 2
>   apicid  : 0
>   initial apicid  : 0
>   fpu : yes
>   fpu_exception   : yes
>   cpuid level : 11
>   wp  : yes
>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>   bugs:
>   bogomips: 6133.55
>   clflush size: 64
>   cache_alignment : 64
>   address sizes   : 40 bits physical, 48 bits virtual
>   power management:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions


-- 
With best regards,
Matwey V. Kornilov
http://blog.matwey.name
xmpp://0x2...@jabber.ru

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assert

[Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

** Attachment added: "dmesg"
   
https://bugs.launchpad.net/qemu/+bug/1661386/+attachment/4814179/+files/dmesg-4.0

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini 
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
  Signed-off-by: Paolo Bonzini 

  My cpuinfo is the following:

  processor   : 0
  vendor_id   : GenuineIntel
  cpu family  : 6
  model   : 44
  model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
  stepping: 2
  microcode   : 0x14
  cpu MHz : 3066.775
  cache size  : 12288 KB
  physical id : 0
  siblings: 2
  core id : 0
  cpu cores   : 2
  apicid  : 0
  initial apicid  : 0
  fpu : yes
  fpu_exception   : yes
  cpuid level : 11
  wp  : yes
  flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
  bugs:
  bogomips: 6133.55
  clflush size: 64
  cache_alignment : 64
  address sizes   : 40 bits physical, 48 bits virtual
  power management:

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

2017-02-03 22:51 GMT+03:00 Dr. David Alan Gilbert <dgilb...@redhat.com>:
> Ah well that is a bit better; you see now it's failing in kvm_**get**_msrs 
> rather
> than put; so the question is which of the two changes made it survive 
> kvm_put_msrs
>
> I'd hoped that the flags in (2) would have turned off the CPU flag and
> thus made it go in both of them.
>
> kvm_msr_entry_add: @103 index=20f value=0
> qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:2218:
> kvm_get_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
> 1) Was it the steal time or the pmu change that made it flip over to the 
> get_msrs?

It was has_msr_architectural_pmu.

> 2) Can you get it to flip over to the get_msrs with the flag rather
than the code change?

Only using code change.

>
> Dave
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1661386
>
> Title:
>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>
> Status in QEMU:
>   New
>
> Bug description:
>   Hello,
>
>
>   I see the following when try to run qemu from master as the following:
>
>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>   loglevel=7' -m 1024 -serial stdio
>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
>   First broken commit has been bisected:
>
>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>   Author: Paolo Bonzini <pbonz...@redhat.com>
>   Date:   Wed Mar 30 22:55:29 2016 +0200
>
>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>
>   This would have caught the bug in the previous patch.
>
>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>
>   My cpuinfo is the following:
>
>   processor   : 0
>   vendor_id   : GenuineIntel
>   cpu family  : 6
>   model   : 44
>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>   stepping: 2
>   microcode   : 0x14
>   cpu MHz : 3066.775
>   cache size  : 12288 KB
>   physical id : 0
>   siblings: 2
>   core id : 0
>   cpu cores   : 2
>   apicid  : 0
>   initial apicid  : 0
>   fpu : yes
>   fpu_exception   : yes
>   cpuid level : 11
>   wp  : yes
>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>   bugs:
>   bogomips    : 6133.55
>   clflush size: 64
>   cache_alignment : 64
>   address sizes   : 40 bits physical, 48 bits virtual
>   power management:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions


-- 
With best regards,
Matwey V. Kornilov
http://blog.matwey.name
xmpp://0x2...@jabber.ru

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini <pbonz...@redhat.com>
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
   

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

n
>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>   loglevel=7' -m 1024 -serial stdio
>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
>   First broken commit has been bisected:
>
>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>   Author: Paolo Bonzini <pbonz...@redhat.com>
>   Date:   Wed Mar 30 22:55:29 2016 +0200
>
>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>
>   This would have caught the bug in the previous patch.
>
>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>
>   My cpuinfo is the following:
>
>   processor   : 0
>   vendor_id   : GenuineIntel
>   cpu family  : 6
>   model   : 44
>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>   stepping: 2
>   microcode   : 0x14
>   cpu MHz : 3066.775
>   cache size  : 12288 KB
>   physical id : 0
>   siblings: 2
>   core id : 0
>   cpu cores   : 2
>   apicid  : 0
>   initial apicid  : 0
>   fpu : yes
>   fpu_exception   : yes
>   cpuid level : 11
>   wp  : yes
>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>   bugs:
>   bogomips: 6133.55
>   clflush size: 64
>   cache_alignment : 64
>   address sizes   : 40 bits physical, 48 bits virtual
>   power management:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions


-- 
With best regards,
Matwey V. Kornilov
http://blog.matwey.name
xmpp://0x2...@jabber.ru

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini <pbonz...@redhat.com>
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
  Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>

  My cpuinfo is the following:

  processor   : 0
  vendor_id   : GenuineIntel
  cpu family  : 6
  model   : 44
  model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
  stepping: 2
  microcode   : 0x14
  cpu MHz : 3066.775
  cache size  : 12288 KB
  physical id : 0
  siblings: 2
  core id : 0
  cpu cores   : 2
  apicid  : 0
  initial apicid  : 0
  fpu : yes
  fpu_exception   : yes
  cpuid level : 11
  wp  : yes
  flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
  bugs:
  bogomips: 6133.55
  clflush size: 64
  cache_alignment : 64
  address sizes   : 40 bits physical, 48 bits virtual
  power management:

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions

Re: [Qemu-devel] [Bug 1661386] Re: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

ans the failure probably previously existed and was ignored and that causes 
> random other problems.
>
> What kernel are you using on the host?
>
> We need to figure out which MSR it's objecting to; probably the easiest
> way is to :
>
> 1) Edit mvm_msr_entry_add in target/i386/kvm.c to something like:
>
> assert((void *)(entry + 1) <= limit);
> fprintf(stderr,"kvm_msr_entry_add: @%d index=%x value=%lx\n", 
> msrs->nmsrs, index, value);
> entry->index = index;
>
> 2) edit kvm_put_msrs near the bottom:
>
> fprintf(stderr,"kvm_put_msrs: ret=%d expected=%d\n", ret, 
> cpu->kvm_msr_buf->nmsrs);
> assert(ret == cpu->kvm_msr_buf->nmsrs);
>
> Now with any luck the 'ret' value will tell you the entry which is bad, and 
> you can match
> that to the @%d value (or maybe it's the entry before that one which failed?) 
> then we get the index, look it up in the intel docs and figure out which MSR 
> it's complaining about.
>
> Dave
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1661386
>
> Title:
>   Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed
>
> Status in QEMU:
>   New
>
> Bug description:
>   Hello,
>
>
>   I see the following when try to run qemu from master as the following:
>
>   # ./x86_64-softmmu/qemu-system-x86_64 --version
>   QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
>   Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
>   # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
>   -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
>   -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
>   loglevel=7' -m 1024 -serial stdio
>   qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
>   kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
>
>   First broken commit has been bisected:
>
>   commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
>   Author: Paolo Bonzini <pbonz...@redhat.com>
>   Date:   Wed Mar 30 22:55:29 2016 +0200
>
>   target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
>
>   This would have caught the bug in the previous patch.
>
>   Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
>
>   My cpuinfo is the following:
>
>   processor   : 0
>   vendor_id   : GenuineIntel
>   cpu family  : 6
>   model   : 44
>   model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
>   stepping: 2
>   microcode   : 0x14
>   cpu MHz : 3066.775
>   cache size  : 12288 KB
>   physical id : 0
>   siblings: 2
>   core id : 0
>   cpu cores   : 2
>   apicid  : 0
>   initial apicid  : 0
>   fpu : yes
>   fpu_exception   : yes
>   cpuid level : 11
>   wp  : yes
>   flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
> constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
> aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
> lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
>   bugs:
>   bogomips: 6133.55
>   clflush size: 64
>   cache_alignment : 64
>   address sizes   : 40 bits physical, 48 bits virtual
>   power management:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions


-- 
With best regards,
Matwey V. Kornilov
http://blog.matwey.name
xmpp://0x2...@jabber.ru

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini <pbon

[Qemu-devel] [Bug 1661386] [NEW] Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

[Matwey V. Kornilov]

Public bug reported:

Hello,


I see the following when try to run qemu from master as the following:

# ./x86_64-softmmu/qemu-system-x86_64 --version
QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
# ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
-no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
-initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
loglevel=7' -m 1024 -serial stdio
qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

First broken commit has been bisected:

commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
Author: Paolo Bonzini 
Date:   Wed Mar 30 22:55:29 2016 +0200

target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs

This would have caught the bug in the previous patch.

Signed-off-by: Paolo Bonzini 

My cpuinfo is the following:

processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
stepping: 2
microcode   : 0x14
cpu MHz : 3066.775
cache size  : 12288 KB
physical id : 0
siblings: 2
core id : 0
cpu cores   : 2
apicid  : 0
initial apicid  : 0
fpu : yes
fpu_exception   : yes
cpuid level : 11
wp  : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov 
pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc 
arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni 
pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat 
epb dtherm tpr_shadow vnmi ept vpid
bugs:
bogomips: 6133.55
clflush size: 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

** Affects: qemu
 Importance: Undecided
 Status: New

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1661386

Title:
  Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed

Status in QEMU:
  New

Bug description:
  Hello,

  
  I see the following when try to run qemu from master as the following:

  # ./x86_64-softmmu/qemu-system-x86_64 --version
  QEMU emulator version 2.8.50 (v2.8.0-1006-g4e9f524)
  Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers
  # ./x86_64-softmmu/qemu-system-x86_64 -machine accel=kvm -nodefaults
  -no-reboot -nographic -cpu host -vga none  -kernel .build.kernel.kvm
  -initrd .build.initrd.kvm -append 'panic=1 no-kvmclock console=ttyS0
  loglevel=7' -m 1024 -serial stdio
  qemu-system-x86_64: /home/matwey/lab/qemu/target/i386/kvm.c:1849:
  kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

  First broken commit has been bisected:

  commit 48e1a45c3166d659f781171a47dabf4a187ed7a5
  Author: Paolo Bonzini 
  Date:   Wed Mar 30 22:55:29 2016 +0200

  target-i386: assert that KVM_GET/SET_MSRS can set all requested MSRs
  
  This would have caught the bug in the previous patch.
  
  Signed-off-by: Paolo Bonzini 

  My cpuinfo is the following:

  processor   : 0
  vendor_id   : GenuineIntel
  cpu family  : 6
  model   : 44
  model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
  stepping: 2
  microcode   : 0x14
  cpu MHz : 3066.775
  cache size  : 12288 KB
  physical id : 0
  siblings: 2
  core id : 0
  cpu cores   : 2
  apicid  : 0
  initial apicid  : 0
  fpu : yes
  fpu_exception   : yes
  cpuid level : 11
  wp  : yes
  flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm 
constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc 
aperfmperf pni pclmulqdq vmx ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor 
lahf_lm ida arat epb dtherm tpr_shadow vnmi ept vpid
  bugs:
  bogomips: 6133.55
  clflush size: 64
  cache_alignment : 64
  address sizes   : 40 bits physical, 48 bits virtual
  power management:

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1661386/+subscriptions