Re: [RFC PATCH 4/6] migration: Add zstd support in multi-thread compression

[Zeyu Jin]

On 2020/11/27 20:10, Markus Armbruster wrote:
> Zeyu Jin  writes:
> 
>> This patch enables zstd option in multi-thread compression.
>>
>> Signed-off-by: Zeyu Jin 
>> Signed-off-by: Ying Fang 
> [...]
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index d262683a38..ac6d06c683 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -536,7 +536,7 @@
>>  #
>>  ##
>>  { 'enum': 'CompressMethod',
>> -  'data': [ 'zlib' ] }
>> +  'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>>  
>>  ##
>>  # @BitmapMigrationBitmapAlias:
> 
> Please document the new enum value @zstd.

I will document that. Thank you.

> 
> .
> 

Re: [PATCH 1/6] migration: Add multi-thread compress method

[Zeyu Jin]

On 2020/11/27 17:48, Markus Armbruster wrote:
> Kevin, Max, suggest to skip right to Qcow2CompressionType.
> 
> Zeyu Jin  writes:
> 
>> A multi-thread compress method parameter is added to hold the method we
>> are going to use. By default the 'zlib' method is used to maintain the
>> compatibility as before.
>>
>> Signed-off-by: Zeyu Jin 
>> Signed-off-by: Ying Fang 
> [...]
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 3c75820527..2ed6a55b92 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -525,6 +525,19 @@
>>'data': [ 'none', 'zlib',
>>  { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>>  
>> +##
>> +# @CompressMethod:
>> +#
>> +# An enumeration of multi-thread compression methods.
>> +#
>> +# @zlib: use zlib compression method.
>> +#
>> +# Since: 6.0
>> +#
>> +##
>> +{ 'enum': 'CompressMethod',
>> +  'data': [ 'zlib' ] }
>> +
>>  ##
>>  # @BitmapMigrationBitmapAlias:
>>  #
>> @@ -599,6 +612,9 @@
>>  #  compression, so set the decompress-threads to the 
>> number about 1/4
>>  #  of compress-threads is adequate.
>>  #
>> +# @compress-method: Set compression method to use in multi-thread 
>> compression.
>> +#   Defaults to zlib. (Since 6.0)
> 
> We already have @multifd-compression.  Why do we need to control the two
> separately?  Can you give a use case for different settings?
> 

Generally, mulit-thread compression deals with the situation
where network bandwith is limited but cpu resource is adequate. Multifd
instead aims to situation where single fd cannot take full advantage of
network bandwith. So compression based on multifd cannot fully cover the
cases for multi-thread compression.

For example, for migration with a bandwith limitation of 10M
bytes/second, single fd is enough for data delivery. This is the case
for multi-thread compression.

> If we do want two parameters: the names @compress-method and
> @multifd-compression are inconsistent.  According to your comment,
> @compress-method applies only to multi-thread compression.  That leads
> me to suggest renaming it to @multi-thread-compression.
>

For the names, my original idea is to make 'CompressMethod' consistent
with other multi-thread compression parameters, like 'compress-threads'
and 'compress-level'. There is truly some inconsistency here, between
multifd-compression params and old multi-thread compression params.

For now, I agree with you that 'multi-thread-compression' is better. It
is more specific and clear. I will rename the params in next version.
Thanks for the suggestion.

> After PATCH 4, CompressMethod is almost the same as MultiFDCompression:
> 
>{ 'enum': 'CompressMethod',
>  'data': [ 'zlib' ] }
>  'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
> 
>{ 'enum': 'MultiFDCompression',
>  'data': [ 'none', 'zlib',
>{ 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
> 
> The difference is member 'none'.  Why does compression 'none' make sense
> for multi-fd, but not for multi-thread?
> 

When you set 'none'in multi-fd compression, you would not use the
compression capability in multi-fd.

In comparison, once you turn on multi-thread compression capability, you
have already admitted to use compression. In this case, member 'none' is
meaningless.

> If the difference is wanted: the names are inconsistent.  Less of an
> issue than member names, because type names are not externally visible.
> Let's make them consistent anyway.
> 
> Hmm, there's also Qcow2CompressionType.  That's another conversation;
> I'll start a new thread for it.
> 
> [...]
> 
> .
> 

Re: [PATCH V17 5/6] hw/mips: Add Loongson-3 machine support

[Huacai Chen]

Hi, Philippe,

On Tue, Nov 24, 2020 at 5:54 AM Philippe Mathieu-Daudé  wrote:
>
> Hi Huacai,
>
> On 11/6/20 5:21 AM, Huacai Chen wrote:
> > Add Loongson-3 based machine support, it use liointc as the interrupt
> > controler and use GPEX as the pci controller. Currently it can work with
> > both TCG and KVM.
> >
> > As the machine model is not based on any exiting physical hardware, the
> > name of the machine is "loongson3-virt". It may be superseded in future
> > by a real machine model. If this happens, then a regular deprecation
> > procedure shall occur for "loongson3-virt" machine.
> >
> > We now already have a full functional Linux kernel (based on Linux-5.4.x
> > LTS) here:
> >
> > https://github.com/chenhuacai/linux
> >
> > Of course the upstream kernel is also usable (the kvm host side and
> > guest side have both been upstream in Linux-5.9):
> >
> > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
> >
> > How to use QEMU/Loongson-3?
> > 1, Download kernel source from the above URL;
> > 2, Build a kernel with arch/mips/configs/loongson3_defconfig;
> > 3, Boot a Loongson-3A4000 host with this kernel (for KVM mode);
> > 4, Build QEMU-master with this patchset;
> > 5, modprobe kvm (only necessary for KVM mode);
> > 6, Use QEMU with TCG:
> >qemu-system-mips64el -M loongson3-virt,accel=tcg -cpu 
> > Loongson-3A1000 -kernel  -append ...
> >Use QEMU with KVM:
> >qemu-system-mips64el -M loongson3-virt,accel=kvm -cpu 
> > Loongson-3A4000 -kernel  -append ...
> >
> >The "-cpu" parameter is optional here and QEMU will use the correct type 
> > for TCG/KVM automatically.
> >
> > Signed-off-by: Huacai Chen 
> > Co-developed-by: Jiaxun Yang 
> > Signed-off-by: Jiaxun Yang 
> > ---
> >  default-configs/devices/mips64el-softmmu.mak |   1 +
> >  hw/mips/Kconfig  |  12 +
> >  hw/mips/loongson3_virt.c | 614 
> > +++
> >  hw/mips/meson.build  |   2 +-
> >  4 files changed, 628 insertions(+), 1 deletion(-)
> >  create mode 100644 hw/mips/loongson3_virt.c
> >
> > diff --git a/default-configs/devices/mips64el-softmmu.mak 
> > b/default-configs/devices/mips64el-softmmu.mak
> > index 9f8a3ef..26c660a 100644
> > --- a/default-configs/devices/mips64el-softmmu.mak
> > +++ b/default-configs/devices/mips64el-softmmu.mak
> > @@ -3,6 +3,7 @@
> >  include mips-softmmu-common.mak
> >  CONFIG_IDE_VIA=y
> >  CONFIG_FULOONG=y
> > +CONFIG_LOONGSON3V=y
> >  CONFIG_ATI_VGA=y
> >  CONFIG_RTL8139_PCI=y
> >  CONFIG_JAZZ=y
> > diff --git a/hw/mips/Kconfig b/hw/mips/Kconfig
> > index 8be7012..ef5cee1 100644
> > --- a/hw/mips/Kconfig
> > +++ b/hw/mips/Kconfig
> > @@ -32,6 +32,18 @@ config FULOONG
> >  bool
> >  select PCI_BONITO
> >
> > +config LOONGSON3V
> > +bool
> > +select PCKBD
>
> Is it used? I only see USB.
OK, PCKBD will be removed.

>
> > +select SERIAL
> > +select GOLDFISH_RTC
> > +select LOONGSON_LIOINTC
> > +select PCI_DEVICES
> > +select PCI_EXPRESS_GENERIC_BRIDGE
> > +select VIRTIO_VGA
> > +select QXL if SPICE
>
> I don't understand the UI dependencies, as we should
> be able to start this machine without UI (just console
> for example).
>
> Maybe you want the 'imply' keyword instead?
OK, I will use imply instead.

>
> > +select MSI_NONBROKEN
> > +
> >  config MIPS_CPS
> >  bool
> >  select PTIMER
> > diff --git a/hw/mips/loongson3_virt.c b/hw/mips/loongson3_virt.c
> > new file mode 100644
> > index 000..c5db2db
> > --- /dev/null
> > +++ b/hw/mips/loongson3_virt.c
> > @@ -0,0 +1,614 @@
> > +/*
> > + * Generic Loongson-3 Platform support
> > + *
> > + * Copyright (c) 2018-2020 Huacai Chen (che...@lemote.com)
> > + * Copyright (c) 2018-2020 Jiaxun Yang 
> > + *
> > + * This program is free software: you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License as published by
> > + * the Free Software Foundation, either version 2 of the License, or
> > + * (at your option) any later version.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > + * GNU General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU General Public License
> > + * along with this program. If not, see .
> > + */
> > +
> > +/*
> > + * Generic virtualized PC Platform based on Loongson-3 CPU (MIPS64R2 with
> > + * extensions, 800~2000MHz)
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include "qemu-common.h"
> > +#include "qemu/units.h"
> > +#include "qemu/cutils.h"
> > +#include "qapi/error.h"
> > +#include "cpu.h"
> > +#include "elf.h"
> > +#include "kvm_mips.h"
> > +#include "hw/boards.h"
> > +#include "hw/char/serial.h"
> > +#include "hw/intc/loongson_liointc.h"
> > +#include 

回复

[Zeyu Jin]

On 2020/11/27 17:48, Markus Armbruster wrote:
> Kevin, Max, suggest to skip right to Qcow2CompressionType.
> 
> Zeyu Jin  writes:
> 
>> A multi-thread compress method parameter is added to hold the method we
>> are going to use. By default the 'zlib' method is used to maintain the
>> compatibility as before.
>>
>> Signed-off-by: Zeyu Jin 
>> Signed-off-by: Ying Fang 
> [...]
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 3c75820527..2ed6a55b92 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -525,6 +525,19 @@
>>'data': [ 'none', 'zlib',
>>  { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>>  
>> +##
>> +# @CompressMethod:
>> +#
>> +# An enumeration of multi-thread compression methods.
>> +#
>> +# @zlib: use zlib compression method.
>> +#
>> +# Since: 6.0
>> +#
>> +##
>> +{ 'enum': 'CompressMethod',
>> +  'data': [ 'zlib' ] }
>> +
>>  ##
>>  # @BitmapMigrationBitmapAlias:
>>  #
>> @@ -599,6 +612,9 @@
>>  #  compression, so set the decompress-threads to the 
>> number about 1/4
>>  #  of compress-threads is adequate.
>>  #
>> +# @compress-method: Set compression method to use in multi-thread 
>> compression.
>> +#   Defaults to zlib. (Since 6.0)
> 
> We already have @multifd-compression.  Why do we need to control the two
> separately?  Can you give a use case for different settings?

Generally, mulit-thread compression deals with the situation
where network bandwith is limited but cpu resource is adequate. Multifd
instead aims to situation where single fd cannot take full advantage of
network bandwith. So compression based on multifd cannot fully cover the
cases for multi-thread compression.

For example, for migration with a bandwith limitation of 10M
bytes/second, single fd is enough for data delivery. This is the case
for multi-thread compression.

> 
> If we do want two parameters: the names @compress-method and
> @multifd-compression are inconsistent.  According to your comment,
> @compress-method applies only to multi-thread compression.  That leads
> me to suggest renaming it to @multi-thread-compression.
> 

For the names, my original idea is to make 'CompressMethod' consistent
with other multi-thread compression parameters, like 'compress-threads'
and 'compress-level'. There is truly some inconsistency here, between
multifd-compression params and old multi-thread compression params.

For now, I agree with you that 'multi-thread-compression' is better. It
is more specific and clear. I will rename the params in next version.
Thanks for the suggestion.

> After PATCH 4, CompressMethod is almost the same as MultiFDCompression:
> 
>{ 'enum': 'CompressMethod',
>  'data': [ 'zlib' ] }
>  'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
> 
>{ 'enum': 'MultiFDCompression',
>  'data': [ 'none', 'zlib',
>{ 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
> 
> The difference is member 'none'.  Why does compression 'none' make sense
> for multi-fd, but not for multi-thread?
> 

When you set 'none'in multi-fd compression, you would not use the
compression capability in multi-fd.

In comparison, once you turn on multi-thread compression capability, you
have already admitted to use compression. In this case, member 'none' is
meaningless.

> If the difference is wanted: the names are inconsistent.  Less of an
> issue than member names, because type names are not externally visible.
> Let's make them consistent anyway.
> 
> Hmm, there's also Qcow2CompressionType.  That's another conversation;
> I'll start a new thread for it.
> 
> [...]
>> .

Re: [PATCH V17 2/6] hw/intc: Rework Loongson LIOINTC

[Huacai Chen]

Hi, Philippe,

On Tue, Nov 24, 2020 at 6:24 AM Philippe Mathieu-Daudé  wrote:
>
> On 11/23/20 9:52 PM, Philippe Mathieu-Daudé wrote:
> > On 11/6/20 5:21 AM, Huacai Chen wrote:
> >> As suggested by Philippe Mathieu-Daudé, rework Loongson's liointc:
> >> 1, Move macro definitions to loongson_liointc.h;
> >> 2, Remove magic values and use macros instead;
> >> 3, Replace dead D() code by trace events.
> >>
> >> Suggested-by: Philippe Mathieu-Daudé 
> >> Signed-off-by: Huacai Chen 
> >> ---
> >>  hw/intc/loongson_liointc.c | 49 
> >> +++---
> >>  include/hw/intc/loongson_liointc.h | 39 ++
> >>  2 files changed, 53 insertions(+), 35 deletions(-)
> >>  create mode 100644 include/hw/intc/loongson_liointc.h
> >>
> >> diff --git a/hw/intc/loongson_liointc.c b/hw/intc/loongson_liointc.c
> >> index fbbfb57..be29e2f 100644
> >> --- a/hw/intc/loongson_liointc.c
> >> +++ b/hw/intc/loongson_liointc.c
> >> @@ -1,6 +1,7 @@
> >>  /*
> >>   * QEMU Loongson Local I/O interrupt controler.
> >>   *
> >> + * Copyright (c) 2020 Huacai Chen 
> >>   * Copyright (c) 2020 Jiaxun Yang 
> >>   *
> >>   * This program is free software: you can redistribute it and/or modify
> >> @@ -19,33 +20,11 @@
> >>   */
> >>
> >>  #include "qemu/osdep.h"
> >> -#include "hw/sysbus.h"
> >>  #include "qemu/module.h"
> >> +#include "qemu/log.h"
> >>  #include "hw/irq.h"
> >>  #include "hw/qdev-properties.h"
> >> -#include "qom/object.h"
> >> -
> >> -#define D(x)
> >> -
> >> -#define NUM_IRQS32
> >> -
> >> -#define NUM_CORES   4
> >> -#define NUM_IPS 4
> >> -#define NUM_PARENTS (NUM_CORES * NUM_IPS)
> >> -#define PARENT_COREx_IPy(x, y)  (NUM_IPS * x + y)
> >> -
> >> -#define R_MAPPER_START  0x0
> >> -#define R_MAPPER_END0x20
> >> -#define R_ISR   R_MAPPER_END
> >> -#define R_IEN   0x24
> >> -#define R_IEN_SET   0x28
> >> -#define R_IEN_CLR   0x2c
> >> -#define R_PERCORE_ISR(x)(0x40 + 0x8 * x)
> >> -#define R_END   0x64
> >> -
> >> -#define TYPE_LOONGSON_LIOINTC "loongson.liointc"
> >> -DECLARE_INSTANCE_CHECKER(struct loongson_liointc, LOONGSON_LIOINTC,
> >> - TYPE_LOONGSON_LIOINTC)
> >> +#include "hw/intc/loongson_liointc.h"
> >>
> >>  struct loongson_liointc {
> >>  SysBusDevice parent_obj;
> >> @@ -123,14 +102,13 @@ liointc_read(void *opaque, hwaddr addr, unsigned int 
> >> size)
> >>  goto out;
> >>  }
> >>
> >> -/* Rest is 4 byte */
> >> +/* Rest are 4 bytes */
> >>  if (size != 4 || (addr % 4)) {
> >>  goto out;
> >>  }
> >>
> >> -if (addr >= R_PERCORE_ISR(0) &&
> >> -addr < R_PERCORE_ISR(NUM_CORES)) {
> >> -int core = (addr - R_PERCORE_ISR(0)) / 8;
> >> +if (addr >= R_START && addr < R_END) {
> >> +int core = (addr - R_START) / R_ISR_SIZE;
> >>  r = p->per_core_isr[core];
> >>  goto out;
> >>  }
> >> @@ -147,7 +125,8 @@ liointc_read(void *opaque, hwaddr addr, unsigned int 
> >> size)
> >>  }
> >>
> >>  out:
> >> -D(qemu_log("%s: size=%d addr=%lx val=%x\n", __func__, size, addr, r));
> >> +qemu_log_mask(CPU_LOG_INT, "%s: size=%d addr=%lx val=%x\n",
> >> +  __func__, size, addr, r);
> >>  return r;
> >>  }
> >>
> >> @@ -158,7 +137,8 @@ liointc_write(void *opaque, hwaddr addr,
> >>  struct loongson_liointc *p = opaque;
> >>  uint32_t value = val64;
> >>
> >> -D(qemu_log("%s: size=%d, addr=%lx val=%x\n", __func__, size, addr, 
> >> value));
> >> +qemu_log_mask(CPU_LOG_INT, "%s: size=%d, addr=%lx val=%x\n",
> >> +  __func__, size, addr, value);
> >>
> >>  /* Mapper is 1 byte */
> >>  if (size == 1 && addr < R_MAPPER_END) {
> >> @@ -166,14 +146,13 @@ liointc_write(void *opaque, hwaddr addr,
> >>  goto out;
> >>  }
> >>
> >> -/* Rest is 4 byte */
> >> +/* Rest are 4 bytes */
> >>  if (size != 4 || (addr % 4)) {
> >>  goto out;
> >>  }
> >>
> >> -if (addr >= R_PERCORE_ISR(0) &&
> >> -addr < R_PERCORE_ISR(NUM_CORES)) {
> >> -int core = (addr - R_PERCORE_ISR(0)) / 8;
> >> +if (addr >= R_START && addr < R_END) {
> >> +int core = (addr - R_START) / R_ISR_SIZE;
> >>  p->per_core_isr[core] = value;
> >>  goto out;
> >>  }
> >> @@ -224,7 +203,7 @@ static void loongson_liointc_init(Object *obj)
> >>  }
> >>
> >>  memory_region_init_io(>mmio, obj, _ops, p,
> >> - "loongson.liointc", R_END);
> >> + TYPE_LOONGSON_LIOINTC, R_END);
> >>  sysbus_init_mmio(SYS_BUS_DEVICE(obj), >mmio);
> >>  }
> >>
> >> diff --git a/include/hw/intc/loongson_liointc.h 
> >> b/include/hw/intc/loongson_liointc.h
> >> new file mode 100644
> >> index 000..e11f482
> >> --- /dev/null
> >> +++ b/include/hw/intc/loongson_liointc.h
> >> @@ -0,0 

Re: [PATCH V17 2/6] hw/intc: Rework Loongson LIOINTC

[Huacai Chen]

Hi, Philippe,

On Tue, Nov 24, 2020 at 4:52 AM Philippe Mathieu-Daudé  wrote:
>
> On 11/6/20 5:21 AM, Huacai Chen wrote:
> > As suggested by Philippe Mathieu-Daudé, rework Loongson's liointc:
> > 1, Move macro definitions to loongson_liointc.h;
> > 2, Remove magic values and use macros instead;
> > 3, Replace dead D() code by trace events.
> >
> > Suggested-by: Philippe Mathieu-Daudé 
> > Signed-off-by: Huacai Chen 
> > ---
> >  hw/intc/loongson_liointc.c | 49 
> > +++---
> >  include/hw/intc/loongson_liointc.h | 39 ++
> >  2 files changed, 53 insertions(+), 35 deletions(-)
> >  create mode 100644 include/hw/intc/loongson_liointc.h
> >
> > diff --git a/hw/intc/loongson_liointc.c b/hw/intc/loongson_liointc.c
> > index fbbfb57..be29e2f 100644
> > --- a/hw/intc/loongson_liointc.c
> > +++ b/hw/intc/loongson_liointc.c
> > @@ -1,6 +1,7 @@
> >  /*
> >   * QEMU Loongson Local I/O interrupt controler.
> >   *
> > + * Copyright (c) 2020 Huacai Chen 
> >   * Copyright (c) 2020 Jiaxun Yang 
> >   *
> >   * This program is free software: you can redistribute it and/or modify
> > @@ -19,33 +20,11 @@
> >   */
> >
> >  #include "qemu/osdep.h"
> > -#include "hw/sysbus.h"
> >  #include "qemu/module.h"
> > +#include "qemu/log.h"
> >  #include "hw/irq.h"
> >  #include "hw/qdev-properties.h"
> > -#include "qom/object.h"
> > -
> > -#define D(x)
> > -
> > -#define NUM_IRQS32
> > -
> > -#define NUM_CORES   4
> > -#define NUM_IPS 4
> > -#define NUM_PARENTS (NUM_CORES * NUM_IPS)
> > -#define PARENT_COREx_IPy(x, y)  (NUM_IPS * x + y)
> > -
> > -#define R_MAPPER_START  0x0
> > -#define R_MAPPER_END0x20
> > -#define R_ISR   R_MAPPER_END
> > -#define R_IEN   0x24
> > -#define R_IEN_SET   0x28
> > -#define R_IEN_CLR   0x2c
> > -#define R_PERCORE_ISR(x)(0x40 + 0x8 * x)
> > -#define R_END   0x64
> > -
> > -#define TYPE_LOONGSON_LIOINTC "loongson.liointc"
> > -DECLARE_INSTANCE_CHECKER(struct loongson_liointc, LOONGSON_LIOINTC,
> > - TYPE_LOONGSON_LIOINTC)
> > +#include "hw/intc/loongson_liointc.h"
> >
> >  struct loongson_liointc {
> >  SysBusDevice parent_obj;
> > @@ -123,14 +102,13 @@ liointc_read(void *opaque, hwaddr addr, unsigned int 
> > size)
> >  goto out;
> >  }
> >
> > -/* Rest is 4 byte */
> > +/* Rest are 4 bytes */
> >  if (size != 4 || (addr % 4)) {
> >  goto out;
> >  }
> >
> > -if (addr >= R_PERCORE_ISR(0) &&
> > -addr < R_PERCORE_ISR(NUM_CORES)) {
> > -int core = (addr - R_PERCORE_ISR(0)) / 8;
> > +if (addr >= R_START && addr < R_END) {
> > +int core = (addr - R_START) / R_ISR_SIZE;
> >  r = p->per_core_isr[core];
> >  goto out;
> >  }
> > @@ -147,7 +125,8 @@ liointc_read(void *opaque, hwaddr addr, unsigned int 
> > size)
> >  }
> >
> >  out:
> > -D(qemu_log("%s: size=%d addr=%lx val=%x\n", __func__, size, addr, r));
> > +qemu_log_mask(CPU_LOG_INT, "%s: size=%d addr=%lx val=%x\n",
> > +  __func__, size, addr, r);
> >  return r;
> >  }
> >
> > @@ -158,7 +137,8 @@ liointc_write(void *opaque, hwaddr addr,
> >  struct loongson_liointc *p = opaque;
> >  uint32_t value = val64;
> >
> > -D(qemu_log("%s: size=%d, addr=%lx val=%x\n", __func__, size, addr, 
> > value));
> > +qemu_log_mask(CPU_LOG_INT, "%s: size=%d, addr=%lx val=%x\n",
> > +  __func__, size, addr, value);
> >
> >  /* Mapper is 1 byte */
> >  if (size == 1 && addr < R_MAPPER_END) {
> > @@ -166,14 +146,13 @@ liointc_write(void *opaque, hwaddr addr,
> >  goto out;
> >  }
> >
> > -/* Rest is 4 byte */
> > +/* Rest are 4 bytes */
> >  if (size != 4 || (addr % 4)) {
> >  goto out;
> >  }
> >
> > -if (addr >= R_PERCORE_ISR(0) &&
> > -addr < R_PERCORE_ISR(NUM_CORES)) {
> > -int core = (addr - R_PERCORE_ISR(0)) / 8;
> > +if (addr >= R_START && addr < R_END) {
> > +int core = (addr - R_START) / R_ISR_SIZE;
> >  p->per_core_isr[core] = value;
> >  goto out;
> >  }
> > @@ -224,7 +203,7 @@ static void loongson_liointc_init(Object *obj)
> >  }
> >
> >  memory_region_init_io(>mmio, obj, _ops, p,
> > - "loongson.liointc", R_END);
> > + TYPE_LOONGSON_LIOINTC, R_END);
> >  sysbus_init_mmio(SYS_BUS_DEVICE(obj), >mmio);
> >  }
> >
> > diff --git a/include/hw/intc/loongson_liointc.h 
> > b/include/hw/intc/loongson_liointc.h
> > new file mode 100644
> > index 000..e11f482
> > --- /dev/null
> > +++ b/include/hw/intc/loongson_liointc.h
> > @@ -0,0 +1,39 @@
> > +/*
> > + * This file is subject to the terms and conditions of the GNU General 
> > Public
> > + * License.  See the file "COPYING" in the main directory of this archive
> > + * 

RE: [PATCH 1/2] target/nios2: Move cpu_pic code into CPU object proper

[Wu, Wentong]

On 11/28/20 3:13 AM, Peter Maydell wrote:
> The nios2 code uses an old style of interrupt handling, where a separate 
> standalone set of qemu_irqs invoke a function
> nios2_pic_cpu_handler() which signals the interrupt to the CPU proper by 
> directly calling cpu_interrupt() and cpu_reset_interrupt().
> Because CPU objects now inherit (indirectly) from TYPE_DEVICE, they can have 
> GPIO input lines themselves, and the neater modern way to implement this is 
> to simply have the CPU object itself provide the input IRQ lines.
> 
> Create named "NMI" and "IRQ" GPIO inputs to the Nios2 CPU object, and make 
> the only user of nios2_cpu_pic_init() wire up directly to those instead.
>
> This fixes a Coverity-reported trivial memory leak of the IRQ array allocated 
> in nios2_cpu_pic_init().
>
> Fixes: Coverity CID 1421916
> Signed-off-by: Peter Maydell 
> ---
> target/nios2/cpu.h|  1 -
> hw/nios2/10m50_devboard.c |  8 +++-
> hw/nios2/cpu_pic.c| 31 ---
> target/nios2/cpu.c| 34 ++
> 4 files changed, 37 insertions(+), 37 deletions(-)
>
> diff --git a/target/nios2/cpu.h b/target/nios2/cpu.h index 
> 86bbe1d8670..b7efb54ba7e 100644
> --- a/target/nios2/cpu.h
> +++ b/target/nios2/cpu.h
> @@ -201,7 +201,6 @@ void nios2_cpu_do_unaligned_access(CPUState *cpu, vaddr 
> addr,
>MMUAccessType access_type,
>   int mmu_idx, uintptr_t retaddr);
> 
> -qemu_irq *nios2_cpu_pic_init(Nios2CPU *cpu);  void 
> nios2_check_interrupts(CPUNios2State *env);
> 
> void do_nios2_semihosting(CPUNios2State *env); diff --git 
> a/hw/nios2/10m50_devboard.c b/hw/nios2/10m50_devboard.c index 
> 5c13b74306f..ac1993e8c08 100644
> --- a/hw/nios2/10m50_devboard.c
> +++ b/hw/nios2/10m50_devboard.c
> @@ -52,7 +52,7 @@ static void nios2_10m50_ghrd_init(MachineState *machine)
> ram_addr_t tcm_size = 0x1000;/* 1 kiB, but QEMU limit is 4 kiB */
> ram_addr_t ram_base = 0x0800;
> ram_addr_t ram_size = 0x0800;
>  -qemu_irq *cpu_irq, irq[32];
>  +qemu_irq irq[32];
>int i;
> 
> /* Physical TCM (tb_ram_1k) with alias at 0xc000 */ @@ -76,14 +76,12 
> @@ static void nios2_10m50_ghrd_init(MachineState *machine)
> /* Create CPU -- FIXME */
> cpu = NIOS2_CPU(cpu_create(TYPE_NIOS2_CPU));
> 
> -/* Register: CPU interrupt controller (PIC) */
> -cpu_irq = nios2_cpu_pic_init(cpu);
> -
> /* Register: Internal Interrupt Controller (IIC) */
>dev = qdev_new("altera,iic");
> object_property_add_const_link(OBJECT(dev), "cpu", OBJECT(cpu));
> sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), _fatal);
> -sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0, cpu_irq[0]);
> +sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0,
> +   qdev_get_gpio_in_named(DEVICE(cpu), "IRQ", 0));
> for (i = 0; i < 32; i++) {
>irq[i] = qdev_get_gpio_in(dev, i);
> }
> diff --git a/hw/nios2/cpu_pic.c b/hw/nios2/cpu_pic.c index 
> 5ea7e52ab83..3fb621c5c85 100644
> --- a/hw/nios2/cpu_pic.c
> +++ b/hw/nios2/cpu_pic.c
> @@ -26,32 +26,6 @@
> 
> #include "boot.h"
> 
> -static void nios2_pic_cpu_handler(void *opaque, int irq, int level) -{
> -Nios2CPU *cpu = opaque;
> -CPUNios2State *env = >env;
> -CPUState *cs = CPU(cpu);
> -int type = irq ? CPU_INTERRUPT_NMI : CPU_INTERRUPT_HARD;
> -
> -if (type == CPU_INTERRUPT_HARD) {
> -env->irq_pending = level;
> -
> -if (level && (env->regs[CR_STATUS] & CR_STATUS_PIE)) {
> -env->irq_pending = 0;
> -cpu_interrupt(cs, type);
> -} else if (!level) {
> -env->irq_pending = 0;
> -cpu_reset_interrupt(cs, type);
> -}
> -} else {
> -if (level) {
> -cpu_interrupt(cs, type);
> -} else {
> -cpu_reset_interrupt(cs, type);
> -}
> -}
> -}
> -
> void nios2_check_interrupts(CPUNios2State *env)  {
> if (env->irq_pending &&
> @@ -60,8 +34,3 @@ void nios2_check_interrupts(CPUNios2State *env)
>cpu_interrupt(env_cpu(env), CPU_INTERRUPT_HARD);
> }
> }
> -
> -qemu_irq *nios2_cpu_pic_init(Nios2CPU *cpu) -{
> -return qemu_allocate_irqs(nios2_pic_cpu_handler, cpu, 2);
> -}
> diff --git a/target/nios2/cpu.c b/target/nios2/cpu.c index 
> 8f7011fcb92..4b21e7c6d1c 100644
> --- a/target/nios2/cpu.c
> +++ b/target/nios2/cpu.c
> @@ -64,6 +64,37 @@ static void nios2_cpu_reset(DeviceState *dev)  #endif  }
> 
> +#ifndef CONFIG_USER_ONLY
> +static void nios2_cpu_set_nmi(void *opaque, int irq, int level) {
> +Nios2CPU *cpu = opaque;
> +CPUState *cs = CPU(cpu);
> +
> +if (level) {
> +cpu_interrupt(cs, CPU_INTERRUPT_NMI);
> +} else {
> +cpu_reset_interrupt(cs, CPU_INTERRUPT_NMI);
> +}
> +}
> +
> +static void nios2_cpu_set_irq(void *opaque, int irq, int level) {
> +Nios2CPU *cpu = opaque;
> +CPUNios2State *env = >env;
> +CPUState *cs = 

Re: [RFC] ich9:cpuhp: add support for cpu hot-unplug with SMI broadcast enabled

[Ankur Arora]

On 2020-11-27 7:19 a.m., Laszlo Ersek wrote:

On 11/27/20 05:10, Ankur Arora wrote:


Yeah I was wondering what would happen for simultaneous hot add and remove.
I guess we would always do remove first and then the add, unless we hit
the break due to max_cpus_per_pass and switch to hot-add mode.


Considering the firmware only, I disagree with remove-then-add.

EFI_SMM_CPU_SERVICE_PROTOCOL.AddProcessor() and
EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor() (implemented in
SmmAddProcessor() and SmmRemoveProcessor() in
"UefiCpuPkg/PiSmmCpuDxeSmm/CpuService.c", respectively) only mark the
processors for addition/removal. The actual processing is done only
later, in BSPHandler() --> SmmCpuUpdate(), when "all SMI handlers are
finished" (see the comment in SmmRemoveProcessor()).

Consequently, I would not suggest replacing a valid APIC ID in a
particular mCpuHotPlugData.ApicId[Index] slot with INVALID_APIC_ID
(corresponding to the unplug operation), and then possibly replacing
INVALID_APIC_ID in the *same slot* with the APIC ID of the newly plugged
CPU, in the exact same SMI invocation (= in the same execution of
CpuHotplugMmi()). That might cause some component in edk2 to see the
APIC ID in mCpuHotPlugData.ApicId[Index] to change from one valid ACPI
ID to another valid APIC ID, and I don't even want to think about what
kind of mess that could cause.


Shudders.



So no, please handle plugs first, for which unused slots in
mCpuHotPlugData.ApicId will be populated, and *then* handle removals (in
the same invocation of CpuHotplugMmi()).


Yeah, that ordering makes complete sense.



By the way, for unplug, you will not have to re-set
mCpuHotPlugData.ApicId[Index] to INVALID_APIC_ID, as
SmmRemoveProcessor() does that internally. You just have to locate the
Index for the APIC ID being removed, for calling
EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().


Right. The hotplug is more involved (given the need to pen the new CPU)
but for the unplug, AFAICS all the actual handling for removal is in
.RemoveProcessor() and at SMI exit in SmmCpuUpdate().


Thanks
Ankur




Thanks
Laszlo

Re: [RFC] ich9:cpuhp: add support for cpu hot-unplug with SMI broadcast enabled

[Ankur Arora]

On 2020-11-27 3:47 a.m., Igor Mammedov wrote:

On Thu, 26 Nov 2020 20:10:59 -0800
Ankur Arora  wrote:


On 2020-11-26 12:38 p.m., Igor Mammedov wrote:

On Thu, 26 Nov 2020 12:17:27 +0100
Laszlo Ersek  wrote:
   

On 11/24/20 13:25, Igor Mammedov wrote:

If firmware negotiates ICH9_LPC_SMI_F_CPU_HOT_UNPLUG_BIT feature,
OSPM on CPU eject will set bit #4 in CPU hotplug block for to be
ejected CPU to mark it for removal by firmware and trigger SMI
upcall to let firmware do actual eject.

Signed-off-by: Igor Mammedov 
---
PS:
- abuse 5.1 machine type for now to turn off unplug feature
  (it will be moved to 5.2 machine type once new merge window is open)
---
   include/hw/acpi/cpu.h   |  2 ++
   docs/specs/acpi_cpu_hotplug.txt | 11 +--
   hw/acpi/cpu.c   | 18 --
   hw/i386/acpi-build.c|  5 +
   hw/i386/pc.c|  1 +
   hw/isa/lpc_ich9.c   |  2 +-
   6 files changed, 34 insertions(+), 5 deletions(-)

diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
index 0eeedaa491..999caaf510 100644
--- a/include/hw/acpi/cpu.h
+++ b/include/hw/acpi/cpu.h
@@ -22,6 +22,7 @@ typedef struct AcpiCpuStatus {
   uint64_t arch_id;
   bool is_inserting;
   bool is_removing;
+bool fw_remove;
   uint32_t ost_event;
   uint32_t ost_status;
   } AcpiCpuStatus;
@@ -50,6 +51,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object *owner,
   typedef struct CPUHotplugFeatures {
   bool acpi_1_compatible;
   bool has_legacy_cphp;
+bool fw_unplugs_cpu;
   const char *smi_path;
   } CPUHotplugFeatures;

diff --git a/docs/specs/acpi_cpu_hotplug.txt b/docs/specs/acpi_cpu_hotplug.txt
index 9bb22d1270..f68ef6e06c 100644
--- a/docs/specs/acpi_cpu_hotplug.txt
+++ b/docs/specs/acpi_cpu_hotplug.txt
@@ -57,7 +57,11 @@ read access:
 It's valid only when bit 0 is set.
  2: Device remove event, used to distinguish device for which
 no device eject request to OSPM was issued.
-   3-7: reserved and should be ignored by OSPM
+   3: reserved and should be ignored by OSPM
+   4: if set to 1, OSPM requests firmware to perform device eject,
+  firmware shall clear this event by writing 1 into it before


(1) s/clear this event/clear this event bit/
  

+  performing device eject.


(2) move the second and third lines ("firmware shall clear") over to
the write documentation, below? In particular:
  

+   5-7: reserved and should be ignored by OSPM
   [0x5-0x7] reserved
   [0x8] Command data: (DWORD access)
 contains 0 unless value last stored in 'Command field' is one of:
@@ -82,7 +86,10 @@ write access:
  selected CPU device
   3: if set to 1 initiates device eject, set by OSPM when it
  triggers CPU device removal and calls _EJ0 method
-4-7: reserved, OSPM must clear them before writing to register
+4: if set to 1 OSPM hands over device eject to firmware,
+   Firmware shall issue device eject request as described above
+   (bit #3) and OSPM should not touch device eject bit (#3),


(3) it would be clearer if we documented the exact bit writing order
here:
- clear bit#4, *then* set bit#3 (two write accesses)
- versus clear bit#4 *and* set bit#3 (single access)


I was thinking that FW should not bother with clearing bit #4,
and QEMU should clear it when handling write to bit #3.
(it looks like I forgot to actually do that)


Why involve the firmware with bit #3 at all? If the firmware only reads bit #4
to detect fw_remove and then write (and thus reset) bit #4, isn't that
good enough?


That would needlessly complicate code on QEMU side and I don't want to
overload bit #4 with another semantics, and we already have bit #3 that
does eject. So unless there are issues with that, I'd stick to using
bit #3 for eject.


Sounds good.






   



  

+5-7: reserved, OSPM must clear them before writing to register
   [0x5] Command field: (1 byte access)
 value:
   0: selects a CPU device with inserting/removing events and
diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index f099b50927..09d2f20dae 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -71,6 +71,7 @@ static uint64_t cpu_hotplug_rd(void *opaque, hwaddr addr, 
unsigned size)
   val |= cdev->cpu ? 1 : 0;
   val |= cdev->is_inserting ? 2 : 0;
   val |= cdev->is_removing  ? 4 : 0;
+val |= cdev->fw_remove  ? 16 : 0;
   trace_cpuhp_acpi_read_flags(cpu_st->selector, val);
   break;
   case ACPI_CPU_CMD_DATA_OFFSET_RW:
@@ -148,6 +149,8 @@ static void cpu_hotplug_wr(void *opaque, hwaddr addr, 
uint64_t data,
   hotplug_ctrl = qdev_get_hotplug_handler(dev);
   hotplug_handler_unplug(hotplug_ctrl, dev, NULL);
   object_unparent(OBJECT(dev));
+ 

Re: [RFC] ich9:cpuhp: add support for cpu hot-unplug with SMI broadcast enabled

[Ankur Arora]

On 2020-11-27 7:02 a.m., Laszlo Ersek wrote:

On 11/27/20 12:33, Igor Mammedov wrote:

On Thu, 26 Nov 2020 19:35:30 -0800
Ankur Arora  wrote:


On 2020-11-26 4:46 a.m., Laszlo Ersek wrote:

On 11/26/20 11:24, Ankur Arora wrote:

On 2020-11-24 4:25 a.m., Igor Mammedov wrote:

If firmware negotiates ICH9_LPC_SMI_F_CPU_HOT_UNPLUG_BIT feature,
OSPM on CPU eject will set bit #4 in CPU hotplug block for to be
ejected CPU to mark it for removal by firmware and trigger SMI
upcall to let firmware do actual eject.

Signed-off-by: Igor Mammedov 
---
PS:
     - abuse 5.1 machine type for now to turn off unplug feature
   (it will be moved to 5.2 machine type once new merge window is open)
---
    include/hw/acpi/cpu.h   |  2 ++
    docs/specs/acpi_cpu_hotplug.txt | 11 +--
    hw/acpi/cpu.c   | 18 --
    hw/i386/acpi-build.c    |  5 +
    hw/i386/pc.c    |  1 +
    hw/isa/lpc_ich9.c   |  2 +-
    6 files changed, 34 insertions(+), 5 deletions(-)

diff --git a/include/hw/acpi/cpu.h b/include/hw/acpi/cpu.h
index 0eeedaa491..999caaf510 100644
--- a/include/hw/acpi/cpu.h
+++ b/include/hw/acpi/cpu.h
@@ -22,6 +22,7 @@ typedef struct AcpiCpuStatus {
    uint64_t arch_id;
    bool is_inserting;
    bool is_removing;
+    bool fw_remove;
    uint32_t ost_event;
    uint32_t ost_status;
    } AcpiCpuStatus;
@@ -50,6 +51,7 @@ void cpu_hotplug_hw_init(MemoryRegion *as, Object
*owner,
    typedef struct CPUHotplugFeatures {
    bool acpi_1_compatible;
    bool has_legacy_cphp;
+    bool fw_unplugs_cpu;
    const char *smi_path;
    } CPUHotplugFeatures;
    diff --git a/docs/specs/acpi_cpu_hotplug.txt
b/docs/specs/acpi_cpu_hotplug.txt
index 9bb22d1270..f68ef6e06c 100644
--- a/docs/specs/acpi_cpu_hotplug.txt
+++ b/docs/specs/acpi_cpu_hotplug.txt
@@ -57,7 +57,11 @@ read access:
      It's valid only when bit 0 is set.
   2: Device remove event, used to distinguish device for which
      no device eject request to OSPM was issued.
-   3-7: reserved and should be ignored by OSPM
+   3: reserved and should be ignored by OSPM
+   4: if set to 1, OSPM requests firmware to perform device
eject,
+  firmware shall clear this event by writing 1 into it
before
+  performing device eject> +   5-7: reserved and
should be ignored by OSPM
    [0x5-0x7] reserved
    [0x8] Command data: (DWORD access)
      contains 0 unless value last stored in 'Command field' is
one of:
@@ -82,7 +86,10 @@ write access:
   selected CPU device
    3: if set to 1 initiates device eject, set by OSPM when it
   triggers CPU device removal and calls _EJ0 method
-    4-7: reserved, OSPM must clear them before writing to
register
+    4: if set to 1 OSPM hands over device eject to firmware,
+   Firmware shall issue device eject request as described
above
+   (bit #3) and OSPM should not touch device eject bit (#3),
+    5-7: reserved, OSPM must clear them before writing to
register
    [0x5] Command field: (1 byte access)
      value:
    0: selects a CPU device with inserting/removing events and
diff --git a/hw/acpi/cpu.c b/hw/acpi/cpu.c
index f099b50927..09d2f20dae 100644
--- a/hw/acpi/cpu.c
+++ b/hw/acpi/cpu.c
@@ -71,6 +71,7 @@ static uint64_t cpu_hotplug_rd(void *opaque, hwaddr
addr, unsigned size)
    val |= cdev->cpu ? 1 : 0;
    val |= cdev->is_inserting ? 2 : 0;
    val |= cdev->is_removing  ? 4 : 0;
+    val |= cdev->fw_remove  ? 16 : 0;


I might be missing something but I don't see where cdev->fw_remove is being
set.


See just below, in the cpu_hotplug_wr() hunk. When bit#4 is written --
which happens through the ACPI code change --, fw_remove is inverted.

Thanks that makes sense. I was reading the AML building code all wrong.



   

We do set cdev->is_removing in acpi_cpu_unplug_request_cb() so AFAICS
we would always end up setting this bit:

    val |= cdev->is_removing  ? 4 : 0;


Also, if cdev->fw_remove and cdev->is_removing are both true, val would be
(4 | 16). I'm guessing that in that case the AML determines which case gets
handled but it might make sense to set just one of these?


"is_removing" is set directly in response to the device_del QMP command.
That QMP command is asynchronous to the execution of the guest OS.
j
"fw_remove" is set (by virtue of inverting) by ACPI CEJ0, which is
executed by the guest OS's ACPI interpreter, after the guest OS has
de-scheduled all processes from the CPU being removed (= basically after
the OS has willfully forgotten about the CPU).

Therefore, considering the bitmask (is_removing, fw_remove), three
variations make sense:


Just annotating these with the corresponding ACPI code to make sure
I have it straight. Please correct if my 

Re: [PATCH 2/8] hvf: Move common code out

[Frank Yang]

Hi all,

+Peter Collingbourne 

I'm a developer on the Android Emulator, which is in a fork of QEMU.

Peter and I have been working on an HVF Apple Silicon backend with an eye
toward Android guests.

We have gotten things to basically switch to Android userspace already
(logcat/shell and graphics available at least)

Our strategy so far has been to import logic from the KVM implementation
and hook into QEMU's software devices that previously assumed to only work
with TCG, or have KVM-specific paths.

Thanks to Alexander for the tip on the 36-bit address space limitation btw;
our way of addressing this is to still allow highmem but not put pci high
mmio so high.

Also, note we have a sleep/signal based mechanism to deal with WFx, which
might be worth looking into in Alexander's implementation as well:

https://android-review.googlesource.com/c/platform/external/qemu/+/1512551

Patches so far, FYI:

https://android-review.googlesource.com/c/platform/external/qemu/+/1513429/1
https://android-review.googlesource.com/c/platform/external/qemu/+/1512554/3
https://android-review.googlesource.com/c/platform/external/qemu/+/1512553/3
https://android-review.googlesource.com/c/platform/external/qemu/+/1512552/3
https://android-review.googlesource.com/c/platform/external/qemu/+/1512551/3

https://android.googlesource.com/platform/external/qemu/+/c17eb6a3ffd50047e9646aff6640b710cb8ff48a
https://android.googlesource.com/platform/external/qemu/+/74bed16de1afb41b7a7ab8da1d1861226c9db63b
https://android.googlesource.com/platform/external/qemu/+/eccd9e47ab2ccb9003455e3bb721f57f9ebc3c01
https://android.googlesource.com/platform/external/qemu/+/54fe3d67ed4698e85826537a4f49b2b9074b2228
https://android.googlesource.com/platform/external/qemu/+/82ef91a6fede1d1000f36be037ad4d58fbe0d102
https://android.googlesource.com/platform/external/qemu/+/c28147aa7c74d98b858e99623d2fe46e74a379f6

Peter's also noticed that there are extra steps needed for M1's to allow
TCG to work, as it involves JIT:

https://android.googlesource.com/platform/external/qemu/+/740e3fe47f88926c6bda9abb22ee6eae1bc254a9

We'd appreciate any feedback/comments :)

Best,

Frank

On Fri, Nov 27, 2020 at 1:57 PM Alexander Graf  wrote:

>
> On 27.11.20 21:00, Roman Bolshakov wrote:
> > On Thu, Nov 26, 2020 at 10:50:11PM +0100, Alexander Graf wrote:
> >> Until now, Hypervisor.framework has only been available on x86_64
> systems.
> >> With Apple Silicon shipping now, it extends its reach to aarch64. To
> >> prepare for support for multiple architectures, let's move common code
> out
> >> into its own accel directory.
> >>
> >> Signed-off-by: Alexander Graf 
> >> ---
> >>   MAINTAINERS |   9 +-
> >>   accel/hvf/hvf-all.c |  56 +
> >>   accel/hvf/hvf-cpus.c| 468 
> >>   accel/hvf/meson.build   |   7 +
> >>   accel/meson.build   |   1 +
> >>   include/sysemu/hvf_int.h|  69 ++
> >>   target/i386/hvf/hvf-cpus.c  | 131 --
> >>   target/i386/hvf/hvf-cpus.h  |  25 --
> >>   target/i386/hvf/hvf-i386.h  |  48 +---
> >>   target/i386/hvf/hvf.c   | 360 +--
> >>   target/i386/hvf/meson.build |   1 -
> >>   target/i386/hvf/x86hvf.c|  11 +-
> >>   target/i386/hvf/x86hvf.h|   2 -
> >>   13 files changed, 619 insertions(+), 569 deletions(-)
> >>   create mode 100644 accel/hvf/hvf-all.c
> >>   create mode 100644 accel/hvf/hvf-cpus.c
> >>   create mode 100644 accel/hvf/meson.build
> >>   create mode 100644 include/sysemu/hvf_int.h
> >>   delete mode 100644 target/i386/hvf/hvf-cpus.c
> >>   delete mode 100644 target/i386/hvf/hvf-cpus.h
> >>
> >> diff --git a/MAINTAINERS b/MAINTAINERS
> >> index 68bc160f41..ca4b6d9279 100644
> >> --- a/MAINTAINERS
> >> +++ b/MAINTAINERS
> >> @@ -444,9 +444,16 @@ M: Cameron Esfahani 
> >>   M: Roman Bolshakov 
> >>   W: https://wiki.qemu.org/Features/HVF
> >>   S: Maintained
> >> -F: accel/stubs/hvf-stub.c
> > There was a patch for that in the RFC series from Claudio.
>
>
> Yeah, I'm not worried about this hunk :).
>
>
> >
> >>   F: target/i386/hvf/
> >> +
> >> +HVF
> >> +M: Cameron Esfahani 
> >> +M: Roman Bolshakov 
> >> +W: https://wiki.qemu.org/Features/HVF
> >> +S: Maintained
> >> +F: accel/hvf/
> >>   F: include/sysemu/hvf.h
> >> +F: include/sysemu/hvf_int.h
> >>
> >>   WHPX CPUs
> >>   M: Sunil Muthuswamy 
> >> diff --git a/accel/hvf/hvf-all.c b/accel/hvf/hvf-all.c
> >> new file mode 100644
> >> index 00..47d77a472a
> >> --- /dev/null
> >> +++ b/accel/hvf/hvf-all.c
> >> @@ -0,0 +1,56 @@
> >> +/*
> >> + * QEMU Hypervisor.framework support
> >> + *
> >> + * This work is licensed under the terms of the GNU GPL, version 2.
> See
> >> + * the COPYING file in the top-level directory.
> >> + *
> >> + * Contributions after 2012-01-13 are licensed under the terms of the
> >> + * GNU GPL, version 2 or (at your option) any later version.
> >> + */
> >> +
> >> +#include "qemu/osdep.h"
> >> +#include "qemu-common.h"
> >> 

Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Niek Linnenbank]

Hi Philippe, Thomas,

Op vr 27 nov. 2020 18:57 schreef Philippe Mathieu-Daudé :

> On 11/27/20 6:47 PM, Thomas Huth wrote:
> > On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
> >> We lately realized that the Avocado framework was not designed
> >> to be regularly run on CI environments. Therefore, as of 5.2
> >> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
> >> current users, it is possible to keep the current behavior by
> >> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
> >> (see [*]).
> >> From now on, using these jobs (or adding new tests to them)
> >> is strongly discouraged.
> >>
> >> Tests based on Avocado will be ported to new job schemes during
> >> the next releases, with better documentation and templates.
> >
> > Why should we disable the jobs by default as long as there is no
> replacement
> > available yet?
>
> Why keep it enabled if it is failing randomly, if images hardcoded
> in tests are being removed from public servers, etc...?
>
>
> They are not disabled, they are still runnable manually or setting
> QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE...
>
> We realized by default Avocado runs all tests on the CI jobs,
> triggering failures and complains. Developer stopped to contribute/
> review integration tests because of that. We want developers be
> able to contribute tests to the repository fearlessly.
>
> If we don't change anything, we'll keep having CI failures due
> to Avocado design issues (artifacts removed from remote servers,
> etc...).
>

I share your concern about the artifacts not being stored on a reliable
location that can be used for all Qemu acceptance tests. In particular for
the Orange Pi PC tests we have seen it happening, that the image we use was
removed from the armbian server.

As a temporary solution perhaps we can use github to store artifacts, until
we have a proper location?


> I haven't seen any attempt on this list to improve the current
> fragile situation, but better suggestions are certainly welcome.
>
> Thanks,
>
> Phil.
>
>
>

[PATCH 3/3] target/openrisc: Move pic_cpu code into CPU object proper

[Peter Maydell]

The openrisc code uses an old style of interrupt handling, where a
separate standalone set of qemu_irqs invoke a function
openrisc_pic_cpu_handler() which signals the interrupt to the CPU
proper by directly calling cpu_interrupt() and cpu_reset_interrupt().
Because CPU objects now inherit (indirectly) from TYPE_DEVICE, they
can have GPIO input lines themselves, and the neater modern way to
implement this is to simply have the CPU object itself provide the
input IRQ lines.

Create GPIO inputs to the OpenRISC CPU object, and make the only user
of cpu_openrisc_pic_init() wire up directly to those instead.

This allows us to delete the hw/openrisc/pic_cpu.c file entirely.

This fixes a trivial memory leak reported by Coverity of the IRQs
allocated in cpu_openrisc_pic_init().

Fixes: Coverity CID 1421934
Signed-off-by: Peter Maydell 
---
 target/openrisc/cpu.h  |  1 -
 hw/openrisc/openrisc_sim.c |  3 +-
 hw/openrisc/pic_cpu.c  | 61 --
 target/openrisc/cpu.c  | 32 
 hw/openrisc/meson.build|  2 +-
 5 files changed, 34 insertions(+), 65 deletions(-)
 delete mode 100644 hw/openrisc/pic_cpu.c

diff --git a/target/openrisc/cpu.h b/target/openrisc/cpu.h
index bd42faf144f..82cbaeb4f84 100644
--- a/target/openrisc/cpu.h
+++ b/target/openrisc/cpu.h
@@ -293,7 +293,6 @@ typedef struct CPUOpenRISCState {
 uint32_t picmr; /* Interrupt mask register */
 uint32_t picsr; /* Interrupt contrl register*/
 #endif
-void *irq[32];  /* Interrupt irq input */
 } CPUOpenRISCState;
 
 /**
diff --git a/hw/openrisc/openrisc_sim.c b/hw/openrisc/openrisc_sim.c
index 75ba0f47444..39f1d344ae9 100644
--- a/hw/openrisc/openrisc_sim.c
+++ b/hw/openrisc/openrisc_sim.c
@@ -54,7 +54,7 @@ static void main_cpu_reset(void *opaque)
 
 static qemu_irq get_cpu_irq(OpenRISCCPU *cpus[], int cpunum, int irq_pin)
 {
-return cpus[cpunum]->env.irq[irq_pin];
+return qdev_get_gpio_in_named(DEVICE(cpus[cpunum]), "IRQ", irq_pin);
 }
 
 static void openrisc_sim_net_init(hwaddr base, hwaddr descriptors,
@@ -154,7 +154,6 @@ static void openrisc_sim_init(MachineState *machine)
 fprintf(stderr, "Unable to find CPU definition!\n");
 exit(1);
 }
-cpu_openrisc_pic_init(cpus[n]);
 
 cpu_openrisc_clock_init(cpus[n]);
 
diff --git a/hw/openrisc/pic_cpu.c b/hw/openrisc/pic_cpu.c
deleted file mode 100644
index 36f93508309..000
--- a/hw/openrisc/pic_cpu.c
+++ /dev/null
@@ -1,61 +0,0 @@
-/*
- * OpenRISC Programmable Interrupt Controller support.
- *
- * Copyright (c) 2011-2012 Jia Liu 
- * Feng Gao 
- *
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public
- * License as published by the Free Software Foundation; either
- * version 2.1 of the License, or (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public
- * License along with this library; if not, see .
- */
-
-#include "qemu/osdep.h"
-#include "hw/irq.h"
-#include "cpu.h"
-
-/* OpenRISC pic handler */
-static void openrisc_pic_cpu_handler(void *opaque, int irq, int level)
-{
-OpenRISCCPU *cpu = (OpenRISCCPU *)opaque;
-CPUState *cs = CPU(cpu);
-uint32_t irq_bit;
-
-if (irq > 31 || irq < 0) {
-return;
-}
-
-irq_bit = 1U << irq;
-
-if (level) {
-cpu->env.picsr |= irq_bit;
-} else {
-cpu->env.picsr &= ~irq_bit;
-}
-
-if (cpu->env.picsr & cpu->env.picmr) {
-cpu_interrupt(cs, CPU_INTERRUPT_HARD);
-} else {
-cpu_reset_interrupt(cs, CPU_INTERRUPT_HARD);
-cpu->env.picsr = 0;
-}
-}
-
-void cpu_openrisc_pic_init(OpenRISCCPU *cpu)
-{
-int i;
-qemu_irq *qi;
-qi = qemu_allocate_irqs(openrisc_pic_cpu_handler, cpu, NR_IRQS);
-
-for (i = 0; i < NR_IRQS; i++) {
-cpu->env.irq[i] = qi[i];
-}
-}
diff --git a/target/openrisc/cpu.c b/target/openrisc/cpu.c
index 5528c0918f4..b0bdfbe4fe2 100644
--- a/target/openrisc/cpu.c
+++ b/target/openrisc/cpu.c
@@ -65,6 +65,34 @@ static void openrisc_cpu_reset(DeviceState *dev)
 #endif
 }
 
+#ifndef CONFIG_USER_ONLY
+static void openrisc_cpu_set_irq(void *opaque, int irq, int level)
+{
+OpenRISCCPU *cpu = (OpenRISCCPU *)opaque;
+CPUState *cs = CPU(cpu);
+uint32_t irq_bit;
+
+if (irq > 31 || irq < 0) {
+return;
+}
+
+irq_bit = 1U << irq;
+
+if (level) {
+cpu->env.picsr |= irq_bit;
+} else {
+cpu->env.picsr &= ~irq_bit;
+}
+
+if (cpu->env.picsr & cpu->env.picmr) {
+cpu_interrupt(cs, CPU_INTERRUPT_HARD);
+} else 

[PATCH 2/3] hw/openrisc/openrisc_sim: Abstract out "get IRQ x of CPU y"

[Peter Maydell]

We're about to refactor the OpenRISC pic_cpu code in a way that means
that just grabbing the whole qemu_irq[] array of inbound IRQs for a
CPU won't be possible any more.  Abstract out a function for "return
the qemu_irq for IRQ x input of CPU y" so we can more easily replace
the implementation.

Signed-off-by: Peter Maydell 
---
 hw/openrisc/openrisc_sim.c | 38 +-
 1 file changed, 21 insertions(+), 17 deletions(-)

diff --git a/hw/openrisc/openrisc_sim.c b/hw/openrisc/openrisc_sim.c
index a8adf6b70d7..75ba0f47444 100644
--- a/hw/openrisc/openrisc_sim.c
+++ b/hw/openrisc/openrisc_sim.c
@@ -52,8 +52,13 @@ static void main_cpu_reset(void *opaque)
 cpu_set_pc(cs, boot_info.bootstrap_pc);
 }
 
+static qemu_irq get_cpu_irq(OpenRISCCPU *cpus[], int cpunum, int irq_pin)
+{
+return cpus[cpunum]->env.irq[irq_pin];
+}
+
 static void openrisc_sim_net_init(hwaddr base, hwaddr descriptors,
-  int num_cpus, qemu_irq **cpu_irqs,
+  int num_cpus, OpenRISCCPU *cpus[],
   int irq_pin, NICInfo *nd)
 {
 DeviceState *dev;
@@ -70,18 +75,18 @@ static void openrisc_sim_net_init(hwaddr base, hwaddr 
descriptors,
 qdev_prop_set_uint32(splitter, "num-lines", num_cpus);
 qdev_realize_and_unref(splitter, NULL, _fatal);
 for (i = 0; i < num_cpus; i++) {
-qdev_connect_gpio_out(splitter, i, cpu_irqs[i][irq_pin]);
+qdev_connect_gpio_out(splitter, i, get_cpu_irq(cpus, i, irq_pin));
 }
 sysbus_connect_irq(s, 0, qdev_get_gpio_in(splitter, 0));
 } else {
-sysbus_connect_irq(s, 0, cpu_irqs[0][irq_pin]);
+sysbus_connect_irq(s, 0, get_cpu_irq(cpus, 0, irq_pin));
 }
 sysbus_mmio_map(s, 0, base);
 sysbus_mmio_map(s, 1, descriptors);
 }
 
 static void openrisc_sim_ompic_init(hwaddr base, int num_cpus,
-qemu_irq **cpu_irqs, int irq_pin)
+OpenRISCCPU *cpus[], int irq_pin)
 {
 DeviceState *dev;
 SysBusDevice *s;
@@ -93,7 +98,7 @@ static void openrisc_sim_ompic_init(hwaddr base, int num_cpus,
 s = SYS_BUS_DEVICE(dev);
 sysbus_realize_and_unref(s, _fatal);
 for (i = 0; i < num_cpus; i++) {
-sysbus_connect_irq(s, i, cpu_irqs[i][irq_pin]);
+sysbus_connect_irq(s, i, get_cpu_irq(cpus, i, irq_pin));
 }
 sysbus_mmio_map(s, 0, base);
 }
@@ -136,26 +141,24 @@ static void openrisc_sim_init(MachineState *machine)
 {
 ram_addr_t ram_size = machine->ram_size;
 const char *kernel_filename = machine->kernel_filename;
-OpenRISCCPU *cpu = NULL;
+OpenRISCCPU *cpus[2] = {};
 MemoryRegion *ram;
-qemu_irq *cpu_irqs[2];
 qemu_irq serial_irq;
 int n;
 unsigned int smp_cpus = machine->smp.cpus;
 
 assert(smp_cpus >= 1 && smp_cpus <= 2);
 for (n = 0; n < smp_cpus; n++) {
-cpu = OPENRISC_CPU(cpu_create(machine->cpu_type));
-if (cpu == NULL) {
+cpus[n] = OPENRISC_CPU(cpu_create(machine->cpu_type));
+if (cpus[n] == NULL) {
 fprintf(stderr, "Unable to find CPU definition!\n");
 exit(1);
 }
-cpu_openrisc_pic_init(cpu);
-cpu_irqs[n] = (qemu_irq *) cpu->env.irq;
+cpu_openrisc_pic_init(cpus[n]);
 
-cpu_openrisc_clock_init(cpu);
+cpu_openrisc_clock_init(cpus[n]);
 
-qemu_register_reset(main_cpu_reset, cpu);
+qemu_register_reset(main_cpu_reset, cpus[n]);
 }
 
 ram = g_malloc(sizeof(*ram));
@@ -164,15 +167,16 @@ static void openrisc_sim_init(MachineState *machine)
 
 if (nd_table[0].used) {
 openrisc_sim_net_init(0x9200, 0x92000400, smp_cpus,
-  cpu_irqs, 4, nd_table);
+  cpus, 4, nd_table);
 }
 
 if (smp_cpus > 1) {
-openrisc_sim_ompic_init(0x9800, smp_cpus, cpu_irqs, 1);
+openrisc_sim_ompic_init(0x9800, smp_cpus, cpus, 1);
 
-serial_irq = qemu_irq_split(cpu_irqs[0][2], cpu_irqs[1][2]);
+serial_irq = qemu_irq_split(get_cpu_irq(cpus, 0, 2),
+get_cpu_irq(cpus, 1, 2));
 } else {
-serial_irq = cpu_irqs[0][2];
+serial_irq = get_cpu_irq(cpus, 0, 2);
 }
 
 serial_mm_init(get_system_memory(), 0x9000, 0, serial_irq,
-- 
2.20.1

[PATCH 0/3] target/openrisc: Move pic_cpu code into CPU object

[Peter Maydell]

The openrisc code uses an old style of interrupt handling, where a
separate standalone set of qemu_irqs invoke a function
openrisc_pic_cpu_handler() which signals the interrupt to the CPU
proper by directly calling cpu_interrupt() and cpu_reset_interrupt().
Because CPU objects now inherit (indirectly) from TYPE_DEVICE, they
can have GPIO input lines themselves, and the neater modern way to
implement this is to simply have the CPU object itself provide the
input IRQ lines.

The main aim of this patch series is to make that refactoring,
which fixes a trivial memory leak reported by Coverity of the IRQs
allocated in cpu_openrisc_pic_init(), and removes one callsite of
the qemu_allocate_irqs() function.

Patch 1 is a minor bugfix noticed along the way; patch 2 is
there to make the change in patch 3 simpler and clearer to review.

Tested with 'make check' and 'make check-acceptance'.

thanks
-- PMM

Peter Maydell (3):
  hw/openrisc/openrisc_sim: Use IRQ splitter when connecting IRQ to
multiple CPUs
  hw/openrisc/openrisc_sim: Abstract out "get IRQ x of CPU y"
  target/openrisc: Move pic_cpu code into CPU object proper

 target/openrisc/cpu.h  |  1 -
 hw/openrisc/openrisc_sim.c | 46 +---
 hw/openrisc/pic_cpu.c  | 61 --
 target/openrisc/cpu.c  | 32 
 hw/openrisc/Kconfig|  1 +
 hw/openrisc/meson.build|  2 +-
 6 files changed, 63 insertions(+), 80 deletions(-)
 delete mode 100644 hw/openrisc/pic_cpu.c

-- 
2.20.1

[PATCH 1/3] hw/openrisc/openrisc_sim: Use IRQ splitter when connecting IRQ to multiple CPUs

[Peter Maydell]

openrisc_sim_net_init() attempts to connect the IRQ line from the
ethernet device to both CPUs in an SMP configuration by simply caling
sysbus_connect_irq() for it twice.  This doesn't work, because the
second connection simply overrides the first.

Fix this by creating a TYPE_SPLIT_IRQ to split the IRQ in the SMP
case.

Signed-off-by: Peter Maydell 
---
 hw/openrisc/openrisc_sim.c | 13 +++--
 hw/openrisc/Kconfig|  1 +
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/hw/openrisc/openrisc_sim.c b/hw/openrisc/openrisc_sim.c
index d752282e675..a8adf6b70d7 100644
--- a/hw/openrisc/openrisc_sim.c
+++ b/hw/openrisc/openrisc_sim.c
@@ -34,6 +34,7 @@
 #include "hw/sysbus.h"
 #include "sysemu/qtest.h"
 #include "sysemu/reset.h"
+#include "hw/core/split-irq.h"
 
 #define KERNEL_LOAD_ADDR 0x100
 
@@ -64,8 +65,16 @@ static void openrisc_sim_net_init(hwaddr base, hwaddr 
descriptors,
 
 s = SYS_BUS_DEVICE(dev);
 sysbus_realize_and_unref(s, _fatal);
-for (i = 0; i < num_cpus; i++) {
-sysbus_connect_irq(s, 0, cpu_irqs[i][irq_pin]);
+if (num_cpus > 1) {
+DeviceState *splitter = qdev_new(TYPE_SPLIT_IRQ);
+qdev_prop_set_uint32(splitter, "num-lines", num_cpus);
+qdev_realize_and_unref(splitter, NULL, _fatal);
+for (i = 0; i < num_cpus; i++) {
+qdev_connect_gpio_out(splitter, i, cpu_irqs[i][irq_pin]);
+}
+sysbus_connect_irq(s, 0, qdev_get_gpio_in(splitter, 0));
+} else {
+sysbus_connect_irq(s, 0, cpu_irqs[0][irq_pin]);
 }
 sysbus_mmio_map(s, 0, base);
 sysbus_mmio_map(s, 1, descriptors);
diff --git a/hw/openrisc/Kconfig b/hw/openrisc/Kconfig
index 6c1e86884e2..8f284f3ba04 100644
--- a/hw/openrisc/Kconfig
+++ b/hw/openrisc/Kconfig
@@ -3,3 +3,4 @@ config OR1K_SIM
 select SERIAL
 select OPENCORES_ETH
 select OMPIC
+select SPLIT_IRQ
-- 
2.20.1

Re: [PATCH v4 6/6] introduce simple linear scan rate limiting mechanism

[Peter Xu]

On Thu, Nov 26, 2020 at 06:17:34PM +0300, Andrey Gruzdev wrote:
> Since reading UFFD events and saving paged data are performed
> from the same thread, write fault latencies are sensitive to
> migration stream stalls. Limiting total page saving rate is a
> method to reduce amount of noticiable fault resolution latencies.
> 
> Migration bandwidth limiting is achieved via noticing cases of
> out-of-threshold write fault latencies and temporarily disabling
> (strictly speaking, severely throttling) saving non-faulting pages.

So have you done any measurements out of it, as we've talked in previous
version?  Thanks,

-- 
Peter Xu

Re: [PATCH v4 5/6] the rest of write tracking migration code

[Peter Xu]

On Thu, Nov 26, 2020 at 06:17:33PM +0300, Andrey Gruzdev wrote:
> Implementation of bg_migration- iteration/completion/finish.
> 
> Signed-off-by: Andrey Gruzdev 

Both patch 4 & 5 look good to me in general.  Maybe you can even squash this
patch into 4?  It's not scarily hugeas a single patch, but a complete one.

-- 
Peter Xu

Re: [PATCH v3] Fix build with 64 bits time_t

[Fabrice Fontaine]

Hi,

Le ven. 27 nov. 2020 à 08:18, Gerd Hoffmann  a écrit :
>
>   Hi,
>
> > Changes v2 -> v3 (after review of Gerd Hoffmann):
> >  - Replace include on  by
> >"standard-headers/linux/input.h" to try to fix build on rhel-7
>
> Now it complains about ioctl() not being declared.
>
> /me suggests to create a fork @ gitlab.com and push your branch there,
> that'll run it through CI and we have a centos7 build job there which
> should catch this ...
I fixed the issue by including sys/ioctl.h.
However, I have a test failure due to a timeout on
check-system-centos:
https://gitlab.com/ffontaine/qemu/-/jobs/877810804
It does seem to be related to my patch. Should I send a v4 on the mailing list?
>
> take care,
>   Gerd
>
Best Regards,

Fabrice

Re: [PATCH v4 0/6] UFFD write-tracking migration/snapshots

[Peter Xu]

On Thu, Nov 26, 2020 at 06:17:28PM +0300, Andrey Gruzdev wrote:
> Changes v3->v4:
> 
> * 1. Renamed migrate capability 'track-writes-ram'->'background-snapshot'.
> * 2. Use array of incompatible caps to replace bulky 'if' constructs.
> * 3. Moved UFFD low-level code to the separate module ('util/userfaultfd.c').
> * 4. Always do UFFD wr-unprotect on cleanup; just closing file descriptor
> *won't cleanup PTEs anyhow, it will release registration ranges, wait 
> *queues etc. but won't cleanup process MM context on MMU level.
> * 5. Allow to enable 'background-snapshot' capability on Linux-only hosts.
> * 6. Put UFFD code usage under '#ifdef CONFIG_LINUX' prerequisite.
> * 7. Removed 'wt_' from RAMState struct.
> * 8. Refactored ram_find_and_save_block() to make more clean - poll UFFD
> *wr-fault events in get_queued_page(), use ram_save_host_page_pre(),
> *ram_save_host_page_post() notifiers around ram_save_host_page()
> *instead of bulky inline write-unprotect code.

One thing I mentioned previously but it seems still got lost is that we don't
need dirty tracking for live snapshot.

A few pointers for reference:

  memory_global_dirty_log_start()
  migration_bitmap_sync_precopy()
  memory_region_clear_dirty_bitmap()
  ...

These should not be needed.  But this can also be done on top.

Thanks,

-- 
Peter Xu

Re: [PATCH 2/8] hvf: Move common code out

[Alexander Graf]

On 27.11.20 21:00, Roman Bolshakov wrote:

On Thu, Nov 26, 2020 at 10:50:11PM +0100, Alexander Graf wrote:

Until now, Hypervisor.framework has only been available on x86_64 systems.
With Apple Silicon shipping now, it extends its reach to aarch64. To
prepare for support for multiple architectures, let's move common code out
into its own accel directory.

Signed-off-by: Alexander Graf 
---
  MAINTAINERS |   9 +-
  accel/hvf/hvf-all.c |  56 +
  accel/hvf/hvf-cpus.c| 468 
  accel/hvf/meson.build   |   7 +
  accel/meson.build   |   1 +
  include/sysemu/hvf_int.h|  69 ++
  target/i386/hvf/hvf-cpus.c  | 131 --
  target/i386/hvf/hvf-cpus.h  |  25 --
  target/i386/hvf/hvf-i386.h  |  48 +---
  target/i386/hvf/hvf.c   | 360 +--
  target/i386/hvf/meson.build |   1 -
  target/i386/hvf/x86hvf.c|  11 +-
  target/i386/hvf/x86hvf.h|   2 -
  13 files changed, 619 insertions(+), 569 deletions(-)
  create mode 100644 accel/hvf/hvf-all.c
  create mode 100644 accel/hvf/hvf-cpus.c
  create mode 100644 accel/hvf/meson.build
  create mode 100644 include/sysemu/hvf_int.h
  delete mode 100644 target/i386/hvf/hvf-cpus.c
  delete mode 100644 target/i386/hvf/hvf-cpus.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 68bc160f41..ca4b6d9279 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -444,9 +444,16 @@ M: Cameron Esfahani 
  M: Roman Bolshakov 
  W: https://wiki.qemu.org/Features/HVF
  S: Maintained
-F: accel/stubs/hvf-stub.c

There was a patch for that in the RFC series from Claudio.



Yeah, I'm not worried about this hunk :).





  F: target/i386/hvf/
+
+HVF
+M: Cameron Esfahani 
+M: Roman Bolshakov 
+W: https://wiki.qemu.org/Features/HVF
+S: Maintained
+F: accel/hvf/
  F: include/sysemu/hvf.h
+F: include/sysemu/hvf_int.h
  
  WHPX CPUs

  M: Sunil Muthuswamy 
diff --git a/accel/hvf/hvf-all.c b/accel/hvf/hvf-all.c
new file mode 100644
index 00..47d77a472a
--- /dev/null
+++ b/accel/hvf/hvf-all.c
@@ -0,0 +1,56 @@
+/*
+ * QEMU Hypervisor.framework support
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Contributions after 2012-01-13 are licensed under the terms of the
+ * GNU GPL, version 2 or (at your option) any later version.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu-common.h"
+#include "qemu/error-report.h"
+#include "sysemu/hvf.h"
+#include "sysemu/hvf_int.h"
+#include "sysemu/runstate.h"
+
+#include "qemu/main-loop.h"
+#include "sysemu/accel.h"
+
+#include 
+
+bool hvf_allowed;
+HVFState *hvf_state;
+
+void assert_hvf_ok(hv_return_t ret)
+{
+if (ret == HV_SUCCESS) {
+return;
+}
+
+switch (ret) {
+case HV_ERROR:
+error_report("Error: HV_ERROR");
+break;
+case HV_BUSY:
+error_report("Error: HV_BUSY");
+break;
+case HV_BAD_ARGUMENT:
+error_report("Error: HV_BAD_ARGUMENT");
+break;
+case HV_NO_RESOURCES:
+error_report("Error: HV_NO_RESOURCES");
+break;
+case HV_NO_DEVICE:
+error_report("Error: HV_NO_DEVICE");
+break;
+case HV_UNSUPPORTED:
+error_report("Error: HV_UNSUPPORTED");
+break;
+default:
+error_report("Unknown Error");
+}
+
+abort();
+}
diff --git a/accel/hvf/hvf-cpus.c b/accel/hvf/hvf-cpus.c
new file mode 100644
index 00..f9bb5502b7
--- /dev/null
+++ b/accel/hvf/hvf-cpus.c
@@ -0,0 +1,468 @@
+/*
+ * Copyright 2008 IBM Corporation
+ *   2008 Red Hat, Inc.
+ * Copyright 2011 Intel Corporation
+ * Copyright 2016 Veertu, Inc.
+ * Copyright 2017 The Android Open Source Project
+ *
+ * QEMU Hypervisor.framework support
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of version 2 of the GNU General Public
+ * License as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see .
+ *
+ * This file contain code under public domain from the hvdos project:
+ * https://github.com/mist64/hvdos
+ *
+ * Parts Copyright (c) 2011 NetApp, Inc.
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *notice, this list of conditions and the following disclaimer in the
+ 

Re: [PATCH 1/8] hvf: Add hypervisor entitlement to output binaries

[Alexander Graf]

On 27.11.20 20:44, Roman Bolshakov wrote:

On Thu, Nov 26, 2020 at 10:50:10PM +0100, Alexander Graf wrote:

In macOS 11, QEMU only gets access to Hypervisor.framework if it has the
respective entitlement. Add an entitlement template and automatically self
sign and apply the entitlement in the build.

Signed-off-by: Alexander Graf 
---
  accel/hvf/entitlements.plist |  8 
  meson.build  | 30 ++
  scripts/entitlement.sh   | 11 +++
  3 files changed, 45 insertions(+), 4 deletions(-)
  create mode 100644 accel/hvf/entitlements.plist
  create mode 100755 scripts/entitlement.sh

Hi,

I think the patch should go ahead of other changes (with Paolo's fix for
^C) and land into 5.2 because entitlements are needed for x86_64 hvf too
since Big Sur Beta 3. Ad-hoc signing is very convenient for development.

Also, It might be good to have configure/meson option to disable signing
at all. Primarily for homebrew:

https://discourse.brew.sh/t/code-signing-installed-executables/2131/10

There's no established process how to deal with it, e.g. GDB in homebrew
has caveats section for now:

   ==> Caveats
   gdb requires special privileges to access Mach ports.
   You will need to codesign the binary. For instructions, see:

 https://sourceware.org/gdb/wiki/BuildingOnDarwin

The discussion on discourse mentions some plans to do signing in
homebrew CI (with real Developer ID) but none of them are implemented
now.

For now it'd be helpful to provide a way to disable signing and install
the entitlements (if one wants to sign after installation). Similar
issue was raised to fish-shell a while ago:

https://github.com/fish-shell/fish-shell/issues/6952
https://github.com/fish-shell/fish-shell/issues/7467



All binaries are signed in Big Sur by the linker as far as I understand, 
so I don't quite see the point in not signing :). If the build system 
doesn't have access to codesign, it sounds to me like one should fix the 
build system instead? Worst case by injecting a fake codesign binary 
that just calls /bin/true.






diff --git a/accel/hvf/entitlements.plist b/accel/hvf/entitlements.plist
new file mode 100644
index 00..154f3308ef
--- /dev/null
+++ b/accel/hvf/entitlements.plist
@@ -0,0 +1,8 @@
+
+http://www.apple.com/DTDs/PropertyList-1.0.dtd;>
+
+
+com.apple.security.hypervisor
+
+
+
diff --git a/meson.build b/meson.build
index 5062407c70..2a7ff5560c 100644
--- a/meson.build
+++ b/meson.build
@@ -1844,9 +1844,14 @@ foreach target : target_dirs
  }]
endif
foreach exe: execs
-emulators += {exe['name']:
- executable(exe['name'], exe['sources'],
-   install: true,
+exe_name = exe['name']
+exe_sign = 'CONFIG_HVF' in config_target

I don't have Apple Silicon HW but it may require different kind of
entitlements for CONFIG_TCG:

https://developer.apple.com/documentation/apple_silicon/porting_just-in-time_compilers_to_apple_silicon



You only need the JIT entitlement for the App Store. Locally signed 
applications work just fine without. I don't know about binaries you 
download from the internet that were signed with a developer key though.


Keep in mind that for this to work you also need the MAP_JIT and RWX 
toggle changes from another patch set on the ML.



Alex

Re: [PATCH v4 3/6] support UFFD write fault processing in ram_save_iterate()

[Peter Xu]

On Thu, Nov 26, 2020 at 06:17:31PM +0300, Andrey Gruzdev wrote:
> In this particular implementation the same single migration
> thread is responsible for both normal linear dirty page
> migration and procesing UFFD page fault events.
> 
> Processing write faults includes reading UFFD file descriptor,
> finding respective RAM block and saving faulting page to
> the migration stream. After page has been saved, write protection
> can be removed. Since asynchronous version of qemu_put_buffer()
> is expected to be used to save pages, we also have to flush
> migraion stream prior to un-protecting saved memory range.
> 
> Write protection is being removed for any previously protected
> memory chunk that has hit the migration stream. That's valid
> for pages from linear page scan along with write fault pages.

Thanks for working on this version, it looks much cleaner.

> 
> Signed-off-by: Andrey Gruzdev 
> ---
>  migration/ram.c | 155 +---
>  1 file changed, 147 insertions(+), 8 deletions(-)
> 
> diff --git a/migration/ram.c b/migration/ram.c
> index 3adfd1948d..bcdccdaef7 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -1441,6 +1441,76 @@ static RAMBlock *unqueue_page(RAMState *rs, ram_addr_t 
> *offset)
>  return block;
>  }
>  
> +#ifdef CONFIG_LINUX
> +/**
> + * ram_find_block_by_host_address: find RAM block containing host page
> + *
> + * Returns pointer to RAMBlock if found, NULL otherwise
> + *
> + * @rs: current RAM state
> + * @page_address: host page address
> + */
> +static RAMBlock *ram_find_block_by_host_address(RAMState *rs, hwaddr 
> page_address)

Reuse qemu_ram_block_from_host() somehow?

> +{
> +RAMBlock *bs = rs->last_seen_block;
> +
> +do {
> +if (page_address >= (hwaddr) bs->host && (page_address + 
> TARGET_PAGE_SIZE) <=
> +((hwaddr) bs->host + bs->max_length)) {
> +return bs;
> +}
> +
> +bs = QLIST_NEXT_RCU(bs, next);
> +if (!bs) {
> +/* Hit the end of the list */
> +bs = QLIST_FIRST_RCU(_list.blocks);
> +}
> +} while (bs != rs->last_seen_block);
> +
> +return NULL;
> +}
> +
> +/**
> + * poll_fault_page: try to get next UFFD write fault page and, if pending 
> fault
> + *   is found, return RAM block pointer and page offset
> + *
> + * Returns pointer to the RAMBlock containing faulting page,
> + *   NULL if no write faults are pending
> + *
> + * @rs: current RAM state
> + * @offset: page offset from the beginning of the block
> + */
> +static RAMBlock *poll_fault_page(RAMState *rs, ram_addr_t *offset)
> +{
> +struct uffd_msg uffd_msg;
> +hwaddr page_address;
> +RAMBlock *bs;
> +int res;
> +
> +if (!migrate_background_snapshot()) {
> +return NULL;
> +}
> +
> +res = uffd_read_events(rs->uffdio_fd, _msg, 1);
> +if (res <= 0) {
> +return NULL;
> +}
> +
> +page_address = uffd_msg.arg.pagefault.address;
> +bs = ram_find_block_by_host_address(rs, page_address);
> +if (!bs) {
> +/* In case we couldn't find respective block, just unprotect 
> faulting page. */
> +uffd_protect_memory(rs->uffdio_fd, page_address, TARGET_PAGE_SIZE, 
> false);
> +error_report("ram_find_block_by_host_address() failed: 
> address=0x%0lx",
> +page_address);

Looks ok to error_report() instead of assert(), but I'll suggest drop the call
to uffd_protect_memory() at least.  The only reason to not use assert() is
because we try our best to avoid crashing the vm, however I really doubt
whether uffd_protect_memory() is the right thing to do even if it happens - we
may at last try to unprotect some strange pages that we don't even know where
it is...

> +return NULL;
> +}
> +
> +*offset = (ram_addr_t) (page_address - (hwaddr) bs->host);
> +return bs;
> +}
> +#endif /* CONFIG_LINUX */
> +
>  /**
>   * get_queued_page: unqueue a page from the postcopy requests
>   *
> @@ -1480,6 +1550,16 @@ static bool get_queued_page(RAMState *rs, 
> PageSearchStatus *pss)
>  
>  } while (block && !dirty);
>  
> +#ifdef CONFIG_LINUX
> +if (!block) {
> +/*
> + * Poll write faults too if background snapshot is enabled; that's
> + * when we have vcpus got blocked by the write protected pages.
> + */
> +block = poll_fault_page(rs, );
> +}
> +#endif /* CONFIG_LINUX */
> +
>  if (block) {
>  /*
>   * As soon as we start servicing pages out of order, then we have
> @@ -1753,6 +1833,55 @@ static int ram_save_host_page(RAMState *rs, 
> PageSearchStatus *pss,
>  return pages;
>  }
>  
> +/**
> + * ram_save_host_page_pre: ram_save_host_page() pre-notifier
> + *
> + * @rs: current RAM state
> + * @pss: page-search-status structure
> + * @opaque: pointer to receive opaque context value
> + */
> +static inline
> +void ram_save_host_page_pre(RAMState *rs, PageSearchStatus *pss, void 
> **opaque)
> 

Re: [PATCH 1/8] hvf: Add hypervisor entitlement to output binaries

[Paolo Bonzini]

Il ven 27 nov 2020, 20:44 Roman Bolshakov  ha
scritto:

> On Thu, Nov 26, 2020 at 10:50:10PM +0100, Alexander Graf wrote:
> > In macOS 11, QEMU only gets access to Hypervisor.framework if it has the
> > respective entitlement. Add an entitlement template and automatically
> self
> > sign and apply the entitlement in the build.
> >
> > Signed-off-by: Alexander Graf 
> > ---
> >  accel/hvf/entitlements.plist |  8 
> >  meson.build  | 30 ++
> >  scripts/entitlement.sh   | 11 +++
> >  3 files changed, 45 insertions(+), 4 deletions(-)
> >  create mode 100644 accel/hvf/entitlements.plist
> >  create mode 100755 scripts/entitlement.sh
>
> Hi,
>
> I think the patch should go ahead of other changes (with Paolo's fix for
> ^C) and land into 5.2 because entitlements are needed for x86_64 hvf too
> since Big Sur Beta 3. Ad-hoc signing is very convenient for development.
>

It's certainly too late for 5.2, but we could include the patch in the
release notes and in 5.2.1.

Paolo

Also, It might be good to have configure/meson option to disable signing
> at all. Primarily for homebrew:
>
> https://discourse.brew.sh/t/code-signing-installed-executables/2131/10
>
> There's no established process how to deal with it, e.g. GDB in homebrew
> has caveats section for now:
>
>   ==> Caveats
>   gdb requires special privileges to access Mach ports.
>   You will need to codesign the binary. For instructions, see:
>
> https://sourceware.org/gdb/wiki/BuildingOnDarwin
>
> The discussion on discourse mentions some plans to do signing in
> homebrew CI (with real Developer ID) but none of them are implemented
> now.
>
> For now it'd be helpful to provide a way to disable signing and install
> the entitlements (if one wants to sign after installation). Similar
> issue was raised to fish-shell a while ago:
>
> https://github.com/fish-shell/fish-shell/issues/6952
> https://github.com/fish-shell/fish-shell/issues/7467
>
> >
> > diff --git a/accel/hvf/entitlements.plist b/accel/hvf/entitlements.plist
> > new file mode 100644
> > index 00..154f3308ef
> > --- /dev/null
> > +++ b/accel/hvf/entitlements.plist
> > @@ -0,0 +1,8 @@
> > +
> > + http://www.apple.com/DTDs/PropertyList-1.0.dtd;>
> > +
> > +
> > +com.apple.security.hypervisor
> > +
> > +
> > +
> > diff --git a/meson.build b/meson.build
> > index 5062407c70..2a7ff5560c 100644
> > --- a/meson.build
> > +++ b/meson.build
> > @@ -1844,9 +1844,14 @@ foreach target : target_dirs
> >  }]
> >endif
> >foreach exe: execs
> > -emulators += {exe['name']:
> > - executable(exe['name'], exe['sources'],
> > -   install: true,
> > +exe_name = exe['name']
> > +exe_sign = 'CONFIG_HVF' in config_target
>
> I don't have Apple Silicon HW but it may require different kind of
> entitlements for CONFIG_TCG:
>
>
> https://developer.apple.com/documentation/apple_silicon/porting_just-in-time_compilers_to_apple_silicon
>
> Thanks,
> Roman
>
> > +if exe_sign
> > +  exe_name += '-unsigned'
> > +endif
> > +
> > +emulator = executable(exe_name, exe['sources'],
> > +   install: not exe_sign,
> > c_args: c_args,
> > dependencies: arch_deps + deps + exe['dependencies'],
> > objects: lib.extract_all_objects(recursive: true),
> > @@ -1854,7 +1859,24 @@ foreach target : target_dirs
> > link_depends: [block_syms, qemu_syms] +
> exe.get('link_depends', []),
> > link_args: link_args,
> > gui_app: exe['gui'])
> > -}
> > +
> > +if exe_sign
> > +  exe_full = meson.current_build_dir() / exe['name']
> > +  emulators += {exe['name'] : custom_target(exe['name'],
> > +   install: true,
> > +   install_dir: get_option('bindir'),
> > +   depends: emulator,
> > +   output: exe['name'],
> > +   command: [
> > + meson.current_source_dir() /
> 'scripts/entitlement.sh',
> > + meson.current_build_dir() / exe['name'] +
> '-unsigned',
> > + meson.current_build_dir() / exe['name'],
> > + meson.current_source_dir() /
> 'accel/hvf/entitlements.plist'
> > +   ])
> > +  }
> > +else
> > +  emulators += {exe['name']: emulator}
> > +endif
> >
> >  if 'CONFIG_TRACE_SYSTEMTAP' in config_host
> >foreach stp: [
> > diff --git a/scripts/entitlement.sh b/scripts/entitlement.sh
> > new file mode 100755
> > index 00..7ed9590bf9
> > --- /dev/null
> > +++ b/scripts/entitlement.sh
> > @@ -0,0 +1,11 @@
> > +#!/bin/sh -e
> > +#
> > +# Helper script for the build process to apply entitlements
> > +
> > +SRC="$1"
> > +DST="$2"
> > +ENTITLEMENT="$3"
> > +
> > +rm -f "$2"
> > +cp -a "$SRC" "$DST"
> > +codesign --entitlements "$ENTITLEMENT" --force -s - "$DST"
> > --
> > 2.24.3 (Apple Git-128)
> >
> 

Re: [PATCH v4 2/6] introduce UFFD-WP low-level interface helpers

[Peter Xu]

On Thu, Nov 26, 2020 at 06:17:30PM +0300, Andrey Gruzdev wrote:
> Implemented support for the whole RAM block memory
> protection/un-protection. Introduced higher level
> ram_write_tracking_start() and ram_write_tracking_stop()
> to start/stop tracking guest memory writes.

The whole patch looks good to me in general.  A few nitpickings below..

> 
> Signed-off-by: Andrey Gruzdev 
> ---
>  include/exec/memory.h  |   7 ++
>  include/qemu/userfaultfd.h |  29 +
>  migration/ram.c| 120 +
>  migration/ram.h|   4 +
>  util/meson.build   |   1 +
>  util/userfaultfd.c | 215 +
>  6 files changed, 376 insertions(+)
>  create mode 100644 include/qemu/userfaultfd.h
>  create mode 100644 util/userfaultfd.c
> 
> diff --git a/include/exec/memory.h b/include/exec/memory.h
> index 0f3e6bcd5e..3d798fce16 100644
> --- a/include/exec/memory.h
> +++ b/include/exec/memory.h
> @@ -139,6 +139,13 @@ typedef struct IOMMUNotifier IOMMUNotifier;
>  /* RAM is a persistent kind memory */
>  #define RAM_PMEM (1 << 5)
>  
> +/*
> + * UFFDIO_WRITEPROTECT is used on this RAMBlock to
> + * support 'write-tracking' migration type.
> + * Implies ram_state->ram_wt_enabled.
> + */
> +#define RAM_UF_WRITEPROTECT (1 << 6)
> +
>  static inline void iommu_notifier_init(IOMMUNotifier *n, IOMMUNotify fn,
> IOMMUNotifierFlag flags,
> hwaddr start, hwaddr end,
> diff --git a/include/qemu/userfaultfd.h b/include/qemu/userfaultfd.h
> new file mode 100644
> index 00..fb843c76db
> --- /dev/null
> +++ b/include/qemu/userfaultfd.h
> @@ -0,0 +1,29 @@
> +/*
> + * Linux UFFD-WP support
> + *
> + * Copyright Virtuozzo GmbH, 2020
> + *
> + * Authors:
> + *  Andrey Gruzdev   
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or
> + * later.  See the COPYING file in the top-level directory.
> + */
> +
> +#ifndef USERFAULTFD_H
> +#define USERFAULTFD_H
> +
> +#include "qemu/osdep.h"
> +#include "exec/hwaddr.h"
> +#include 
> +
> +int uffd_create_fd(void);
> +void uffd_close_fd(int uffd);
> +int uffd_register_memory(int uffd, hwaddr start, hwaddr length,
> +bool track_missing, bool track_wp);
> +int uffd_unregister_memory(int uffd, hwaddr start, hwaddr length);
> +int uffd_protect_memory(int uffd, hwaddr start, hwaddr length, bool wp);
> +int uffd_read_events(int uffd, struct uffd_msg *msgs, int count);
> +bool uffd_poll_events(int uffd, int tmo);
> +
> +#endif /* USERFAULTFD_H */
> diff --git a/migration/ram.c b/migration/ram.c
> index 7811cde643..3adfd1948d 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -56,6 +56,11 @@
>  #include "savevm.h"
>  #include "qemu/iov.h"
>  #include "multifd.h"
> +#include "sysemu/runstate.h"
> +
> +#ifdef CONFIG_LINUX
> +#include "qemu/userfaultfd.h"
> +#endif
>  
>  /***/
>  /* ram save/restore */
> @@ -298,6 +303,8 @@ struct RAMSrcPageRequest {
>  struct RAMState {
>  /* QEMUFile used for this migration */
>  QEMUFile *f;
> +/* UFFD file descriptor, used in 'write-tracking' migration */
> +int uffdio_fd;
>  /* Last block that we have visited searching for dirty pages */
>  RAMBlock *last_seen_block;
>  /* Last block from where we have sent data */
> @@ -3788,6 +3795,119 @@ static int ram_resume_prepare(MigrationState *s, void 
> *opaque)
>  return 0;
>  }
>  
> +/*
> + * ram_write_tracking_start: start UFFD-WP memory tracking
> + *
> + * Returns 0 for success or negative value in case of error
> + *

(extra new line)

> + */
> +int ram_write_tracking_start(void)
> +{
> +#ifdef CONFIG_LINUX
> +int uffd;
> +RAMState *rs = ram_state;
> +RAMBlock *bs;
> +
> +/* Open UFFD file descriptor */
> +uffd = uffd_create_fd();
> +if (uffd < 0) {
> +return uffd;
> +}
> +rs->uffdio_fd = uffd;

May need a rcu_read_lock() here to guarantee safe access to
RAMBLOCK_FOREACH_NOT_IGNORED.

> +
> +RAMBLOCK_FOREACH_NOT_IGNORED(bs) {
> +/* Nothing to do with read-only and MMIO-writable regions */
> +if (bs->mr->readonly || bs->mr->rom_device) {
> +continue;
> +}
> +
> +bs->flags |= RAM_UF_WRITEPROTECT;
> +/* Register block memory with UFFD to track writes */
> +if (uffd_register_memory(rs->uffdio_fd, (hwaddr) bs->host,
> +bs->max_length, false, true)) {
> +goto fail;
> +}
> +/* Apply UFFD write protection to the block memory range */
> +if (uffd_protect_memory(rs->uffdio_fd, (hwaddr) bs->host,
> +bs->max_length, true)) {
> +goto fail;
> +}
> +
> +info_report("UFFD-WP write-tracking enabled: "
> +"block_id=%s page_size=%zu start=%p length=%lu "
> +"romd_mode=%i ram=%i readonly=%i nonvolatile=%i 
> 

Re: [RFC v6 07/11] i386: move TCG cpu class initialization out of helper.c

[Eduardo Habkost]

On Fri, Nov 27, 2020 at 08:47:00PM +0100, Claudio Fontana wrote:
> On 11/27/20 8:04 PM, Eduardo Habkost wrote:
[...]
> > Maybe we should rename CPUClass.synchronize_from_tb to
> > CPUClass.tcg_synchronize_from_tb?  Maybe we should have a
> 
> possibly, yes.
> 
> > separate TCGCpuOperations struct to carry TCG-specific methods?
> 
> 
> interesting, will think about it.

I'm working on it at:
https://gitlab.com/ehabkost/qemu/-/commits/work/tcg-cpu-ops

Feel free to reuse it, if you want to do it before your series.
Otherwise, I can rebase it after your series is merged.

I didn't touch do_interrupt(), because of the aarch64 weirdness.

-- 
Eduardo

Re: [PATCH v2] drivers/virt: vmgenid: add vm generation id driver

[Jann Horn]

On Fri, Nov 27, 2020 at 8:04 PM Catangiu, Adrian Costin
 wrote:
> On 27/11/2020 20:22, Jann Horn wrote:
> > On Fri, Nov 20, 2020 at 11:29 PM Jann Horn  wrote:
> >> On Mon, Nov 16, 2020 at 4:35 PM Catangiu, Adrian Costin
> >>  wrote:
> >>> This patch is a driver that exposes a monotonic incremental Virtual
> >>> Machine Generation u32 counter via a char-dev FS interface that
> >>> provides sync and async VmGen counter updates notifications. It also
> >>> provides VmGen counter retrieval and confirmation mechanisms.
> >>>
> >>> The hw provided UUID is not exposed to userspace, it is internally
> >>> used by the driver to keep accounting for the exposed VmGen counter.
> >>> The counter starts from zero when the driver is initialized and
> >>> monotonically increments every time the hw UUID changes (the VM
> >>> generation changes).
> >>>
> >>> On each hw UUID change, the new hypervisor-provided UUID is also fed
> >>> to the kernel RNG.
> >> As for v1:
> >>
> >> Is there a reasonable usecase for the "confirmation" mechanism? It
> >> doesn't seem very useful to me.
>
> I think it adds value in complex scenarios with multiple users of the
> mechanism, potentially at varying layers of the stack, different
> processes and/or runtime libraries.
>
> The driver offers a natural place to handle minimal orchestration
> support and offer visibility in system-wide status.
>
> A high-level service that trusts all system components to properly use
> the confirmation mechanism can actually block and wait patiently for the
> system to adjust to the new world. Even if it doesn't trust all
> components it can still do a best-effort, timeout block.

What concrete action would that high-level service be able to take
after waiting for such an event?

My model of the vmgenid mechanism is that RNGs and cryptographic
libraries that use it need to be fundamentally written such that it is
guaranteed that a VM fork can not cause the same random number /
counter / ... to be reused for two different cryptographic operations
in a way visible to an attacker. This means that e.g. TLS libraries
need to, between accepting unencrypted input and sending out encrypted
data, check whether the vmgenid changed since the connection was set
up, and if a vmgenid change occurred, kill the connection.

Can you give a concrete example of a usecase where the vmgenid
mechanism is used securely and the confirmation mechanism is necessary
as part of that?

> >> How do you envision integrating this with libraries that have to work
> >> in restrictive seccomp sandboxes? If this was in the vDSO, that would
> >> be much easier.
>
> Since this mechanism targets all of userspace stack, the usecase greatly
> vary. I doubt we can have a single silver bullet interface.
>
> For example, the mmap interface targets user space RNGs, where as fast
> and as race free as possible is key. But there also higher level
> applications that don't manage their own memory or don't have access to
> low-level primitives so they can't use the mmap or even vDSO interfaces.
> That's what the rest of the logic is there for, the read+poll interface
> and all of the orchestration logic.

Are you saying that, because people might not want to write proper
bindings for this interface while also being unwilling to take the
performance hit of calling read() in every place where they would have
to do so to be fully correct, you want to build a "best-effort"
mechanism that is deliberately designed to allow some cryptographic
state reuse in a limited time window?

> Like you correctly point out, there are also scenarios like tight
> seccomp jails where even the FS interfaces is inaccessible. For cases
> like this and others, I believe we will have to work incrementally to
> build up the interface diversity to cater to all the user scenarios
> diversity.

It would be much nicer if we could have one simple interface that lets
everyone correctly do what they need to, though...

Re: [PATCH 2/8] hvf: Move common code out

[Roman Bolshakov]

On Thu, Nov 26, 2020 at 10:50:11PM +0100, Alexander Graf wrote:
> Until now, Hypervisor.framework has only been available on x86_64 systems.
> With Apple Silicon shipping now, it extends its reach to aarch64. To
> prepare for support for multiple architectures, let's move common code out
> into its own accel directory.
> 
> Signed-off-by: Alexander Graf 
> ---
>  MAINTAINERS |   9 +-
>  accel/hvf/hvf-all.c |  56 +
>  accel/hvf/hvf-cpus.c| 468 
>  accel/hvf/meson.build   |   7 +
>  accel/meson.build   |   1 +
>  include/sysemu/hvf_int.h|  69 ++
>  target/i386/hvf/hvf-cpus.c  | 131 --
>  target/i386/hvf/hvf-cpus.h  |  25 --
>  target/i386/hvf/hvf-i386.h  |  48 +---
>  target/i386/hvf/hvf.c   | 360 +--
>  target/i386/hvf/meson.build |   1 -
>  target/i386/hvf/x86hvf.c|  11 +-
>  target/i386/hvf/x86hvf.h|   2 -
>  13 files changed, 619 insertions(+), 569 deletions(-)
>  create mode 100644 accel/hvf/hvf-all.c
>  create mode 100644 accel/hvf/hvf-cpus.c
>  create mode 100644 accel/hvf/meson.build
>  create mode 100644 include/sysemu/hvf_int.h
>  delete mode 100644 target/i386/hvf/hvf-cpus.c
>  delete mode 100644 target/i386/hvf/hvf-cpus.h
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 68bc160f41..ca4b6d9279 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -444,9 +444,16 @@ M: Cameron Esfahani 
>  M: Roman Bolshakov 
>  W: https://wiki.qemu.org/Features/HVF
>  S: Maintained
> -F: accel/stubs/hvf-stub.c

There was a patch for that in the RFC series from Claudio.

>  F: target/i386/hvf/
> +
> +HVF
> +M: Cameron Esfahani 
> +M: Roman Bolshakov 
> +W: https://wiki.qemu.org/Features/HVF
> +S: Maintained
> +F: accel/hvf/
>  F: include/sysemu/hvf.h
> +F: include/sysemu/hvf_int.h
>  
>  WHPX CPUs
>  M: Sunil Muthuswamy 
> diff --git a/accel/hvf/hvf-all.c b/accel/hvf/hvf-all.c
> new file mode 100644
> index 00..47d77a472a
> --- /dev/null
> +++ b/accel/hvf/hvf-all.c
> @@ -0,0 +1,56 @@
> +/*
> + * QEMU Hypervisor.framework support
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2.  See
> + * the COPYING file in the top-level directory.
> + *
> + * Contributions after 2012-01-13 are licensed under the terms of the
> + * GNU GPL, version 2 or (at your option) any later version.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu-common.h"
> +#include "qemu/error-report.h"
> +#include "sysemu/hvf.h"
> +#include "sysemu/hvf_int.h"
> +#include "sysemu/runstate.h"
> +
> +#include "qemu/main-loop.h"
> +#include "sysemu/accel.h"
> +
> +#include 
> +
> +bool hvf_allowed;
> +HVFState *hvf_state;
> +
> +void assert_hvf_ok(hv_return_t ret)
> +{
> +if (ret == HV_SUCCESS) {
> +return;
> +}
> +
> +switch (ret) {
> +case HV_ERROR:
> +error_report("Error: HV_ERROR");
> +break;
> +case HV_BUSY:
> +error_report("Error: HV_BUSY");
> +break;
> +case HV_BAD_ARGUMENT:
> +error_report("Error: HV_BAD_ARGUMENT");
> +break;
> +case HV_NO_RESOURCES:
> +error_report("Error: HV_NO_RESOURCES");
> +break;
> +case HV_NO_DEVICE:
> +error_report("Error: HV_NO_DEVICE");
> +break;
> +case HV_UNSUPPORTED:
> +error_report("Error: HV_UNSUPPORTED");
> +break;
> +default:
> +error_report("Unknown Error");
> +}
> +
> +abort();
> +}
> diff --git a/accel/hvf/hvf-cpus.c b/accel/hvf/hvf-cpus.c
> new file mode 100644
> index 00..f9bb5502b7
> --- /dev/null
> +++ b/accel/hvf/hvf-cpus.c
> @@ -0,0 +1,468 @@
> +/*
> + * Copyright 2008 IBM Corporation
> + *   2008 Red Hat, Inc.
> + * Copyright 2011 Intel Corporation
> + * Copyright 2016 Veertu, Inc.
> + * Copyright 2017 The Android Open Source Project
> + *
> + * QEMU Hypervisor.framework support
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of version 2 of the GNU General Public
> + * License as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
> + * General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, see .
> + *
> + * This file contain code under public domain from the hvdos project:
> + * https://github.com/mist64/hvdos
> + *
> + * Parts Copyright (c) 2011 NetApp, Inc.
> + * All rights reserved.
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + *notice, this list of conditions and 

Re: [PATCH v4 1/6] introduce 'background-snapshot' migration capability

[Peter Xu]

On Thu, Nov 26, 2020 at 06:17:29PM +0300, Andrey Gruzdev wrote:
> Signed-off-by: Andrey Gruzdev 

The patch looks good to me, but again, any commit message would be more than
welcomed... as long as it's not empty. :)

-- 
Peter Xu

Re: [RFC v6 07/11] i386: move TCG cpu class initialization out of helper.c

[Claudio Fontana]

Hi Eduardo,

On 11/27/20 8:04 PM, Eduardo Habkost wrote:
> Now that I understand what you are doing here, I have specific
> questions about the functions you are moving, below:
> 
> On Thu, Nov 26, 2020 at 11:32:14PM +0100, Claudio Fontana wrote:
>> Signed-off-by: Claudio Fontana 
> [...]
>> @@ -1495,7 +1497,8 @@ static inline uint64_t x86_cpu_xsave_components(X86CPU 
>> *cpu)
>> cpu->env.features[FEAT_XSAVE_COMP_LO];
>>  }
>>  
>> -const char *get_register_name_32(unsigned int reg)
>> +/* Return name of 32-bit register, from a R_* constant */
>> +static const char *get_register_name_32(unsigned int reg)
>>  {
>>  if (reg >= CPU_NB_REGS32) {
>>  return NULL;
>> @@ -7012,13 +7015,6 @@ static void x86_cpu_set_pc(CPUState *cs, vaddr value)
>>  cpu->env.eip = value;
>>  }
>>  
>> -static void x86_cpu_synchronize_from_tb(CPUState *cs, TranslationBlock *tb)
>> -{
>> -X86CPU *cpu = X86_CPU(cs);
>> -
>> -cpu->env.eip = tb->pc - tb->cs_base;
>> -}
> 
> Question to be answered in the commit message: how can somebody
> be sure this code is not necessary for any other accelerators?
> The TranslationBlock* argument is a hint, but not a guarantee.


easiest is to just trace the calls, but there is a also helpful description of 
all these methods in hw/core/cpu.h.

$ mkid --include="C"
$ gid synchronize_from_tb

include/hw/core/cpu.h:110: *   also implement the synchronize_from_tb hook.
include/hw/core/cpu.h:111: * @synchronize_from_tb: Callback for synchronizing 
state from a TCG
include/hw/core/cpu.h:194:void (*synchronize_from_tb)(CPUState *cpu, struct 
TranslationBlock *tb);
accel/tcg/cpu-exec.c:195:if (cc->synchronize_from_tb) {
accel/tcg/cpu-exec.c:196:cc->synchronize_from_tb(cpu, last_tb);
target/arm/cpu.c:2245:cc->synchronize_from_tb = arm_cpu_synchronize_from_tb;
target/avr/cpu.c:210:cc->synchronize_from_tb = avr_cpu_synchronize_from_tb;
target/hppa/cpu.c:146:cc->synchronize_from_tb = 
hppa_cpu_synchronize_from_tb;
target/microblaze/cpu.c:325:cc->synchronize_from_tb = 
mb_cpu_synchronize_from_tb;
target/mips/cpu.c:241:cc->synchronize_from_tb = 
mips_cpu_synchronize_from_tb;
target/riscv/cpu.c:546:cc->synchronize_from_tb = 
riscv_cpu_synchronize_from_tb;
target/rx/cpu.c:192:cc->synchronize_from_tb = rx_cpu_synchronize_from_tb;
target/sh4/cpu.c:225:cc->synchronize_from_tb = 
superh_cpu_synchronize_from_tb;
target/sparc/cpu.c:871:cc->synchronize_from_tb = 
sparc_cpu_synchronize_from_tb;
target/tricore/cpu.c:165:cc->synchronize_from_tb = 
tricore_cpu_synchronize_from_tb;
target/i386/tcg/cpu.c:129:cc->synchronize_from_tb = 
x86_cpu_synchronize_from_tb;

> Maybe we should rename CPUClass.synchronize_from_tb to
> CPUClass.tcg_synchronize_from_tb?  Maybe we should have a

possibly, yes.

> separate TCGCpuOperations struct to carry TCG-specific methods?


interesting, will think about it.

> 
> (The same questions above apply to the other methods below)
> 
> 
>> -
>>  int x86_cpu_pending_interrupt(CPUState *cs, int interrupt_request)
>>  {
>>  X86CPU *cpu = X86_CPU(cs);
>> @@ -7252,17 +7248,18 @@ static void x86_cpu_common_class_init(ObjectClass 
>> *oc, void *data)
>>  cc->class_by_name = x86_cpu_class_by_name;
>>  cc->parse_features = x86_cpu_parse_featurestr;
>>  cc->has_work = x86_cpu_has_work;
>> +
>>  #ifdef CONFIG_TCG
>> -cc->do_interrupt = x86_cpu_do_interrupt;
>> -cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
> 
> These two are in seg_helper.c, so I agree it makes sense to keep
> it in tcg_cpu_common_class_init().
> 
> I'd like to understand why they are TCG-specific, though.  Are
> CPUClass.do_interrupt and CPUClass.cpu_exec_enter TCG-specific on
> all architectures, or only in x86?

cpu_exec_enter: yes, tcg only on all archs.

Traces as above, it is only called by

accel/tcg/cpu-exec.c

cc->do_interrupt() is very interesting, it _should_ be the same story,
the use of cc->do_interrupt() is normally relegated to accel/tcg/cpu-exec.c,
or to the various implementations of cpu_exec_interrupt (also tcg-specific).

_BUT_ there is an exception in arm64 where it seems to be (strangely) used for 
kvm64:

So in this case the arm kvm64 code is the outlier. There are two calls, one 
introduced in 2015 and one in 2020:

commit e24fd076a59604c4ba3c05fe9d19ea6fc5320a12
Author: Dongjiu Geng 
Date:   Tue May 12 11:06:08 2020 +0800

target-arm: kvm64: handle SIGBUS signal from kernel or KVM


commit 34c45d53026d9c135415d034a8bc30c1a30acf1c
Author: Alex Bennée 
Date:   Thu Dec 17 13:37:15 2015 +

target-arm: kvm - re-inject guest debug exceptions


Maybe Alex and Dongjiu Geng (or maybe Peter?) can shed some light on whether 
this is intentional, or an oversight?

It is the one and only use of cc->do_interrupt outside of TCG. So strange.


> 
>> -#endif
>> +tcg_cpu_common_class_init(cc);
>> +#endif /* CONFIG_TCG */
>> +
>>  cc->dump_state = x86_cpu_dump_state;
>>  

Re: [PATCH 1/8] hvf: Add hypervisor entitlement to output binaries

[Roman Bolshakov]

On Thu, Nov 26, 2020 at 10:50:10PM +0100, Alexander Graf wrote:
> In macOS 11, QEMU only gets access to Hypervisor.framework if it has the
> respective entitlement. Add an entitlement template and automatically self
> sign and apply the entitlement in the build.
> 
> Signed-off-by: Alexander Graf 
> ---
>  accel/hvf/entitlements.plist |  8 
>  meson.build  | 30 ++
>  scripts/entitlement.sh   | 11 +++
>  3 files changed, 45 insertions(+), 4 deletions(-)
>  create mode 100644 accel/hvf/entitlements.plist
>  create mode 100755 scripts/entitlement.sh

Hi,

I think the patch should go ahead of other changes (with Paolo's fix for
^C) and land into 5.2 because entitlements are needed for x86_64 hvf too
since Big Sur Beta 3. Ad-hoc signing is very convenient for development.

Also, It might be good to have configure/meson option to disable signing
at all. Primarily for homebrew:

https://discourse.brew.sh/t/code-signing-installed-executables/2131/10

There's no established process how to deal with it, e.g. GDB in homebrew
has caveats section for now:

  ==> Caveats
  gdb requires special privileges to access Mach ports.
  You will need to codesign the binary. For instructions, see:

https://sourceware.org/gdb/wiki/BuildingOnDarwin

The discussion on discourse mentions some plans to do signing in
homebrew CI (with real Developer ID) but none of them are implemented
now.

For now it'd be helpful to provide a way to disable signing and install
the entitlements (if one wants to sign after installation). Similar
issue was raised to fish-shell a while ago:

https://github.com/fish-shell/fish-shell/issues/6952
https://github.com/fish-shell/fish-shell/issues/7467

> 
> diff --git a/accel/hvf/entitlements.plist b/accel/hvf/entitlements.plist
> new file mode 100644
> index 00..154f3308ef
> --- /dev/null
> +++ b/accel/hvf/entitlements.plist
> @@ -0,0 +1,8 @@
> +
> + "http://www.apple.com/DTDs/PropertyList-1.0.dtd;>
> +
> +
> +com.apple.security.hypervisor
> +
> +
> +
> diff --git a/meson.build b/meson.build
> index 5062407c70..2a7ff5560c 100644
> --- a/meson.build
> +++ b/meson.build
> @@ -1844,9 +1844,14 @@ foreach target : target_dirs
>  }]
>endif
>foreach exe: execs
> -emulators += {exe['name']:
> - executable(exe['name'], exe['sources'],
> -   install: true,
> +exe_name = exe['name']
> +exe_sign = 'CONFIG_HVF' in config_target

I don't have Apple Silicon HW but it may require different kind of
entitlements for CONFIG_TCG:

https://developer.apple.com/documentation/apple_silicon/porting_just-in-time_compilers_to_apple_silicon

Thanks,
Roman

> +if exe_sign
> +  exe_name += '-unsigned'
> +endif
> +
> +emulator = executable(exe_name, exe['sources'],
> +   install: not exe_sign,
> c_args: c_args,
> dependencies: arch_deps + deps + exe['dependencies'],
> objects: lib.extract_all_objects(recursive: true),
> @@ -1854,7 +1859,24 @@ foreach target : target_dirs
> link_depends: [block_syms, qemu_syms] + 
> exe.get('link_depends', []),
> link_args: link_args,
> gui_app: exe['gui'])
> -}
> +
> +if exe_sign
> +  exe_full = meson.current_build_dir() / exe['name']
> +  emulators += {exe['name'] : custom_target(exe['name'],
> +   install: true,
> +   install_dir: get_option('bindir'),
> +   depends: emulator,
> +   output: exe['name'],
> +   command: [
> + meson.current_source_dir() / 'scripts/entitlement.sh',
> + meson.current_build_dir() / exe['name'] + '-unsigned',
> + meson.current_build_dir() / exe['name'],
> + meson.current_source_dir() / 
> 'accel/hvf/entitlements.plist'
> +   ])
> +  }
> +else
> +  emulators += {exe['name']: emulator}
> +endif
>  
>  if 'CONFIG_TRACE_SYSTEMTAP' in config_host
>foreach stp: [
> diff --git a/scripts/entitlement.sh b/scripts/entitlement.sh
> new file mode 100755
> index 00..7ed9590bf9
> --- /dev/null
> +++ b/scripts/entitlement.sh
> @@ -0,0 +1,11 @@
> +#!/bin/sh -e
> +#
> +# Helper script for the build process to apply entitlements
> +
> +SRC="$1"
> +DST="$2"
> +ENTITLEMENT="$3"
> +
> +rm -f "$2"
> +cp -a "$SRC" "$DST"
> +codesign --entitlements "$ENTITLEMENT" --force -s - "$DST"
> -- 
> 2.24.3 (Apple Git-128)
> 
> 

Re: [PATCH 1/2] target/nios2: Move cpu_pic code into CPU object proper

[Philippe Mathieu-Daudé]

On 11/27/20 8:12 PM, Peter Maydell wrote:
> The nios2 code uses an old style of interrupt handling, where a
> separate standalone set of qemu_irqs invoke a function
> nios2_pic_cpu_handler() which signals the interrupt to the CPU proper
> by directly calling cpu_interrupt() and cpu_reset_interrupt().
> Because CPU objects now inherit (indirectly) from TYPE_DEVICE, they
> can have GPIO input lines themselves, and the neater modern way to
> implement this is to simply have the CPU object itself provide the
> input IRQ lines.
> 
> Create named "NMI" and "IRQ" GPIO inputs to the Nios2 CPU object, and
> make the only user of nios2_cpu_pic_init() wire up directly to those
> instead.
> 
> This fixes a Coverity-reported trivial memory leak of the IRQ array
> allocated in nios2_cpu_pic_init().
> 
> Fixes: Coverity CID 1421916
> Signed-off-by: Peter Maydell 
> ---
>  target/nios2/cpu.h|  1 -
>  hw/nios2/10m50_devboard.c |  8 +++-
>  hw/nios2/cpu_pic.c| 31 ---
>  target/nios2/cpu.c| 34 ++
>  4 files changed, 37 insertions(+), 37 deletions(-)
> 
> diff --git a/target/nios2/cpu.h b/target/nios2/cpu.h
> index 86bbe1d8670..b7efb54ba7e 100644
> --- a/target/nios2/cpu.h
> +++ b/target/nios2/cpu.h
> @@ -201,7 +201,6 @@ void nios2_cpu_do_unaligned_access(CPUState *cpu, vaddr 
> addr,
> MMUAccessType access_type,
> int mmu_idx, uintptr_t retaddr);
>  
> -qemu_irq *nios2_cpu_pic_init(Nios2CPU *cpu);
>  void nios2_check_interrupts(CPUNios2State *env);
>  
>  void do_nios2_semihosting(CPUNios2State *env);
> diff --git a/hw/nios2/10m50_devboard.c b/hw/nios2/10m50_devboard.c
> index 5c13b74306f..ac1993e8c08 100644
> --- a/hw/nios2/10m50_devboard.c
> +++ b/hw/nios2/10m50_devboard.c
> @@ -52,7 +52,7 @@ static void nios2_10m50_ghrd_init(MachineState *machine)
>  ram_addr_t tcm_size = 0x1000;/* 1 kiB, but QEMU limit is 4 kiB */
>  ram_addr_t ram_base = 0x0800;
>  ram_addr_t ram_size = 0x0800;
> -qemu_irq *cpu_irq, irq[32];
> +qemu_irq irq[32];
>  int i;
>  
>  /* Physical TCM (tb_ram_1k) with alias at 0xc000 */
> @@ -76,14 +76,12 @@ static void nios2_10m50_ghrd_init(MachineState *machine)
>  /* Create CPU -- FIXME */
>  cpu = NIOS2_CPU(cpu_create(TYPE_NIOS2_CPU));
>  
> -/* Register: CPU interrupt controller (PIC) */
> -cpu_irq = nios2_cpu_pic_init(cpu);
> -
>  /* Register: Internal Interrupt Controller (IIC) */
>  dev = qdev_new("altera,iic");
>  object_property_add_const_link(OBJECT(dev), "cpu", OBJECT(cpu));
>  sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), _fatal);
> -sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0, cpu_irq[0]);
> +sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0,
> +   qdev_get_gpio_in_named(DEVICE(cpu), "IRQ", 0));

Ah, NMI is never used.

Reviewed-by: Philippe Mathieu-Daudé 

>  for (i = 0; i < 32; i++) {
>  irq[i] = qdev_get_gpio_in(dev, i);
>  }
...

Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Wainer dos Santos Moschetta]

Hi Phil,

On 11/27/20 2:41 PM, Philippe Mathieu-Daudé wrote:

We lately realized that the Avocado framework was not designed
to be regularly run on CI environments. Therefore, as of 5.2
we deprecate the gitlab-ci jobs using Avocado. To not disrupt
current users, it is possible to keep the current behavior by
setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
(see [*]).



Could you please detail where have you seen the avocado based tests 
failing that much to justify their removal of CI?


See below some data that I generated from the pipelines for 
https://gitlab.com/qemu-project/qemu/. It seems that jobs which failed 
due timeout are marked as 'skip' and that explain why some pipelines 
don't have a list of failed jobs.


---

Failed 21 out of 99
Pipeline -- 221617704 (staging)
check-patch
Pipeline -- 219276598 (master)
pages
Pipeline -- 218121424 (master)
Pipeline -- 217995873 (staging)
acceptance-system-fedora
Pipeline -- 217995872 (master)
build-tci
Pipeline -- 217551771 (v5.2.0-rc2)
Pipeline -- 217503505 (master)
Pipeline -- 217362429 (master)
Pipeline -- 217328707 (master)
check-crypto-old-nettle
Pipeline -- 216770735 (staging)
acceptance-system-fedora
check-patch
Pipeline -- 215772908 (master)
Pipeline -- 215715025 (staging)
check-crypto-old-gcrypt
Pipeline -- 215715024 (master)
check-crypto-old-nettle
Pipeline -- 214944520 (master)
acceptance-system-debian
Pipeline -- 214460230 (v5.2.0-rc1)
Pipeline -- 214438601 (master)
check-crypto-only-gnutls
Pipeline -- 214273938 (master)
Pipeline -- 214183970 (master)
Pipeline -- 214140305 (master)
Pipeline -- 213892224 (master)
Pipeline -- 213871132 (master)

 From now on, using these jobs (or adding new tests to them)
is strongly discouraged.

Tests based on Avocado will be ported to new job schemes during
the next releases, with better documentation and templates.

[*] 
https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-variable-in-the-ui

Signed-off-by: Philippe Mathieu-Daudé 
---
  .gitlab-ci.yml | 9 +
  1 file changed, 9 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d0173e82b16..2674407cd13 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -66,6 +66,15 @@ include:
  - cd build
  - python3 -c 'import json; r = json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for t in 
r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat
  - du -chs ${CI_PROJECT_DIR}/avocado-cache
+  rules:
+  # As of QEMU 5.2, Avocado is not yet ready to run in CI environments, 
therefore
+  # the jobs based on this template are not run automatically (except if the 
user
+  # explicitly sets the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE environment
+  # variable). Adding new jobs on top of this template is strongly discouraged.
+  - if: $QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE == null
+when: manual
+allow_failure: true
+  - when: always
  
  build-system-ubuntu:

<<: *native_build_job_definition



The script I used to generate the above metrics:

$ pip install --user python-gitlab

$ cat gitlab-stats
#!/bin/env python3

import gitlab

gl = gitlab.Gitlab('http://gitlab.com')
qemu_project = gl.projects.get(11167699)
pipelines = qemu_project.pipelines.list()

total=0
failed=[]
for pipeline in qemu_project.pipelines.list(page=1, per_page=300):
    if pipeline.status == 'running':
    continue
    elif pipeline.status == 'failed':
    failed.append(pipeline)
    total += 1

print("Failed %d out of %d" % (len(failed), total))
for pipeline in failed:
    print('Pipeline -- %d (%s)' % (pipeline.id, pipeline.ref))
    jobs_failed=[]
    for job in pipeline.jobs.list():
    if job.status == 'failed':
    print(job.name)

Re: [PATCH 2/2] target/nios2: Move nios2_check_interrupts() into target/nios2

[Philippe Mathieu-Daudé]

On 11/27/20 8:12 PM, Peter Maydell wrote:
> The function nios2_check_interrupts)() looks only at CPU-internal
> state; it belongs in target/nios2, not hw/nios2.  Move it into the
> same file as its only caller, so it can just be local to that file.
> 
> This removes the only remaining code from cpu_pic.c, so we can delete
> that file entirely.
> 
> Signed-off-by: Peter Maydell 
> ---
>  target/nios2/cpu.h   |  2 --
>  hw/nios2/cpu_pic.c   | 36 
>  target/nios2/op_helper.c |  9 +
>  hw/nios2/meson.build |  2 +-
>  4 files changed, 10 insertions(+), 39 deletions(-)
>  delete mode 100644 hw/nios2/cpu_pic.c

Reviewed-by: Philippe Mathieu-Daudé 

[PATCH 0/2] target/nios2: Roll cpu_pic code into CPU itself

[Peter Maydell]

(As well as the listed nios2 maintainers, I've cc'd a couple
of the more recent contributors to this target in case they're
interested or wish to test the changes.)

The nios2 code uses an old style of interrupt handling, where a
separate standalone set of qemu_irqs invoke a function
nios2_pic_cpu_handler() which signals the interrupt to the CPU proper
by directly calling cpu_interrupt() and cpu_reset_interrupt().
Because CPU objects now inherit (indirectly) from TYPE_DEVICE, they
can have GPIO input lines themselves, and the neater modern way to
implement this is to simply have the CPU object itself provide the
input IRQ lines.

This allows us to roll all of the code in hw/nios/cpu_pic.c
into target/nios2 one way or another.

The motivation here is fixing a trivial Coverity-reported leak
(CID 1421916) of the IRQ array allocated in nios2_cpu_pic_init(),
and also part of the extremely slow refactoring of code away
from using qemu_allocate_irqs() at all.

Tested with 'make check' and 'make check-acceptance' (which does
include a boot test of a nios2 10m50 guest).

thanks
-- PMM

Peter Maydell (2):
  target/nios2: Move cpu_pic code into CPU object proper
  target/nios2: Move nios2_check_interrupts() into target/nios2

 target/nios2/cpu.h|  3 --
 hw/nios2/10m50_devboard.c |  8 ++---
 hw/nios2/cpu_pic.c| 67 ---
 target/nios2/cpu.c| 34 
 target/nios2/op_helper.c  |  9 ++
 hw/nios2/meson.build  |  2 +-
 6 files changed, 47 insertions(+), 76 deletions(-)
 delete mode 100644 hw/nios2/cpu_pic.c

-- 
2.20.1

Re: [PATCH v2] drivers/virt: vmgenid: add vm generation id driver

[Catangiu, Adrian Costin]

Sorry Jann for missing your original email.

On 27/11/2020 20:22, Jann Horn wrote:
> CAUTION: This email originated from outside of the organization. Do not click 
> links or open attachments unless you can confirm the sender and know the 
> content is safe.
>
>
>
> [resend in the hope that amazon will accept my mail this time instead
> of replying "550 Too many invalid recipients" again]
>
> On Fri, Nov 20, 2020 at 11:29 PM Jann Horn  wrote:
>> On Mon, Nov 16, 2020 at 4:35 PM Catangiu, Adrian Costin
>>  wrote:
>>> This patch is a driver that exposes a monotonic incremental Virtual
>>> Machine Generation u32 counter via a char-dev FS interface that
>>> provides sync and async VmGen counter updates notifications. It also
>>> provides VmGen counter retrieval and confirmation mechanisms.
>>>
>>> The hw provided UUID is not exposed to userspace, it is internally
>>> used by the driver to keep accounting for the exposed VmGen counter.
>>> The counter starts from zero when the driver is initialized and
>>> monotonically increments every time the hw UUID changes (the VM
>>> generation changes).
>>>
>>> On each hw UUID change, the new hypervisor-provided UUID is also fed
>>> to the kernel RNG.
>> As for v1:
>>
>> Is there a reasonable usecase for the "confirmation" mechanism? It
>> doesn't seem very useful to me.

I think it adds value in complex scenarios with multiple users of the
mechanism, potentially at varying layers of the stack, different
processes and/or runtime libraries.

The driver offers a natural place to handle minimal orchestration
support and offer visibility in system-wide status.

A high-level service that trusts all system components to properly use
the confirmation mechanism can actually block and wait patiently for the
system to adjust to the new world. Even if it doesn't trust all
components it can still do a best-effort, timeout block.

>>
>> How do you envision integrating this with libraries that have to work
>> in restrictive seccomp sandboxes? If this was in the vDSO, that would
>> be much easier.

Since this mechanism targets all of userspace stack, the usecase greatly
vary. I doubt we can have a single silver bullet interface.

For example, the mmap interface targets user space RNGs, where as fast
and as race free as possible is key. But there also higher level
applications that don't manage their own memory or don't have access to
low-level primitives so they can't use the mmap or even vDSO interfaces.
That's what the rest of the logic is there for, the read+poll interface
and all of the orchestration logic.

Like you correctly point out, there are also scenarios like tight
seccomp jails where even the FS interfaces is inaccessible. For cases
like this and others, I believe we will have to work incrementally to
build up the interface diversity to cater to all the user scenarios
diversity.


Thanks,

Adrian.





Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar 
Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in 
Romania. Registration number J22/2621/2005.

Re: [PATCH] python 3.5 compatibility

[Peter Maydell]

On Fri, 27 Nov 2020 at 19:12, Enrico Weigelt, metux IT consult
 wrote:
> Several stable distros are still on python-3.5, so qemu cannot be
> built there, w/o backporting the whole python stack. In order to
> make qemu build there, make it running w/ python-3.5. The changes
> are mostly removing the new type annotations (which, IMHO, just
> serve for documentation purpose) and replacing f-strings by classic
> string formatting.

We dropped Python 3.5 support deliberately in commit
1b11f28d05121fec43 on the basis that all our supported build
platforms are on at least 3.6 and there were features we wanted there.

Could you say which "stable distros" you have in mind, and whether
they are covered by our "supported build platforms" policy
https://www.qemu.org/docs/master/system/build-platforms.html  ?

thanks
-- PMM

[PATCH 2/2] target/nios2: Move nios2_check_interrupts() into target/nios2

[Peter Maydell]

The function nios2_check_interrupts)() looks only at CPU-internal
state; it belongs in target/nios2, not hw/nios2.  Move it into the
same file as its only caller, so it can just be local to that file.

This removes the only remaining code from cpu_pic.c, so we can delete
that file entirely.

Signed-off-by: Peter Maydell 
---
 target/nios2/cpu.h   |  2 --
 hw/nios2/cpu_pic.c   | 36 
 target/nios2/op_helper.c |  9 +
 hw/nios2/meson.build |  2 +-
 4 files changed, 10 insertions(+), 39 deletions(-)
 delete mode 100644 hw/nios2/cpu_pic.c

diff --git a/target/nios2/cpu.h b/target/nios2/cpu.h
index b7efb54ba7e..2ab82fdc713 100644
--- a/target/nios2/cpu.h
+++ b/target/nios2/cpu.h
@@ -201,8 +201,6 @@ void nios2_cpu_do_unaligned_access(CPUState *cpu, vaddr 
addr,
MMUAccessType access_type,
int mmu_idx, uintptr_t retaddr);
 
-void nios2_check_interrupts(CPUNios2State *env);
-
 void do_nios2_semihosting(CPUNios2State *env);
 
 #define CPU_RESOLVING_TYPE TYPE_NIOS2_CPU
diff --git a/hw/nios2/cpu_pic.c b/hw/nios2/cpu_pic.c
deleted file mode 100644
index 3fb621c5c85..000
--- a/hw/nios2/cpu_pic.c
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- * Altera Nios2 CPU PIC
- *
- * Copyright (c) 2016 Marek Vasut 
- *
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public
- * License as published by the Free Software Foundation; either
- * version 2.1 of the License, or (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
- * Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public
- * License along with this library; if not, see
- * 
- */
-
-#include "qemu/osdep.h"
-#include "cpu.h"
-#include "hw/irq.h"
-
-#include "qemu/config-file.h"
-
-#include "boot.h"
-
-void nios2_check_interrupts(CPUNios2State *env)
-{
-if (env->irq_pending &&
-(env->regs[CR_STATUS] & CR_STATUS_PIE)) {
-env->irq_pending = 0;
-cpu_interrupt(env_cpu(env), CPU_INTERRUPT_HARD);
-}
-}
diff --git a/target/nios2/op_helper.c b/target/nios2/op_helper.c
index a60730faac3..a59003855ab 100644
--- a/target/nios2/op_helper.c
+++ b/target/nios2/op_helper.c
@@ -36,6 +36,15 @@ void helper_mmu_write(CPUNios2State *env, uint32_t rn, 
uint32_t v)
 mmu_write(env, rn, v);
 }
 
+static void nios2_check_interrupts(CPUNios2State *env)
+{
+if (env->irq_pending &&
+(env->regs[CR_STATUS] & CR_STATUS_PIE)) {
+env->irq_pending = 0;
+cpu_interrupt(env_cpu(env), CPU_INTERRUPT_HARD);
+}
+}
+
 void helper_check_interrupts(CPUNios2State *env)
 {
 qemu_mutex_lock_iothread();
diff --git a/hw/nios2/meson.build b/hw/nios2/meson.build
index dd66ebb32f6..6c58e8082b4 100644
--- a/hw/nios2/meson.build
+++ b/hw/nios2/meson.build
@@ -1,5 +1,5 @@
 nios2_ss = ss.source_set()
-nios2_ss.add(files('boot.c', 'cpu_pic.c'))
+nios2_ss.add(files('boot.c'))
 nios2_ss.add(when: 'CONFIG_NIOS2_10M50', if_true: files('10m50_devboard.c'))
 nios2_ss.add(when: 'CONFIG_NIOS2_GENERIC_NOMMU', if_true: 
files('generic_nommu.c'))
 
-- 
2.20.1

[PATCH 1/2] target/nios2: Move cpu_pic code into CPU object proper

[Peter Maydell]

The nios2 code uses an old style of interrupt handling, where a
separate standalone set of qemu_irqs invoke a function
nios2_pic_cpu_handler() which signals the interrupt to the CPU proper
by directly calling cpu_interrupt() and cpu_reset_interrupt().
Because CPU objects now inherit (indirectly) from TYPE_DEVICE, they
can have GPIO input lines themselves, and the neater modern way to
implement this is to simply have the CPU object itself provide the
input IRQ lines.

Create named "NMI" and "IRQ" GPIO inputs to the Nios2 CPU object, and
make the only user of nios2_cpu_pic_init() wire up directly to those
instead.

This fixes a Coverity-reported trivial memory leak of the IRQ array
allocated in nios2_cpu_pic_init().

Fixes: Coverity CID 1421916
Signed-off-by: Peter Maydell 
---
 target/nios2/cpu.h|  1 -
 hw/nios2/10m50_devboard.c |  8 +++-
 hw/nios2/cpu_pic.c| 31 ---
 target/nios2/cpu.c| 34 ++
 4 files changed, 37 insertions(+), 37 deletions(-)

diff --git a/target/nios2/cpu.h b/target/nios2/cpu.h
index 86bbe1d8670..b7efb54ba7e 100644
--- a/target/nios2/cpu.h
+++ b/target/nios2/cpu.h
@@ -201,7 +201,6 @@ void nios2_cpu_do_unaligned_access(CPUState *cpu, vaddr 
addr,
MMUAccessType access_type,
int mmu_idx, uintptr_t retaddr);
 
-qemu_irq *nios2_cpu_pic_init(Nios2CPU *cpu);
 void nios2_check_interrupts(CPUNios2State *env);
 
 void do_nios2_semihosting(CPUNios2State *env);
diff --git a/hw/nios2/10m50_devboard.c b/hw/nios2/10m50_devboard.c
index 5c13b74306f..ac1993e8c08 100644
--- a/hw/nios2/10m50_devboard.c
+++ b/hw/nios2/10m50_devboard.c
@@ -52,7 +52,7 @@ static void nios2_10m50_ghrd_init(MachineState *machine)
 ram_addr_t tcm_size = 0x1000;/* 1 kiB, but QEMU limit is 4 kiB */
 ram_addr_t ram_base = 0x0800;
 ram_addr_t ram_size = 0x0800;
-qemu_irq *cpu_irq, irq[32];
+qemu_irq irq[32];
 int i;
 
 /* Physical TCM (tb_ram_1k) with alias at 0xc000 */
@@ -76,14 +76,12 @@ static void nios2_10m50_ghrd_init(MachineState *machine)
 /* Create CPU -- FIXME */
 cpu = NIOS2_CPU(cpu_create(TYPE_NIOS2_CPU));
 
-/* Register: CPU interrupt controller (PIC) */
-cpu_irq = nios2_cpu_pic_init(cpu);
-
 /* Register: Internal Interrupt Controller (IIC) */
 dev = qdev_new("altera,iic");
 object_property_add_const_link(OBJECT(dev), "cpu", OBJECT(cpu));
 sysbus_realize_and_unref(SYS_BUS_DEVICE(dev), _fatal);
-sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0, cpu_irq[0]);
+sysbus_connect_irq(SYS_BUS_DEVICE(dev), 0,
+   qdev_get_gpio_in_named(DEVICE(cpu), "IRQ", 0));
 for (i = 0; i < 32; i++) {
 irq[i] = qdev_get_gpio_in(dev, i);
 }
diff --git a/hw/nios2/cpu_pic.c b/hw/nios2/cpu_pic.c
index 5ea7e52ab83..3fb621c5c85 100644
--- a/hw/nios2/cpu_pic.c
+++ b/hw/nios2/cpu_pic.c
@@ -26,32 +26,6 @@
 
 #include "boot.h"
 
-static void nios2_pic_cpu_handler(void *opaque, int irq, int level)
-{
-Nios2CPU *cpu = opaque;
-CPUNios2State *env = >env;
-CPUState *cs = CPU(cpu);
-int type = irq ? CPU_INTERRUPT_NMI : CPU_INTERRUPT_HARD;
-
-if (type == CPU_INTERRUPT_HARD) {
-env->irq_pending = level;
-
-if (level && (env->regs[CR_STATUS] & CR_STATUS_PIE)) {
-env->irq_pending = 0;
-cpu_interrupt(cs, type);
-} else if (!level) {
-env->irq_pending = 0;
-cpu_reset_interrupt(cs, type);
-}
-} else {
-if (level) {
-cpu_interrupt(cs, type);
-} else {
-cpu_reset_interrupt(cs, type);
-}
-}
-}
-
 void nios2_check_interrupts(CPUNios2State *env)
 {
 if (env->irq_pending &&
@@ -60,8 +34,3 @@ void nios2_check_interrupts(CPUNios2State *env)
 cpu_interrupt(env_cpu(env), CPU_INTERRUPT_HARD);
 }
 }
-
-qemu_irq *nios2_cpu_pic_init(Nios2CPU *cpu)
-{
-return qemu_allocate_irqs(nios2_pic_cpu_handler, cpu, 2);
-}
diff --git a/target/nios2/cpu.c b/target/nios2/cpu.c
index 8f7011fcb92..4b21e7c6d1c 100644
--- a/target/nios2/cpu.c
+++ b/target/nios2/cpu.c
@@ -64,6 +64,37 @@ static void nios2_cpu_reset(DeviceState *dev)
 #endif
 }
 
+#ifndef CONFIG_USER_ONLY
+static void nios2_cpu_set_nmi(void *opaque, int irq, int level)
+{
+Nios2CPU *cpu = opaque;
+CPUState *cs = CPU(cpu);
+
+if (level) {
+cpu_interrupt(cs, CPU_INTERRUPT_NMI);
+} else {
+cpu_reset_interrupt(cs, CPU_INTERRUPT_NMI);
+}
+}
+
+static void nios2_cpu_set_irq(void *opaque, int irq, int level)
+{
+Nios2CPU *cpu = opaque;
+CPUNios2State *env = >env;
+CPUState *cs = CPU(cpu);
+
+env->irq_pending = level;
+
+if (level && (env->regs[CR_STATUS] & CR_STATUS_PIE)) {
+env->irq_pending = 0;
+cpu_interrupt(cs, CPU_INTERRUPT_HARD);
+} else if (!level) {
+env->irq_pending = 0;
+cpu_reset_interrupt(cs, 

[PATCH] hw: usb: fix break on older kernel headers that miss USBDEVFS_GET_SPEED

[Enrico Weigelt, metux IT consult]

Older kernel headers lack the USBDEVFS_GET_SPEED ioctl. Qemu can easily
work without it, but the api version check isn't sufficient. Instead
time consuming bisection, for finding the really correct version, just
change the ifdef to look for the symbol, instead of api version.

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 hw/usb/host-libusb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/usb/host-libusb.c b/hw/usb/host-libusb.c
index b950501d10..c894a4b148 100644
--- a/hw/usb/host-libusb.c
+++ b/hw/usb/host-libusb.c
@@ -941,7 +941,7 @@ static int usb_host_open(USBHostDevice *s, libusb_device 
*dev, int hostfd)
 usb_host_ep_update(s);
 
 libusb_speed = libusb_get_device_speed(dev);
-#if LIBUSB_API_VERSION >= 0x01000107 && defined(CONFIG_LINUX)
+#ifdef USBDEVFS_GET_SPEED
 if (hostfd && libusb_speed == 0) {
 /*
  * Workaround libusb bug: libusb_get_device_speed() does not
-- 
2.11.0

[PATCH] scripts: minikconf: support config titles

[Enrico Weigelt, metux IT consult]

Add support for config option titles, like the real kconfig does.
Even though they're not presented anywhere yet (since minikconf
only acts in the background), it's good to have them supported,
so we can start adding descriptions in the Kconfig files.

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 scripts/minikconf.py | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/scripts/minikconf.py b/scripts/minikconf.py
index bcd91015d3..28c67906cb 100644
--- a/scripts/minikconf.py
+++ b/scripts/minikconf.py
@@ -206,6 +206,7 @@ class KconfigData:
 self.defined_vars = set()
 self.referenced_vars = dict()
 self.clauses = list()
+self.title = None
 
 # semantic analysis -
 
@@ -290,6 +291,9 @@ class KconfigData:
 cond = (cond & var) if cond is not None else var
 self.clauses.append(KconfigData.SelectClause(symbol, cond))
 
+def do_title(self, title):
+self.title = title
+
 def do_imply(self, var, symbol, cond=None):
 # "config X imply Y [if COND]" is the same as
 # "config Y default y if X [&& COND]"
@@ -323,6 +327,7 @@ TOK_BOOL = 15;TOKENS[TOK_BOOL] = '"bool"';
 TOK_IF = 16;  TOKENS[TOK_IF] = '"if"';
 TOK_ID = 17;  TOKENS[TOK_ID] = 'identifier';
 TOK_EOF = 18; TOKENS[TOK_EOF] = 'end of file';
+TOK_QUOTED = 19;  TOKENS[TOK_QUOTED] = 'quoted string';
 
 class KconfigParserError(Exception):
 def __init__(self, parser, msg, tok=None):
@@ -501,6 +506,7 @@ class KconfigParser:
 # property: DEFAULT y_or_n condition
 #   | DEPENDS ON expr
 #   | SELECT var condition
+#   | BOOL "comment"
 #   | BOOL
 def parse_property(self, var):
 if self.tok == TOK_DEFAULT:
@@ -526,6 +532,9 @@ class KconfigParser:
 self.data.do_imply(var, symbol, cond)
 elif self.tok == TOK_BOOL:
 self.get_token()
+if self.tok == TOK_QUOTED:
+self.data.do_title(self.val)
+self.get_token()
 else:
 raise KconfigParserError(self, 'Error in recursive descent?')
 
@@ -645,6 +654,11 @@ class KconfigParser:
 self.cursor = self.src.find('\n', self.cursor)
 self.val = self.src[start:self.cursor]
 return TOK_SOURCE
+elif self.tok == '"':
+start = self.cursor
+self.cursor = self.src.find('"', self.cursor)+1
+self.val = self.src[start:self.cursor]
+return TOK_QUOTED;
 elif self.tok.isalnum():
 # identifier
 while self.src[self.cursor].isalnum() or self.src[self.cursor] == 
'_':
-- 
2.11.0

[PATCH] python 3.5 compatibility

[Enrico Weigelt, metux IT consult]

Several stable distros are still on python-3.5, so qemu cannot be
built there, w/o backporting the whole python stack. In order to
make qemu build there, make it running w/ python-3.5. The changes
are mostly removing the new type annotations (which, IMHO, just
serve for documentation purpose) and replacing f-strings by classic
string formatting.

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 configure  |  4 +--
 scripts/block-coroutine-wrapper.py | 56 ++
 scripts/qapi/commands.py   |  2 +-
 scripts/qapi/common.py |  2 +-
 scripts/qapi/events.py |  2 +-
 scripts/qapi/gen.py| 10 +++
 scripts/qapi/main.py   |  6 ++--
 scripts/qapi/schema.py |  2 +-
 scripts/qapi/source.py | 10 +++
 9 files changed, 46 insertions(+), 48 deletions(-)

diff --git a/configure b/configure
index 18c26e0389..e1119929fa 100755
--- a/configure
+++ b/configure
@@ -1866,8 +1866,8 @@ fi
 
 # Note that if the Python conditional here evaluates True we will exit
 # with status 1 which is a shell 'false' value.
-if ! $python -c 'import sys; sys.exit(sys.version_info < (3,6))'; then
-  error_exit "Cannot use '$python', Python >= 3.6 is required." \
+if ! $python -c 'import sys; sys.exit(sys.version_info < (3,5))'; then
+  error_exit "Cannot use '$python', Python >= 3.5 is required." \
   "Use --python=/path/to/python to specify a supported Python."
 fi
 
diff --git a/scripts/block-coroutine-wrapper.py 
b/scripts/block-coroutine-wrapper.py
index 0461fd1c45..a36f7af06f 100644
--- a/scripts/block-coroutine-wrapper.py
+++ b/scripts/block-coroutine-wrapper.py
@@ -32,11 +32,11 @@ def gen_header():
 copyright = re.sub('^.*Copyright', 'Copyright', __doc__, flags=re.DOTALL)
 copyright = re.sub('^(?=.)', ' * ', copyright.strip(), flags=re.MULTILINE)
 copyright = re.sub('^$', ' *', copyright, flags=re.MULTILINE)
-return f"""\
+return """\
 /*
  * File is generated by scripts/block-coroutine-wrapper.py
  *
-{copyright}
+"""+copyright+"""
  */
 
 #include "qemu/osdep.h"
@@ -55,7 +55,7 @@ class ParamDecl:
 def __init__(self, param_decl: str) -> None:
 m = self.param_re.match(param_decl.strip())
 if m is None:
-raise ValueError(f'Wrong parameter declaration: "{param_decl}"')
+raise ValueError('Wrong parameter declaration: '+param_decl)
 self.decl = m.group('decl')
 self.type = m.group('type')
 self.name = m.group('name')
@@ -107,44 +107,42 @@ def gen_wrapper(func: FuncDecl) -> str:
 bs = 'bs' if func.args[0].type == 'BlockDriverState *' else 'child->bs'
 struct_name = snake_to_camel(name)
 
-return f"""\
+return """\
 /*
- * Wrappers for {name}
+ * Wrappers for """+name+"""
  */
 
-typedef struct {struct_name} {{
+typedef struct """+struct_name+""" {
 BdrvPollCo poll_state;
-{ func.gen_block('{decl};') }
-}} {struct_name};
+"""+func.gen_block('{decl};')+"""
+} """+struct_name+""";
 
-static void coroutine_fn {name}_entry(void *opaque)
-{{
-{struct_name} *s = opaque;
+static void coroutine_fn """+name+"""_entry(void *opaque)
+{
+"""+struct_name+""" *s = opaque;
 
-s->poll_state.ret = {name}({ func.gen_list('s->{name}') });
+s->poll_state.ret = """+name+"""("""+func.gen_list('s->{name}')+""");
 s->poll_state.in_progress = false;
 
 aio_wait_kick();
-}}
-
-int {func.name}({ func.gen_list('{decl}') })
-{{
-if (qemu_in_coroutine()) {{
-return {name}({ func.gen_list('{name}') });
-}} else {{
-{struct_name} s = {{
-.poll_state.bs = {bs},
+}
+
+int """+func.name+"""("""+func.gen_list('{decl}')+""")
+{
+if (qemu_in_coroutine()) {
+return """+name+"""("""+func.gen_list('{name}')+""");
+} else {
+"""+struct_name+""" s = {
+.poll_state.bs = """+bs+""",
 .poll_state.in_progress = true,
+"""+func.gen_block('.{name} = {name},')+"""
+};
 
-{ func.gen_block('.{name} = {name},') }
-}};
-
-s.poll_state.co = qemu_coroutine_create({name}_entry, );
+s.poll_state.co = qemu_coroutine_create("""+name+"""_entry, );
 
 return bdrv_poll_co(_state);
-}}
-}}"""
-
+}
+}"""
 
 def gen_wrappers(input_code: str) -> str:
 res = ''
@@ -157,7 +155,7 @@ def gen_wrappers(input_code: str) -> str:
 
 if __name__ == '__main__':
 if len(sys.argv) < 3:
-exit(f'Usage: {sys.argv[0]} OUT_FILE.c IN_FILE.[ch]...')
+exit('Usage: '+sys.argv[0]+' OUT_FILE.c IN_FILE.[ch]...')
 
 with open(sys.argv[1], 'w', encoding='utf-8') as f_out:
 f_out.write(gen_header())
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 50978090b4..3b22a16759 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -256,7 +256,7 @@ class 
QAPISchemaGenCommandVisitor(QAPISchemaModularCVisitor):
 prefix, 

Re: [RFC v6 07/11] i386: move TCG cpu class initialization out of helper.c

[Eduardo Habkost]

Now that I understand what you are doing here, I have specific
questions about the functions you are moving, below:

On Thu, Nov 26, 2020 at 11:32:14PM +0100, Claudio Fontana wrote:
> Signed-off-by: Claudio Fontana 
[...]
> @@ -1495,7 +1497,8 @@ static inline uint64_t x86_cpu_xsave_components(X86CPU 
> *cpu)
> cpu->env.features[FEAT_XSAVE_COMP_LO];
>  }
>  
> -const char *get_register_name_32(unsigned int reg)
> +/* Return name of 32-bit register, from a R_* constant */
> +static const char *get_register_name_32(unsigned int reg)
>  {
>  if (reg >= CPU_NB_REGS32) {
>  return NULL;
> @@ -7012,13 +7015,6 @@ static void x86_cpu_set_pc(CPUState *cs, vaddr value)
>  cpu->env.eip = value;
>  }
>  
> -static void x86_cpu_synchronize_from_tb(CPUState *cs, TranslationBlock *tb)
> -{
> -X86CPU *cpu = X86_CPU(cs);
> -
> -cpu->env.eip = tb->pc - tb->cs_base;
> -}

Question to be answered in the commit message: how can somebody
be sure this code is not necessary for any other accelerators?
The TranslationBlock* argument is a hint, but not a guarantee.

Maybe we should rename CPUClass.synchronize_from_tb to
CPUClass.tcg_synchronize_from_tb?  Maybe we should have a
separate TCGCpuOperations struct to carry TCG-specific methods?

(The same questions above apply to the other methods below)


> -
>  int x86_cpu_pending_interrupt(CPUState *cs, int interrupt_request)
>  {
>  X86CPU *cpu = X86_CPU(cs);
> @@ -7252,17 +7248,18 @@ static void x86_cpu_common_class_init(ObjectClass 
> *oc, void *data)
>  cc->class_by_name = x86_cpu_class_by_name;
>  cc->parse_features = x86_cpu_parse_featurestr;
>  cc->has_work = x86_cpu_has_work;
> +
>  #ifdef CONFIG_TCG
> -cc->do_interrupt = x86_cpu_do_interrupt;
> -cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;

These two are in seg_helper.c, so I agree it makes sense to keep
it in tcg_cpu_common_class_init().

I'd like to understand why they are TCG-specific, though.  Are
CPUClass.do_interrupt and CPUClass.cpu_exec_enter TCG-specific on
all architectures, or only in x86?

> -#endif
> +tcg_cpu_common_class_init(cc);
> +#endif /* CONFIG_TCG */
> +
>  cc->dump_state = x86_cpu_dump_state;
>  cc->set_pc = x86_cpu_set_pc;
> -cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
>  cc->gdb_read_register = x86_cpu_gdb_read_register;
>  cc->gdb_write_register = x86_cpu_gdb_write_register;
>  cc->get_arch_id = x86_cpu_get_arch_id;
>  cc->get_paging_enabled = x86_cpu_get_paging_enabled;
> +
>  #ifndef CONFIG_USER_ONLY
>  cc->asidx_from_attrs = x86_asidx_from_attrs;
>  cc->get_memory_mapping = x86_cpu_get_memory_mapping;
> @@ -7273,7 +7270,8 @@ static void x86_cpu_common_class_init(ObjectClass *oc, 
> void *data)
>  cc->write_elf32_note = x86_cpu_write_elf32_note;
>  cc->write_elf32_qemunote = x86_cpu_write_elf32_qemunote;
>  cc->vmsd = _x86_cpu;
> -#endif
> +#endif /* !CONFIG_USER_ONLY */
> +
>  cc->gdb_arch_name = x86_gdb_arch_name;
>  #ifdef TARGET_X86_64
>  cc->gdb_core_xml_file = "i386-64bit.xml";
> @@ -7281,15 +7279,6 @@ static void x86_cpu_common_class_init(ObjectClass *oc, 
> void *data)
>  #else
>  cc->gdb_core_xml_file = "i386-32bit.xml";
>  cc->gdb_num_core_regs = 50;
> -#endif
> -#if defined(CONFIG_TCG) && !defined(CONFIG_USER_ONLY)
> -cc->debug_excp_handler = breakpoint_handler;

That's in bpt_helper.c, also TCG-specific.  Makes sense to move
it to tcg_cpu_common_class_init().

Is CPUClass.debug_excp_handler() TCG-specific in all
architectures, or only in x86?

> -#endif
> -cc->cpu_exec_enter = x86_cpu_exec_enter;
> -cc->cpu_exec_exit = x86_cpu_exec_exit;

I have a question about those two functions below[1].

> -#ifdef CONFIG_TCG
> -cc->tcg_initialize = tcg_x86_init;

The name makes this is obviously TCG-specific, so it makes sense
to move it to tcg_cpu_common_class_init().

> -cc->tlb_fill = x86_cpu_tlb_fill;

This is in excp_helper.c (TCG-specific), so it makes sense to
move it to tcg_cpu_common_class_init().

Is CPUClass.tlb_fill TCG-specific in all architectures, or only
in x86?

>  #endif
>  cc->disas_set_info = x86_disas_set_info;
>  
[...]
> -/* Frob eflags into and out of the CPU temporary format.  */
> -
> -void x86_cpu_exec_enter(CPUState *cs)
> -{
> -X86CPU *cpu = X86_CPU(cs);
> -CPUX86State *env = >env;
> -
> -CC_SRC = env->eflags & (CC_O | CC_S | CC_Z | CC_A | CC_P | CC_C);
> -env->df = 1 - (2 * ((env->eflags >> 10) & 1));
> -CC_OP = CC_OP_EFLAGS;
> -env->eflags &= ~(DF_MASK | CC_O | CC_S | CC_Z | CC_A | CC_P | CC_C);
> -}
> -
> -void x86_cpu_exec_exit(CPUState *cs)
> -{
> -X86CPU *cpu = X86_CPU(cs);
> -CPUX86State *env = >env;
> -
> -env->eflags = cpu_compute_eflags(env);
> -}

[1]

How exactly can we be 100% sure this is not used by other
accelerators?

> -
>  #ifndef CONFIG_USER_ONLY
>  uint8_t x86_ldub_phys(CPUState *cs, hwaddr addr)
>  {
[...]

-- 
Eduardo

[PATCH 4/5] standard-headers: virtio-gpio protocol headers

[Enrico Weigelt, metux IT consult]

Introduce virtio-gpio protocol headers from Linux kernel.
(work in progress, not mainlined yet)

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 include/standard-headers/linux/virtio_gpio.h | 39 
 include/standard-headers/linux/virtio_ids.h  |  1 +
 2 files changed, 40 insertions(+)
 create mode 100644 include/standard-headers/linux/virtio_gpio.h

diff --git a/include/standard-headers/linux/virtio_gpio.h 
b/include/standard-headers/linux/virtio_gpio.h
new file mode 100644
index 00..d1db0ef1fe
--- /dev/null
+++ b/include/standard-headers/linux/virtio_gpio.h
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: GPL-2.0+
+
+#ifndef _LINUX_VIRTIO_GPIO_H
+#define _LINUX_VIRTIO_GPIO_H
+
+#include 
+
+enum virtio_gpio_event_type {
+   // requests from quest to host
+   VIRTIO_GPIO_EV_GUEST_REQUEST= 0x01, // ->request()
+   VIRTIO_GPIO_EV_GUEST_DIRECTION_INPUT= 0x02, // ->direction_input()
+   VIRTIO_GPIO_EV_GUEST_DIRECTION_OUTPUT   = 0x03, // ->direction_output()
+   VIRTIO_GPIO_EV_GUEST_GET_DIRECTION  = 0x04, // ->get_direction()
+   VIRTIO_GPIO_EV_GUEST_GET_VALUE  = 0x05, // ->get_value()
+   VIRTIO_GPIO_EV_GUEST_SET_VALUE  = 0x06, // ->set_value()
+
+   // messages from host to guest
+   VIRTIO_GPIO_EV_HOST_LEVEL   = 0x11, // gpio state changed
+
+   /* mask bit set on host->guest reply */
+   VIRTIO_GPIO_EV_REPLY= 0xF000,
+};
+
+struct virtio_gpio_config {
+   __u8version;
+   __u8reserved0;
+   __u16   num_gpios;
+   __u32   names_size;
+   __u8reserved1[24];
+   __u8name[32];
+};
+
+struct virtio_gpio_event {
+   __le16 type;
+   __le16 pin;
+   __le32 value;
+};
+
+#endif /* _LINUX_VIRTIO_GPIO_H */
diff --git a/include/standard-headers/linux/virtio_ids.h 
b/include/standard-headers/linux/virtio_ids.h
index b052355ac7..053fe59c73 100644
--- a/include/standard-headers/linux/virtio_ids.h
+++ b/include/standard-headers/linux/virtio_ids.h
@@ -48,5 +48,6 @@
 #define VIRTIO_ID_FS   26 /* virtio filesystem */
 #define VIRTIO_ID_PMEM 27 /* virtio pmem */
 #define VIRTIO_ID_MAC80211_HWSIM 29 /* virtio mac80211-hwsim */
+#define VIRTIO_ID_GPIO  30 /* virtio GPIO */
 
 #endif /* _LINUX_VIRTIO_IDS_H */
-- 
2.11.0

[PATCH 5/5] hw: virtio: add virtio-gpio device emulation

[Enrico Weigelt, metux IT consult]

Adding a driver for virtio-based GPIOs. The driver connects to
specified gpio backend and routes all requests there.

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 MAINTAINERS |   7 +
 hw/virtio/Kconfig   |   7 +
 hw/virtio/meson.build   |   1 +
 hw/virtio/virtio-gpio.c | 371 
 4 files changed, 386 insertions(+)
 create mode 100644 hw/virtio/virtio-gpio.c

diff --git a/MAINTAINERS b/MAINTAINERS
index d3873121e2..57deed6c20 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2294,6 +2294,13 @@ S: Supported
 F: backends/gpio.c
 F: backends/gpio-builtin.c
 F: include/sysemu/gpio.h
+F: include/standard-headers/linux/virtio_gpio.h
+
+GPIO Virtio backend
+M: Enrico Weigelt, metux IT consult 
+S: Supported
+F: hw/virtio/virtio-gpio.c
+F: include/hw/virtio/virtio-gpio.h
 
 Memory API
 M: Paolo Bonzini 
diff --git a/hw/virtio/Kconfig b/hw/virtio/Kconfig
index 0eda25c4e1..81da1ee763 100644
--- a/hw/virtio/Kconfig
+++ b/hw/virtio/Kconfig
@@ -33,6 +33,13 @@ config VIRTIO_BALLOON
 default y
 depends on VIRTIO
 
+config VIRTIO_GPIO
+bool
+default y
+depends on VIRTIO
+select BACKEND_GPIO
+select BACKEND_GPIO_BUILTIN
+
 config VIRTIO_CRYPTO
 bool
 default y
diff --git a/hw/virtio/meson.build b/hw/virtio/meson.build
index fbff9bc9d4..88577ff812 100644
--- a/hw/virtio/meson.build
+++ b/hw/virtio/meson.build
@@ -25,6 +25,7 @@ virtio_ss.add(when: 'CONFIG_VHOST_USER_VSOCK', if_true: 
files('vhost-user-vsock.
 virtio_ss.add(when: 'CONFIG_VIRTIO_RNG', if_true: files('virtio-rng.c'))
 virtio_ss.add(when: 'CONFIG_VIRTIO_IOMMU', if_true: files('virtio-iommu.c'))
 virtio_ss.add(when: 'CONFIG_VIRTIO_MEM', if_true: files('virtio-mem.c'))
+virtio_ss.add(when: 'CONFIG_VIRTIO_GPIO', if_true: files('virtio-gpio.c'))
 
 virtio_pci_ss = ss.source_set()
 virtio_pci_ss.add(when: 'CONFIG_VHOST_VSOCK', if_true: 
files('vhost-vsock-pci.c'))
diff --git a/hw/virtio/virtio-gpio.c b/hw/virtio/virtio-gpio.c
new file mode 100644
index 00..37e7614c96
--- /dev/null
+++ b/hw/virtio/virtio-gpio.c
@@ -0,0 +1,371 @@
+/*
+ * A virtio device implementing a hardware gpio port.
+ *
+ * Copyright 2020 Enrico Weigelt, metux IT consult 
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/iov.h"
+#include "qemu/module.h"
+#include "qemu/timer.h"
+#include "hw/virtio/virtio.h"
+#include "hw/qdev-properties.h"
+#include "sysemu/gpio.h"
+#include "sysemu/runstate.h"
+#include "qom/object.h"
+#include "qom/object_interfaces.h"
+#include "trace.h"
+#include "qemu/error-report.h"
+#include "standard-headers/linux/virtio_ids.h"
+#include "standard-headers/linux/virtio_gpio.h"
+
+#define WARN(...) warn_report("virtio-gpio: " __VA_ARGS__)
+
+#define TYPE_VIRTIO_GPIO "virtio-gpio-device"
+OBJECT_DECLARE_SIMPLE_TYPE(VirtIOGPIO, VIRTIO_GPIO)
+#define VIRTIO_GPIO_GET_PARENT_CLASS(obj) \
+OBJECT_GET_PARENT_CLASS(obj, TYPE_VIRTIO_GPIO)
+
+typedef struct VirtIOGPIO VirtIOGPIO;
+
+struct VirtIOGPIO {
+VirtIODevice parent_obj;
+
+VirtQueue *vq_in;
+VirtQueue *vq_out;
+
+uint32_t num_gpios;
+
+char **gpio_names;
+uint32_t gpio_names_len;
+
+GpioBackend *gpio;
+char *name;
+
+VMChangeStateEntry *vmstate;
+struct virtio_gpio_event reply_buffer;
+
+void *config_buf;
+int config_len;
+};
+
+static bool is_guest_ready(VirtIOGPIO *vgpio)
+{
+VirtIODevice *vdev = VIRTIO_DEVICE(vgpio);
+if (virtio_queue_ready(vgpio->vq_in)
+&& (vdev->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
+return true;
+}
+return false;
+}
+
+static void virtio_gpio_reply(VirtIOGPIO *vgpio, int type, int pin, int value)
+{
+VirtQueueElement *elem;
+size_t len;
+
+if (!virtio_queue_ready(vgpio->vq_out)) {
+WARN("out queue is not ready yet");
+return;
+}
+
+elem = virtqueue_pop(vgpio->vq_out, sizeof(VirtQueueElement));
+if (!elem) {
+WARN("failed to get xmit queue element");
+return;
+}
+
+vgpio->reply_buffer.type = type;
+vgpio->reply_buffer.pin = pin;
+vgpio->reply_buffer.value = value;
+len = iov_from_buf(elem->in_sg, elem->in_num, 0, >reply_buffer,
+   sizeof(struct virtio_gpio_event));
+virtqueue_push(vgpio->vq_out, elem, len);
+g_free(elem);
+virtio_notify(VIRTIO_DEVICE(vgpio), vgpio->vq_out);
+}
+
+static int do_request(VirtIOGPIO *vgpio, struct virtio_gpio_event *reqbuf)
+{
+switch (reqbuf->type) {
+case VIRTIO_GPIO_EV_GUEST_REQUEST:
+return gpio_backend_request(vgpio->gpio, reqbuf->pin);
+case VIRTIO_GPIO_EV_GUEST_DIRECTION_INPUT:
+return gpio_backend_direction_input(vgpio->gpio, reqbuf->pin);
+case VIRTIO_GPIO_EV_GUEST_DIRECTION_OUTPUT:
+return gpio_backend_direction_output(vgpio->gpio, reqbuf->pin,
+ reqbuf->value);
+case VIRTIO_GPIO_EV_GUEST_GET_DIRECTION:

[PATCH] fix .gitignore

[Enrico Weigelt, metux IT consult]

Some entries for autogenerated files have been missing in .gitignore.

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 .gitignore | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/.gitignore b/.gitignore
index b32bca1315..48ec051c2e 100644
--- a/.gitignore
+++ b/.gitignore
@@ -10,3 +10,5 @@ TAGS
 *~
 *.ast_raw
 *.depend_raw
+config-temp/
+config.log
-- 
2.11.0

[PATCH 1/5] scripts: minikconf: support config titles

[Enrico Weigelt, metux IT consult]

Add support for config option titles, like the real kconfig does.
Even though they're not presented anywhere yet (since minikconf
only acts in the background), it's good to have them supported,
so we can start adding descriptions in the Kconfig files.

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 scripts/minikconf.py | 14 ++
 1 file changed, 14 insertions(+)

diff --git a/scripts/minikconf.py b/scripts/minikconf.py
index bcd91015d3..28c67906cb 100644
--- a/scripts/minikconf.py
+++ b/scripts/minikconf.py
@@ -206,6 +206,7 @@ class KconfigData:
 self.defined_vars = set()
 self.referenced_vars = dict()
 self.clauses = list()
+self.title = None
 
 # semantic analysis -
 
@@ -290,6 +291,9 @@ class KconfigData:
 cond = (cond & var) if cond is not None else var
 self.clauses.append(KconfigData.SelectClause(symbol, cond))
 
+def do_title(self, title):
+self.title = title
+
 def do_imply(self, var, symbol, cond=None):
 # "config X imply Y [if COND]" is the same as
 # "config Y default y if X [&& COND]"
@@ -323,6 +327,7 @@ TOK_BOOL = 15;TOKENS[TOK_BOOL] = '"bool"';
 TOK_IF = 16;  TOKENS[TOK_IF] = '"if"';
 TOK_ID = 17;  TOKENS[TOK_ID] = 'identifier';
 TOK_EOF = 18; TOKENS[TOK_EOF] = 'end of file';
+TOK_QUOTED = 19;  TOKENS[TOK_QUOTED] = 'quoted string';
 
 class KconfigParserError(Exception):
 def __init__(self, parser, msg, tok=None):
@@ -501,6 +506,7 @@ class KconfigParser:
 # property: DEFAULT y_or_n condition
 #   | DEPENDS ON expr
 #   | SELECT var condition
+#   | BOOL "comment"
 #   | BOOL
 def parse_property(self, var):
 if self.tok == TOK_DEFAULT:
@@ -526,6 +532,9 @@ class KconfigParser:
 self.data.do_imply(var, symbol, cond)
 elif self.tok == TOK_BOOL:
 self.get_token()
+if self.tok == TOK_QUOTED:
+self.data.do_title(self.val)
+self.get_token()
 else:
 raise KconfigParserError(self, 'Error in recursive descent?')
 
@@ -645,6 +654,11 @@ class KconfigParser:
 self.cursor = self.src.find('\n', self.cursor)
 self.val = self.src[start:self.cursor]
 return TOK_SOURCE
+elif self.tok == '"':
+start = self.cursor
+self.cursor = self.src.find('"', self.cursor)+1
+self.val = self.src[start:self.cursor]
+return TOK_QUOTED;
 elif self.tok.isalnum():
 # identifier
 while self.src[self.cursor].isalnum() or self.src[self.cursor] == 
'_':
-- 
2.11.0

[PATCH 2/5] backends: introduce gpio backend subsystem

[Enrico Weigelt, metux IT consult]

Introducing a backend subsys for hardware GPIOs, so we can now
let simulated GPIO devices actually do something, by talking to
pluggable GPIO backends (eg. speaking to host's gpio subsystem,
gpiod, custom simulation, etc).

This patch does not implement any actual gpio backend, nor any
any any hw simulation drivers yet - just the generic infrastructure.

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 MAINTAINERS   |   6 ++
 backends/Kconfig  |   4 +
 backends/gpio.c   | 255 ++
 backends/meson.build  |   1 +
 include/sysemu/gpio.h |  76 +++
 5 files changed, 342 insertions(+)
 create mode 100644 backends/gpio.c
 create mode 100644 include/sysemu/gpio.h

diff --git a/MAINTAINERS b/MAINTAINERS
index 68bc160f41..bfa29a4560 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2288,6 +2288,12 @@ S: Maintained
 F: gdbstub*
 F: gdb-xml/
 
+GPIO Backend API
+M: Enrico Weigelt, metux IT consult 
+S: Supported
+F: backends/gpio.c
+F: include/sysemu/gpio.h
+
 Memory API
 M: Paolo Bonzini 
 S: Supported
diff --git a/backends/Kconfig b/backends/Kconfig
index f35abc1609..2f17189472 100644
--- a/backends/Kconfig
+++ b/backends/Kconfig
@@ -1 +1,5 @@
 source tpm/Kconfig
+
+config BACKEND_GPIO
+bool "Enable GPIO backends"
+default y
diff --git a/backends/gpio.c b/backends/gpio.c
new file mode 100644
index 00..dc539b0791
--- /dev/null
+++ b/backends/gpio.c
@@ -0,0 +1,255 @@
+/*
+ * QEMU GPIO Backend
+ *
+ * Copyright 2020 Enrico Weigelt, metux IT consult 
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include 
+#include "qemu/osdep.h"
+#include "sysemu/gpio.h"
+#include "qapi/error.h"
+#include "qapi/qmp/qerror.h"
+#include "qemu/module.h"
+#include "qom/object_interfaces.h"
+#include "qemu/error-report.h"
+#include "sysemu/gpio.h"
+
+#define GPIO_FORMAT_VALUE   "gpio%d.value"
+#define GPIO_FORMAT_DIRECTION   "gpio%d.direction"
+
+#define BACKEND_OP_HEAD \
+if (!gpio) \
+return -EFAULT;
+
+#define BACKEND_CLASSOP_HEAD(op) \
+GpioBackendClass *klass; \
+BACKEND_OP_HEAD \
+klass = GPIO_BACKEND_GET_CLASS(gpio); \
+if (!klass) \
+return -EFAULT; \
+if (!klass->op) \
+return -EUNATCH;
+
+int gpio_backend_set_notify(GpioBackend *gpio,
+gpio_backend_notify_t proc,
+void *consumer)
+{
+BACKEND_OP_HEAD
+
+gpio->notify_proc = proc;
+gpio->notify_consumer = consumer;
+
+return 0;
+}
+
+int gpio_backend_send_notify(GpioBackend *gpio, int pin, int event, int value)
+{
+BACKEND_OP_HEAD
+
+if (gpio->notify_proc) {
+return gpio->notify_proc(gpio->notify_consumer, pin, event, value);
+}
+
+return 0;
+}
+
+int gpio_backend_request(GpioBackend *gpio, int pin)
+{
+BACKEND_CLASSOP_HEAD(request);
+return klass->request(gpio, pin);
+}
+
+int gpio_backend_set_value(GpioBackend *gpio, int pin, int state)
+{
+BACKEND_CLASSOP_HEAD(set_value);
+return klass->set_value(gpio, pin, state);
+}
+
+int gpio_backend_get_value(GpioBackend *gpio, int pin)
+{
+BACKEND_CLASSOP_HEAD(get_value);
+return klass->get_value(gpio, pin);
+}
+
+int gpio_backend_direction_output(GpioBackend *gpio, int pin, int state)
+{
+BACKEND_CLASSOP_HEAD(direction_output);
+return klass->direction_output(gpio, pin, state);
+}
+
+int gpio_backend_direction_input(GpioBackend *gpio, int pin)
+{
+BACKEND_CLASSOP_HEAD(direction_input);
+return klass->direction_input(gpio, pin);
+}
+
+int gpio_backend_get_direction(GpioBackend *gpio, int pin)
+{
+BACKEND_CLASSOP_HEAD(get_direction);
+return klass->get_direction(gpio, pin);
+}
+
+int gpio_backend_get_ngpio(GpioBackend *gpio)
+{
+BACKEND_CLASSOP_HEAD(get_ngpio);
+return klass->get_ngpio(gpio);
+}
+
+static void getattr_value(Object *obj, Visitor *v, const char *name,
+  void *opaque, Error **errp)
+{
+int pin;
+int64_t val = 0;
+GpioBackend *gpio = GPIO_BACKEND(obj);
+
+if (sscanf(name, GPIO_FORMAT_VALUE, ) != 1) {
+error_setg(errp,
+  "gpio: getattr_value() illegal property: \"%s\"",
+   name);
+return;
+}
+
+val = gpio_backend_get_value(gpio, pin);
+visit_type_int(v, name, , errp);
+}
+
+static void setattr_value(Object *obj, Visitor *v, const char *name,
+  void *opaque, Error **errp)
+{
+int pin;
+int64_t val = 0;
+GpioBackend *gpio = GPIO_BACKEND(obj);
+
+if (!visit_type_int(v, name, , errp)) {
+return;
+}
+
+if (sscanf(name, GPIO_FORMAT_VALUE, ) != 1) {
+error_setg(errp,
+   "gpio: setattr_value() illegal property: \"%s\"",
+   name);
+return;
+}
+
+gpio_backend_set_value(gpio, pin, val);
+gpio_backend_send_notify(gpio, pin, GPIO_EVENT_LEVEL, val);
+}
+
+static void getattr_direction(Object *obj, Visitor *v, const char 

[PATCH 3/5] backends: gpio: dummy builtin backend

[Enrico Weigelt, metux IT consult]

Adding a dummy GPIO backend driver. Essentially stores the states
in memory and gives some debug output. The current state can be
accessed as a string property.

Signed-off-by: Enrico Weigelt, metux IT consult 
---
 MAINTAINERS |   1 +
 backends/Kconfig|   5 ++
 backends/gpio-builtin.c | 137 
 backends/meson.build|   1 +
 include/sysemu/gpio.h   |   2 +
 5 files changed, 146 insertions(+)
 create mode 100644 backends/gpio-builtin.c

diff --git a/MAINTAINERS b/MAINTAINERS
index bfa29a4560..d3873121e2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2292,6 +2292,7 @@ GPIO Backend API
 M: Enrico Weigelt, metux IT consult 
 S: Supported
 F: backends/gpio.c
+F: backends/gpio-builtin.c
 F: include/sysemu/gpio.h
 
 Memory API
diff --git a/backends/Kconfig b/backends/Kconfig
index 2f17189472..1c8a462b57 100644
--- a/backends/Kconfig
+++ b/backends/Kconfig
@@ -3,3 +3,8 @@ source tpm/Kconfig
 config BACKEND_GPIO
 bool "Enable GPIO backends"
 default y
+
+config BACKEND_GPIO_BUILTIN
+bool "Dummy GPIO backend"
+depends on BACKEND_GPIO
+default y
diff --git a/backends/gpio-builtin.c b/backends/gpio-builtin.c
new file mode 100644
index 00..ac89a88092
--- /dev/null
+++ b/backends/gpio-builtin.c
@@ -0,0 +1,137 @@
+/*
+ * QEMU GPIO Backend - builtin (dummy)
+ *
+ * Copyright 2020 Enrico Weigelt, metux IT consult 
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "sysemu/gpio.h"
+#include "qemu/main-loop.h"
+#include "qemu/guest-random.h"
+#include "qom/object.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "qapi/visitor.h"
+
+#define MAX_GPIO256
+
+#define WARN(...)   warn_report("gpio-builtin: " __VA_ARGS__)
+
+#define OP_HEAD(name) \
+GpioBuiltin *gpio = GPIO_BUILTIN(obj); \
+if (id >= gpio->num_gpio) { \
+WARN("%s: gpio id %d out of range", name, id); \
+return -ERANGE; \
+}
+
+#define FLAG_DIRECTION_INPUT1
+#define FLAG_LINE_ACTIVE2
+
+OBJECT_DECLARE_SIMPLE_TYPE(GpioBuiltin, GPIO_BUILTIN)
+
+struct GpioBuiltin {
+GpioBackend parent;
+char *states;
+int num_gpio;
+};
+
+static int gpio_builtin_request(GpioBackend *obj, int id)
+{
+OP_HEAD("request");
+return 0;
+}
+
+static int gpio_builtin_set_value(GpioBackend *obj, int id, int state)
+{
+OP_HEAD("set");
+if (state & QEMU_GPIO_LINE_ACTIVE) {
+gpio->states[id] |= FLAG_LINE_ACTIVE;
+} else {
+gpio->states[id] &= ~FLAG_LINE_ACTIVE;
+}
+return 0;
+}
+
+static int gpio_builtin_direction_input(GpioBackend *obj, int id)
+{
+OP_HEAD("direction-input");
+gpio->states[id] |= FLAG_DIRECTION_INPUT;
+return gpio_builtin_set_value(obj, id, 0);
+}
+
+static int gpio_builtin_direction_output(GpioBackend *obj, int id, int state)
+{
+OP_HEAD("direction-output");
+gpio->states[id] &= ~FLAG_DIRECTION_INPUT;
+return gpio_builtin_set_value(obj, id, state);
+}
+
+static int gpio_builtin_get_direction(GpioBackend *obj, int id)
+{
+OP_HEAD("get-direction");
+return (gpio->states[id] & FLAG_DIRECTION_INPUT ?
+QEMU_GPIO_DIRECTION_INPUT : QEMU_GPIO_DIRECTION_OUTPUT);
+}
+
+static int gpio_builtin_get_value(GpioBackend *obj, int id)
+{
+OP_HEAD("get");
+return (gpio->states[id] & FLAG_LINE_ACTIVE ?
+QEMU_GPIO_LINE_ACTIVE : QEMU_GPIO_LINE_INACTIVE);
+}
+
+static void gpio_builtin_instance_init(Object *obj)
+{
+GpioBuiltin *gpio = GPIO_BUILTIN(obj);
+
+gpio->num_gpio = MAX_GPIO;
+gpio->states = g_malloc(gpio->num_gpio + 1);
+memset(gpio->states, 'i', gpio->num_gpio);
+gpio->states[gpio->num_gpio] = 0;
+gpio_backend_register(>parent);
+}
+
+static void gpio_builtin_instance_finalize(Object *obj)
+{
+GpioBuiltin *gpio = GPIO_BUILTIN(obj);
+gpio_backend_unregister(>parent);
+g_free(gpio->states);
+}
+
+static int gpio_builtin_get_ngpio(GpioBackend *obj)
+{
+GpioBuiltin *gpio = GPIO_BUILTIN(obj);
+return gpio->num_gpio;
+}
+
+static void gpio_builtin_class_init(ObjectClass *klass, void *data)
+{
+GpioBackendClass *gpio = GPIO_BACKEND_CLASS(klass);
+
+gpio->name = g_strdup("gpio-builtin");
+gpio->get_value= gpio_builtin_get_value;
+gpio->set_value= gpio_builtin_set_value;
+gpio->get_direction= gpio_builtin_get_direction;
+gpio->direction_input  = gpio_builtin_direction_input;
+gpio->direction_output = gpio_builtin_direction_output;
+gpio->request  = gpio_builtin_request;
+gpio->get_ngpio= gpio_builtin_get_ngpio;
+}
+
+static const TypeInfo gpio_builtin_info = {
+.name = TYPE_GPIO_BUILTIN,
+.parent = TYPE_GPIO_BACKEND,
+.instance_size = sizeof(GpioBuiltin),
+.instance_init = gpio_builtin_instance_init,
+.instance_finalize = gpio_builtin_instance_finalize,
+.class_init = gpio_builtin_class_init,
+};
+
+static 

Re: [PULL 0/1] Libslirp CVE-2020-29129 CVE-2020-29130

[Peter Maydell]

On Fri, 27 Nov 2020 at 17:08,  wrote:
>
> From: Marc-André Lureau 
>
> The following changes since commit ea8208249d1082eae0444934efb3b59cd3183f05:
>
>   Merge remote-tracking branch 
> 'remotes/kraxel/tags/fixes-20201127-pull-request' into staging (2020-11-27 
> 11:11:43 +)
>
> are available in the Git repository at:
>
>   g...@github.com:elmarco/qemu.git tags/libslirp-pull-request
>
> for you to fetch changes up to 37c0c885d19a4c2d69faed891b5c02aaffbdccfb:
>
>   slirp: update to fix CVE-2020-29129 CVE-2020-29130 (2020-11-27 20:57:11 
> +0400)
>
> 
>
> 
>
> Marc-André Lureau (1):
>   slirp: update to fix CVE-2020-29129 CVE-2020-29130


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.2
for any user-visible changes.

-- PMM

Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Philippe Mathieu-Daudé]

On 11/27/20 7:29 PM, Thomas Huth wrote:
> On 27/11/2020 18.57, Philippe Mathieu-Daudé wrote:
>> On 11/27/20 6:47 PM, Thomas Huth wrote:
>>> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
 We lately realized that the Avocado framework was not designed
 to be regularly run on CI environments. Therefore, as of 5.2
 we deprecate the gitlab-ci jobs using Avocado. To not disrupt
 current users, it is possible to keep the current behavior by
 setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
 (see [*]).
 From now on, using these jobs (or adding new tests to them)
 is strongly discouraged.

 Tests based on Avocado will be ported to new job schemes during
 the next releases, with better documentation and templates.
>>>
>>> Why should we disable the jobs by default as long as there is no replacement
>>> available yet?
>>
>> Why keep it enabled if it is failing randomly
> 
> We can still disable single jobs if they are not stable, but that's no
> reason to disable all of them by default, is it?
> 
>> if images hardcoded
>> in tests are being removed from public servers, etc...?
> 
> That's independent from Avocado, you'll always have that problem if you want
> to test with external images, unless you mirror them into a repository on
> the same server (ie. gitlab), which, however, might not always be possible...
> 
>> They are not disabled, they are still runnable manually or setting
>> QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE...
> 
> And who do you think is going to set that variable? Hardly anybody, I guess.

Does that mean nobody cares about these tests?

> So you could also simply remove the stuff from the yml file completely 
> instead.

Yes, I'd prefer that too, but Alex objected.

>> We realized by default Avocado runs all tests on the CI jobs,
>> triggering failures and complains. Developer stopped to contribute/
>> review integration tests because of that.
> 
> Did anybody really stop contributing "acceptance" test since they were
> afraid of the gitlab-CI running them? That's new to me, do you have a pointer?

No, but alternatively, how many tests were contributed / reviewed
last year?

>> We want developers be
>> able to contribute tests to the repository fearlessly.
> 
> You can always mark your test with @skipIf(os.getenv('GITLAB_CI')) if you
> don't want to see it running in the gitlab-CI, so that's not a reason to be
> afraid.

This was the idea here (opposite, tag jobs with 'gating-ci' to run
them on GitLab):
https://www.mail-archive.com/qemu-devel@nongnu.org/msg756464.html

> 
>> If we don't change anything, we'll keep having CI failures due
>> to Avocado design issues (artifacts removed from remote servers,
>> etc...).
> 
> I fail to see the relation between Avocado and vanishing artifacts from 3rd
> party servers... what do you plan to do instead if something gets (re-)moved
> from a server that is not under our control?

Avocado tests and CI are orthogonal, but it will be painful to
fix Avocado tests with the current Avocado CI jobs.

> 
>> I haven't seen any attempt on this list to improve the current
>> fragile situation, but better suggestions are certainly welcome.
> 
> At least I am hoping that the "check-acceptance" tests will break a little
> bit less often once Peter uses the gitlab-CI for his gating tests, too. That
> will at least prevent that one of the tests gets completely broken by a new
> merged pull request. Of course there's still the risk that tests only fail
> occasionally due to new bugs, but that can also happen for all other test
> suites (unit, qtest, iotests, ...), too.

Or Peter (as other users) will get grumpy at these tests because they
are unreliable, hard to understand what fail and debug.

Thus the removal suggestion, so we can "fix" the missing Avocado parts
before it is used heavily.

Phil.

Re: [RFC v6 11/11] i386: split cpu accelerators from cpu.c, using AccelCPUClass

[Eduardo Habkost]

On Fri, Nov 27, 2020 at 07:16:25PM +0100, Claudio Fontana wrote:
> On 11/27/20 7:09 PM, Eduardo Habkost wrote:
> > On Fri, Nov 27, 2020 at 06:53:23PM +0100, Claudio Fontana wrote:
> >> On 11/27/20 6:41 PM, Eduardo Habkost wrote:
> >>> On Thu, Nov 26, 2020 at 11:32:18PM +0100, Claudio Fontana wrote:
>  i386 is the first user of AccelCPUClass, allowing to split
>  cpu.c into:
> 
>  cpu.ccpuid and common x86 cpu functionality
>  host-cpu.c   host x86 cpu functions and "host" cpu type
>  kvm/cpu.cKVM x86 AccelCPUClass
>  hvf/cpu.cHVF x86 AccelCPUClass
>  tcg/cpu.cTCG x86 AccelCPUClass
> 
>  Signed-off-by: Claudio Fontana 
>  ---
> >>> [...]
>  +static void tcg_cpu_class_init(CPUClass *cc)
> >>>
> >>> Is this the only case where we need to provide an
> >>> AccelCPUClass.cpu_class_init method?
> >>>
> >>>
>  +{
>  +cc->do_interrupt = x86_cpu_do_interrupt;
>  +cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
>  +cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
>  +cc->cpu_exec_enter = x86_cpu_exec_enter;
>  +cc->cpu_exec_exit = x86_cpu_exec_exit;
>  +cc->tcg_initialize = tcg_x86_init;
>  +cc->tlb_fill = x86_cpu_tlb_fill;
>  +#ifndef CONFIG_USER_ONLY
>  +cc->debug_excp_handler = breakpoint_handler;
>  +#endif /* !CONFIG_USER_ONLY */
> >>>
> >>> I find the need for these method overrides suspicious.
> >>
> >> These mechanisms are preexistent. My refactoring only makes them more 
> >> visible.
> >>
> >>>
> >>> Comparing this with the code on qemu.git master:
> >>>
> >>> static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
> >>> {
> >>> [...]
> >>> #ifdef CONFIG_TCG
> >>> cc->do_interrupt = x86_cpu_do_interrupt;
> >>> cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
> >>> #endif
> >>> [...]
> >>> cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
> >>> [...]
> >>> #if defined(CONFIG_TCG) && !defined(CONFIG_USER_ONLY)
> >>> cc->debug_excp_handler = breakpoint_handler;
> >>> #endif
> >>> cc->cpu_exec_enter = x86_cpu_exec_enter;
> >>> cc->cpu_exec_exit = x86_cpu_exec_exit;
> >>> #ifdef CONFIG_TCG
> >>> cc->tcg_initialize = tcg_x86_init;
> >>> cc->tlb_fill = x86_cpu_tlb_fill;
> >>> #endif
> >>> [...]
> >>> }
> >>>
> >>> So, we have two kinds of CPUClass fields above:
> >>> * Code that was never conditional on CONFIG_TCG, and now is
> >>>   conditional (synchronize_from_tb, cpu_exec_enter,
> >>>   cpu_exec_exit).
> >>
> >>
> >>
> >> synchronize_from_tb, cpu_exec_enter and cpu_exec_exit only makes sense for 
> >> TCG,
> >> and their code should not be compiled in for non-TCG.
> >>
> >> This is part of the effort to separate away non-pertinent code into 
> >> accelerator-specific builds (and in the future modules).
> >>
> >> The fact that they were unconditionally compiled in before was a mistake, 
> >> or at least this is the assumption I am making when changing this.
> > 
> > Can you clarify why that was a mistake?  Patch 07/11 makes
> > existing functions TCG-specific, but doesn't explain why.
> 
> 
> 
> They are functions called only for TCG. You don't need translation blocks and 
> cpu_exec() for other accelerators.
> I can add this information in the commit message.

Thanks!  I believe we can clarify in patch 07/11 why some methods
that were not conditional on CONFIG_TCG were moved to
tcg_cpu_common_class_init().

Do you think it would be too complicated to move those methods
from CPUClass to CpusAccel?  If they are really accel-specific,
CPUClass looks like the wrong place for them.

> [...]

-- 
Eduardo

Re: [PATCH] gitlab-CI: Test 32-bit builds with the fedora-i386-cross container

[Daniel P . Berrangé]

On Fri, Nov 27, 2020 at 07:15:43PM +0100, Thomas Huth wrote:
> On 27/11/2020 19.10, Thomas Huth wrote:
> > After adding some missing packages, it's possible to check 32-bit
> > builds and tests with the fedora-i386-cross container in the gitlab-CI,
> > too.
> > 
> > While we're at it, update it to Fedora 31. Unfortunately the gcc
> > from the later versions emits some very dubious format-truncation
> > warnings, so Fedora 32 and 33 are currently unsuitable for this job.
> > 
> > I also had to remove the gnutls-devel package since linking then
> > failed due to its dependency libtasn1 not being recognized (even
> > if I explicitely installed it into the container).
> 
> In case anybody is interested, the failure can be seen here:
> 
>  https://gitlab.com/huth/qemu/-/jobs/876977206#L4799
> 
> ... which is very strange, since in line 259, meson claimed to have found
> libtasn1 ... anybody got an idea what's going wrong here?

It installed the 32-bit  tasn1 RPM, but the 64-bit tasn1-devel RPM


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Thomas Huth]

On 27/11/2020 18.57, Philippe Mathieu-Daudé wrote:
> On 11/27/20 6:47 PM, Thomas Huth wrote:
>> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
>>> We lately realized that the Avocado framework was not designed
>>> to be regularly run on CI environments. Therefore, as of 5.2
>>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
>>> current users, it is possible to keep the current behavior by
>>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
>>> (see [*]).
>>> From now on, using these jobs (or adding new tests to them)
>>> is strongly discouraged.
>>>
>>> Tests based on Avocado will be ported to new job schemes during
>>> the next releases, with better documentation and templates.
>>
>> Why should we disable the jobs by default as long as there is no replacement
>> available yet?
> 
> Why keep it enabled if it is failing randomly

We can still disable single jobs if they are not stable, but that's no
reason to disable all of them by default, is it?

> if images hardcoded
> in tests are being removed from public servers, etc...?

That's independent from Avocado, you'll always have that problem if you want
to test with external images, unless you mirror them into a repository on
the same server (ie. gitlab), which, however, might not always be possible...

> They are not disabled, they are still runnable manually or setting
> QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE...

And who do you think is going to set that variable? Hardly anybody, I guess.
So you could also simply remove the stuff from the yml file completely instead.

> We realized by default Avocado runs all tests on the CI jobs,
> triggering failures and complains. Developer stopped to contribute/
> review integration tests because of that.

Did anybody really stop contributing "acceptance" test since they were
afraid of the gitlab-CI running them? That's new to me, do you have a pointer?

> We want developers be
> able to contribute tests to the repository fearlessly.

You can always mark your test with @skipIf(os.getenv('GITLAB_CI')) if you
don't want to see it running in the gitlab-CI, so that's not a reason to be
afraid.

> If we don't change anything, we'll keep having CI failures due
> to Avocado design issues (artifacts removed from remote servers,
> etc...).

I fail to see the relation between Avocado and vanishing artifacts from 3rd
party servers... what do you plan to do instead if something gets (re-)moved
from a server that is not under our control?

> I haven't seen any attempt on this list to improve the current
> fragile situation, but better suggestions are certainly welcome.

At least I am hoping that the "check-acceptance" tests will break a little
bit less often once Peter uses the gitlab-CI for his gating tests, too. That
will at least prevent that one of the tests gets completely broken by a new
merged pull request. Of course there's still the risk that tests only fail
occasionally due to new bugs, but that can also happen for all other test
suites (unit, qtest, iotests, ...), too.

 Thomas

[PATCH v3] drivers/virt: vmgenid: add vm generation id driver

[Catangiu, Adrian Costin]

- Background

The VM Generation ID is a feature defined by Microsoft (paper:
http://go.microsoft.com/fwlink/?LinkId=260709) and supported by
multiple hypervisor vendors.

The feature is required in virtualized environments by apps that work
with local copies/caches of world-unique data such as random values,
uuids, monotonically increasing counters, etc.
Such apps can be negatively affected by VM snapshotting when the VM
is either cloned or returned to an earlier point in time.

The VM Generation ID is a simple concept meant to alleviate the issue
by providing a unique ID that changes each time the VM is restored
from a snapshot. The hw provided UUID value can be used to
differentiate between VMs or different generations of the same VM.

- Problem

The VM Generation ID is exposed through an ACPI device by multiple
hypervisor vendors but neither the vendors or upstream Linux have no
default driver for it leaving users to fend for themselves.

Furthermore, simply finding out about a VM generation change is only
the starting point of a process to renew internal states of possibly
multiple applications across the system. This process could benefit
from a driver that provides an interface through which orchestration
can be easily done.

- Solution

This patch is a driver that exposes a monotonic incremental Virtual
Machine Generation u32 counter via a char-dev FS interface. The FS
interface provides sync and async VmGen counter updates notifications.
It also provides VmGen counter retrieval and confirmation mechanisms.

The generation counter and the interface through which it is exposed
are available even when there is no acpi device present.

When the device is present, the hw provided UUID is not exposed to
userspace, it is internally used by the driver to keep accounting for
the exposed VmGen counter. The counter starts from zero when the
driver is initialized and monotonically increments every time the hw
UUID changes (the VM generation changes).
On each hw UUID change, the new hypervisor-provided UUID is also fed
to the kernel RNG.

If there is no acpi vmgenid device present, the generation changes are
not driven by hw vmgenid events but can be driven by software through
a dedicated driver ioctl.

This patch builds on top of Or Idgar 's proposal
https://lkml.org/lkml/2018/3/1/498

- Future improvements

Ideally we would want the driver to register itself based on devices'
_CID and not _HID, but unfortunately I couldn't find a way to do that.
The problem is that ACPI device matching is done by
'__acpi_match_device()' which exclusively looks at
'acpi_hardware_id *hwid'.

There is a path for platform devices to match on _CID when _HID is
'PRP0001' - but this is not the case for the Qemu vmgenid device.

Guidance and help here would be greatly appreciated.

Signed-off-by: Adrian Catangiu 

---

v1 -> v2:

  - expose to userspace a monotonically increasing u32 Vm Gen Counter
    instead of the hw VmGen UUID
  - since the hw/hypervisor-provided 128-bit UUID is not public
    anymore, add it to the kernel RNG as device randomness
  - insert driver page containing Vm Gen Counter in the user vma in
    the driver's mmap handler instead of using a fault handler
  - turn driver into a misc device driver to auto-create /dev/vmgenid
  - change ioctl arg to avoid leaking kernel structs to userspace
  - update documentation
  - various nits
  - rebase on top of linus latest

v2 -> v3:

  - separate the core driver logic and interface, from the ACPI device.
    The ACPI vmgenid device is now one possible backend.
  - fix issue when timeout=0 in VMGENID_WAIT_WATCHERS
  - add locking to avoid races between fs ops handlers and hw irq
    driven generation updates
  - change VMGENID_WAIT_WATCHERS ioctl so if the current caller is
    outdated or a generation change happens while waiting (thus making
    current caller outdated), the ioctl returns -EINTR to signal the
    user to handle event and retry. Fixes blocking on oneself.
  - add VMGENID_FORCE_GEN_UPDATE ioctl conditioned by
    CAP_CHECKPOINT_RESTORE capability, through which software can force
    generation bump.
---
 Documentation/virt/vmgenid.rst | 240 +++
 drivers/virt/Kconfig   |  17 ++
 drivers/virt/Makefile  |   1 +
 drivers/virt/vmgenid.c | 435
+
 include/uapi/linux/vmgenid.h   |  14 ++
 5 files changed, 707 insertions(+)
 create mode 100644 Documentation/virt/vmgenid.rst
 create mode 100644 drivers/virt/vmgenid.c
 create mode 100644 include/uapi/linux/vmgenid.h

diff --git a/Documentation/virt/vmgenid.rst b/Documentation/virt/vmgenid.rst
new file mode 100644
index 000..b6a9f8d
--- /dev/null
+++ b/Documentation/virt/vmgenid.rst
@@ -0,0 +1,240 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+
+VMGENID
+
+
+The VM Generation ID is a feature defined by Microsoft (paper:
+http://go.microsoft.com/fwlink/?LinkId=260709) and supported by
+multiple hypervisor 

Re: [PATCH] gitlab-CI: Test 32-bit builds with the fedora-i386-cross container

[Paolo Bonzini]

On 27/11/20 19:15, Thomas Huth wrote:

On 27/11/2020 19.10, Thomas Huth wrote:

After adding some missing packages, it's possible to check 32-bit
builds and tests with the fedora-i386-cross container in the gitlab-CI,
too.

While we're at it, update it to Fedora 31. Unfortunately the gcc
from the later versions emits some very dubious format-truncation
warnings, so Fedora 32 and 33 are currently unsuitable for this job.

I also had to remove the gnutls-devel package since linking then
failed due to its dependency libtasn1 not being recognized (even
if I explicitely installed it into the container).


In case anybody is interested, the failure can be seen here:

  https://gitlab.com/huth/qemu/-/jobs/876977206#L4799

... which is very strange, since in line 259, meson claimed to have found
libtasn1 ... anybody got an idea what's going wrong here?


You didn't install libtasn1-devel.i686.  So it looks like pkg-config 
could find an (x86_64) tasn1.pc was found, but neither of 
/usr/lib/libtasn1.{a,so} exists.


In general this points out an issue with the setup, in that pkg-config 
falls back to /usr/lib64/pkg-config.  Looking at the man page, it seems 
like you have to specify PKG_CONFIG_LIBDIR instead of PKG_CONFIG_PATH.


Paolo

Re: [PATCH v2] drivers/virt: vmgenid: add vm generation id driver

[Jann Horn]

[resend in the hope that amazon will accept my mail this time instead
of replying "550 Too many invalid recipients" again]

On Fri, Nov 20, 2020 at 11:29 PM Jann Horn  wrote:
> On Mon, Nov 16, 2020 at 4:35 PM Catangiu, Adrian Costin
>  wrote:
> > This patch is a driver that exposes a monotonic incremental Virtual
> > Machine Generation u32 counter via a char-dev FS interface that
> > provides sync and async VmGen counter updates notifications. It also
> > provides VmGen counter retrieval and confirmation mechanisms.
> >
> > The hw provided UUID is not exposed to userspace, it is internally
> > used by the driver to keep accounting for the exposed VmGen counter.
> > The counter starts from zero when the driver is initialized and
> > monotonically increments every time the hw UUID changes (the VM
> > generation changes).
> >
> > On each hw UUID change, the new hypervisor-provided UUID is also fed
> > to the kernel RNG.
>
> As for v1:
>
> Is there a reasonable usecase for the "confirmation" mechanism? It
> doesn't seem very useful to me.
>
> How do you envision integrating this with libraries that have to work
> in restrictive seccomp sandboxes? If this was in the vDSO, that would
> be much easier.

Re: [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework

[Eduardo Habkost]

On Fri, Nov 27, 2020 at 06:16:27PM +, Peter Maydell wrote:
> On Fri, 27 Nov 2020 at 17:18, Eduardo Habkost  wrote:
> > Thanks!  Is the data returned by kvm_arm_get_host_cpu_features()
> > supposed to eventually affect the value of id_aa64mmfr0?  I don't
> > see how that could happen.
> 
> kvm_arm_get_host_cpu_features() does:
> err |= read_sys_reg64(fdarray[2], >isar.id_aa64mmfr0,
>   ARM64_SYS_REG(3, 0, 0, 7, 0));
> 
> which is filling in data in the ARMHostCPUFeatures* that it is
> passed as an argument. The caller is kvm_arm_set_cpu_features_from_host(),
> which does
>  kvm_arm_get_host_cpu_features(_host_cpu_features)
> (assuming it hasn't already done it once and cached the results;
> arm_host_cpu_features is a global) and then
>  cpu->isar = arm_host_cpu_features.isar;
> thus copying the ID values into the "struct ARMISARegisters isar"
> that is part of the ARMCPU struct. (It also copies across the
> 'features' word which gets set up with ARM_FEATURE_* flags
> for the benefit of the parts of the target code which key off
> those rather than ID register fields.)

Thanks!  For some reason I missed the line above when grepping
for id_aa64mmfr0.

-- 
Eduardo

Re: [RFC v6 10/11] accel: introduce AccelCPUClass extending CPUClass

[Claudio Fontana]

On 11/27/20 7:13 PM, Eduardo Habkost wrote:
> On Fri, Nov 27, 2020 at 06:58:22PM +0100, Claudio Fontana wrote:
>> On 11/27/20 6:06 PM, Eduardo Habkost wrote:
>>> On Thu, Nov 26, 2020 at 11:32:17PM +0100, Claudio Fontana wrote:
 add a new optional interface to CPUClass,
 which allows accelerators to extend the CPUClass
 with additional accelerator-specific initializations.

 Signed-off-by: Claudio Fontana 
 ---
>>> [...]
 +static void accel_init_cpu_int_aux(ObjectClass *klass, void *opaque)
 +{
 +CPUClass *cc = CPU_CLASS(klass);
 +AccelCPUClass *accel_cpu_interface = opaque;
 +
 +cc->accel_cpu_interface = accel_cpu_interface;
 +if (accel_cpu_interface->cpu_class_init) {
 +accel_cpu_interface->cpu_class_init(cc);
 +}
 +}
>>>
>>> So, now that the approach we're following to trigger the
>>> accel_init_cpu*() call is less controversial (thanks for your
>>> patience!), we can try to address the monkey patching issue:
>>>
>>> Monkey patching classes like this is acceptable as an initial
>>> solution, but I'd like us to have a plan to eventually get rid of
>>> it.  Monkey patching CPU classes makes querying of CPU model
>>> information less predictable and subtly dependent on QEMU
>>> initialization state.
>>
>>
>> The question of QEMU initialization state and the querying of supported 
>> functionality, also in relationship with the loadable modules, is I think a 
>> larger discussion.
>>
>> Regardless of the amount of glue code and lipstick, this is hiding the fact 
>> that the fundamentals of the object hierarchy for cpus are wrong,
>> and are (unfortunately) codified as part of the external interface.
> 
> That's probably right, and removal of monkey patching might force
> us to change our external interfaces.
> 
>>
>>
>>>
>>> Removing CPUClass.accel_cpu_interface may be easy, because it
>>> should be possible to just call current_accel() when realizing
>>> CPUs.  Getting rid of CPUClass.cpu_class_init might be more
>>> difficult, depending on what the ->cpu_class_init() function is
>>> doing.
>>
>>
>> This seems to be for a next step to me.
> 
> Agreed, although I'd like to understand what makes
> AccelCPUClass.cpu_class_init() so important in the first version
> (considering that existing x86_cpu_class_init() has zero
> tcg_enabled() calls today).
> 

currently x86_cpu_common_class_init() has

#ifdef CONFIG_TCG

and

#ifdef CONFIG_USER_ONLY

I move this code to a tcg specific module,
and I also move the parts that should have been CONFIG_TCG before but were 
probably just missed.

Ciao,

Claudio

Re: [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework

[Peter Maydell]

On Fri, 27 Nov 2020 at 17:18, Eduardo Habkost  wrote:
> Thanks!  Is the data returned by kvm_arm_get_host_cpu_features()
> supposed to eventually affect the value of id_aa64mmfr0?  I don't
> see how that could happen.

kvm_arm_get_host_cpu_features() does:
err |= read_sys_reg64(fdarray[2], >isar.id_aa64mmfr0,
  ARM64_SYS_REG(3, 0, 0, 7, 0));

which is filling in data in the ARMHostCPUFeatures* that it is
passed as an argument. The caller is kvm_arm_set_cpu_features_from_host(),
which does
 kvm_arm_get_host_cpu_features(_host_cpu_features)
(assuming it hasn't already done it once and cached the results;
arm_host_cpu_features is a global) and then
 cpu->isar = arm_host_cpu_features.isar;
thus copying the ID values into the "struct ARMISARegisters isar"
that is part of the ARMCPU struct. (It also copies across the
'features' word which gets set up with ARM_FEATURE_* flags
for the benefit of the parts of the target code which key off
those rather than ID register fields.)

thanks
-- PMM

Re: [RFC v6 11/11] i386: split cpu accelerators from cpu.c, using AccelCPUClass

[Claudio Fontana]

On 11/27/20 7:09 PM, Eduardo Habkost wrote:
> On Fri, Nov 27, 2020 at 06:53:23PM +0100, Claudio Fontana wrote:
>> On 11/27/20 6:41 PM, Eduardo Habkost wrote:
>>> On Thu, Nov 26, 2020 at 11:32:18PM +0100, Claudio Fontana wrote:
 i386 is the first user of AccelCPUClass, allowing to split
 cpu.c into:

 cpu.ccpuid and common x86 cpu functionality
 host-cpu.c   host x86 cpu functions and "host" cpu type
 kvm/cpu.cKVM x86 AccelCPUClass
 hvf/cpu.cHVF x86 AccelCPUClass
 tcg/cpu.cTCG x86 AccelCPUClass

 Signed-off-by: Claudio Fontana 
 ---
>>> [...]
 +static void tcg_cpu_class_init(CPUClass *cc)
>>>
>>> Is this the only case where we need to provide an
>>> AccelCPUClass.cpu_class_init method?
>>>
>>>
 +{
 +cc->do_interrupt = x86_cpu_do_interrupt;
 +cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
 +cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
 +cc->cpu_exec_enter = x86_cpu_exec_enter;
 +cc->cpu_exec_exit = x86_cpu_exec_exit;
 +cc->tcg_initialize = tcg_x86_init;
 +cc->tlb_fill = x86_cpu_tlb_fill;
 +#ifndef CONFIG_USER_ONLY
 +cc->debug_excp_handler = breakpoint_handler;
 +#endif /* !CONFIG_USER_ONLY */
>>>
>>> I find the need for these method overrides suspicious.
>>
>> These mechanisms are preexistent. My refactoring only makes them more 
>> visible.
>>
>>>
>>> Comparing this with the code on qemu.git master:
>>>
>>> static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
>>> {
>>> [...]
>>> #ifdef CONFIG_TCG
>>> cc->do_interrupt = x86_cpu_do_interrupt;
>>> cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
>>> #endif
>>> [...]
>>> cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
>>> [...]
>>> #if defined(CONFIG_TCG) && !defined(CONFIG_USER_ONLY)
>>> cc->debug_excp_handler = breakpoint_handler;
>>> #endif
>>> cc->cpu_exec_enter = x86_cpu_exec_enter;
>>> cc->cpu_exec_exit = x86_cpu_exec_exit;
>>> #ifdef CONFIG_TCG
>>> cc->tcg_initialize = tcg_x86_init;
>>> cc->tlb_fill = x86_cpu_tlb_fill;
>>> #endif
>>> [...]
>>> }
>>>
>>> So, we have two kinds of CPUClass fields above:
>>> * Code that was never conditional on CONFIG_TCG, and now is
>>>   conditional (synchronize_from_tb, cpu_exec_enter,
>>>   cpu_exec_exit).
>>
>>
>>
>> synchronize_from_tb, cpu_exec_enter and cpu_exec_exit only makes sense for 
>> TCG,
>> and their code should not be compiled in for non-TCG.
>>
>> This is part of the effort to separate away non-pertinent code into 
>> accelerator-specific builds (and in the future modules).
>>
>> The fact that they were unconditionally compiled in before was a mistake, or 
>> at least this is the assumption I am making when changing this.
> 
> Can you clarify why that was a mistake?  Patch 07/11 makes
> existing functions TCG-specific, but doesn't explain why.



They are functions called only for TCG. You don't need translation blocks and 
cpu_exec() for other accelerators.
I can add this information in the commit message.

> 
>>
>>
>>> * Code that was conditional on CONFIG_TCG at compile time, and is
>>>   now conditional on TCG being enabled at runtime.
>>>
>>> On both cases, we are replacing static initialization of CPUClass
>>> fields with dynamically initialization of CPUClass depending on
>>> the chosen accelerator.  What makes this necessary?
>>
>>
>> I am not sure about the definitions of "static" and "dynamic"
>> you are using here, as all the above looks "dynamic" to me,
>> before and after.
> 
> The existing code initializes all CPUClass fields as soon as
> possible (at class_init), and we can be 100% sure the CPUClass
> methods above are always pointing to the right function and will
> never change.
> 
> This patch makes those CPUClass fields change at runtime, after
> accelerator initialization.
> 
> This might be a reasonable approach if we have a good reason for
> it, but what can we do to simplify this?  If those methods really
> depend on the accelerator (and not the CPU class), why aren't
> they part of the CpusAccel struct instead of CPUClass?
> 
> 
>>
>> The overrides and additional fields in the CPUClass are only
>> necessary for TCG, and all that code should not be compiled in
>> if TCG is not built-in.
>>
>> Ciao,
>>
>> Claudio
>>
>>>
>>>
 +}
 +
 +/*
 + * TCG-specific defaults that override all CPU models when using TCG
 + */
 +static PropValue tcg_default_props[] = {
 +{ "vme", "off" },
 +{ NULL, NULL },
 +};
 +
 +static void tcg_cpu_instance_init(CPUState *cs)
 +{
 +X86CPU *cpu = X86_CPU(cs);
 +/* Special cases not set in the X86CPUDefinition structs: */
 +x86_cpu_apply_props(cpu, tcg_default_props);
 +}
 +
 +static void tcg_cpu_accel_class_init(ObjectClass *oc, void *data)
 +{
 +AccelCPUClass *acc = ACCEL_CPU_CLASS(oc);
 +
 +

Re: [PATCH] gitlab-CI: Test 32-bit builds with the fedora-i386-cross container

[Thomas Huth]

On 27/11/2020 19.10, Thomas Huth wrote:
> After adding some missing packages, it's possible to check 32-bit
> builds and tests with the fedora-i386-cross container in the gitlab-CI,
> too.
> 
> While we're at it, update it to Fedora 31. Unfortunately the gcc
> from the later versions emits some very dubious format-truncation
> warnings, so Fedora 32 and 33 are currently unsuitable for this job.
> 
> I also had to remove the gnutls-devel package since linking then
> failed due to its dependency libtasn1 not being recognized (even
> if I explicitely installed it into the container).

In case anybody is interested, the failure can be seen here:

 https://gitlab.com/huth/qemu/-/jobs/876977206#L4799

... which is very strange, since in line 259, meson claimed to have found
libtasn1 ... anybody got an idea what's going wrong here?

 Thomas


> ---
>  Test runs can be seen here:
>   https://gitlab.com/huth/qemu/-/jobs/877519207
>  and here:
>   https://gitlab.com/huth/qemu/-/jobs/877519208
> 
>  .gitlab-ci.d/crossbuilds.yml | 16 ++--
>  .../docker/dockerfiles/fedora-i386-cross.docker  | 14 --
>  2 files changed, 26 insertions(+), 4 deletions(-)
> 
> diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
> index 03ebfabb3f..283c42b100 100644
> --- a/.gitlab-ci.d/crossbuilds.yml
> +++ b/.gitlab-ci.d/crossbuilds.yml
> @@ -11,7 +11,7 @@
>  --target-list-exclude="aarch64-softmmu i386-softmmu 
> microblaze-softmmu
>mips-softmmu mipsel-softmmu mips64-softmmu ppc64-softmmu 
> sh4-softmmu
>xtensa-softmmu"
> -- make -j$(expr $(nproc) + 1) all check-build
> +- make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
>  
>  .cross_user_build_job_template: _user_build_job_definition
>stage: build
> @@ -21,7 +21,7 @@
>  - cd build
>  - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
>../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system
> -- make -j$(expr $(nproc) + 1) all check-build
> +- make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
>  
>  cross-armel-system:
><<: *cross_system_build_job_definition
> @@ -53,6 +53,18 @@ cross-arm64-user:
>variables:
>  IMAGE: debian-arm64-cross
>  
> +cross-i386-system:
> +  <<: *cross_system_build_job_definition
> +  variables:
> +IMAGE: fedora-i386-cross
> +MAKE_CHECK_ARGS: check
> +
> +cross-i386-user:
> +  <<: *cross_user_build_job_definition
> +  variables:
> +IMAGE: fedora-i386-cross
> +MAKE_CHECK_ARGS: check
> +
>  cross-mips-system:
><<: *cross_system_build_job_definition
>variables:
> diff --git a/tests/docker/dockerfiles/fedora-i386-cross.docker 
> b/tests/docker/dockerfiles/fedora-i386-cross.docker
> index cd16cd1bfa..7824cc0da3 100644
> --- a/tests/docker/dockerfiles/fedora-i386-cross.docker
> +++ b/tests/docker/dockerfiles/fedora-i386-cross.docker
> @@ -1,14 +1,24 @@
> -FROM fedora:30
> +FROM fedora:31
>  ENV PACKAGES \
> +bzip2 \
> +diffutils \
> +findutils \
>  gcc \
> +git \
> +make \
> +meson \
> +ninja-build \
>  glib2-devel.i686 \
>  glibc-devel.i686 \
>  glibc-static.i686 \
> -gnutls-devel.i686 \
>  nettle-devel.i686 \
> +perl-Test-Harness \
>  pixman-devel.i686 \
>  zlib-devel.i686 \
>  libzstd-devel.i686
>  
> +ENV QEMU_CONFIGURE_OPTS --extra-cflags=-m32
> +ENV PKG_CONFIG_PATH /usr/lib/pkgconfig
> +
>  RUN dnf install -y $PACKAGES
>  RUN rpm -q $PACKAGES | sort > /packages.txt
> 

Re: [RFC v6 10/11] accel: introduce AccelCPUClass extending CPUClass

[Eduardo Habkost]

On Fri, Nov 27, 2020 at 06:58:22PM +0100, Claudio Fontana wrote:
> On 11/27/20 6:06 PM, Eduardo Habkost wrote:
> > On Thu, Nov 26, 2020 at 11:32:17PM +0100, Claudio Fontana wrote:
> >> add a new optional interface to CPUClass,
> >> which allows accelerators to extend the CPUClass
> >> with additional accelerator-specific initializations.
> >>
> >> Signed-off-by: Claudio Fontana 
> >> ---
> > [...]
> >> +static void accel_init_cpu_int_aux(ObjectClass *klass, void *opaque)
> >> +{
> >> +CPUClass *cc = CPU_CLASS(klass);
> >> +AccelCPUClass *accel_cpu_interface = opaque;
> >> +
> >> +cc->accel_cpu_interface = accel_cpu_interface;
> >> +if (accel_cpu_interface->cpu_class_init) {
> >> +accel_cpu_interface->cpu_class_init(cc);
> >> +}
> >> +}
> > 
> > So, now that the approach we're following to trigger the
> > accel_init_cpu*() call is less controversial (thanks for your
> > patience!), we can try to address the monkey patching issue:
> > 
> > Monkey patching classes like this is acceptable as an initial
> > solution, but I'd like us to have a plan to eventually get rid of
> > it.  Monkey patching CPU classes makes querying of CPU model
> > information less predictable and subtly dependent on QEMU
> > initialization state.
> 
> 
> The question of QEMU initialization state and the querying of supported 
> functionality, also in relationship with the loadable modules, is I think a 
> larger discussion.
> 
> Regardless of the amount of glue code and lipstick, this is hiding the fact 
> that the fundamentals of the object hierarchy for cpus are wrong,
> and are (unfortunately) codified as part of the external interface.

That's probably right, and removal of monkey patching might force
us to change our external interfaces.

> 
> 
> > 
> > Removing CPUClass.accel_cpu_interface may be easy, because it
> > should be possible to just call current_accel() when realizing
> > CPUs.  Getting rid of CPUClass.cpu_class_init might be more
> > difficult, depending on what the ->cpu_class_init() function is
> > doing.
> 
> 
> This seems to be for a next step to me.

Agreed, although I'd like to understand what makes
AccelCPUClass.cpu_class_init() so important in the first version
(considering that existing x86_cpu_class_init() has zero
tcg_enabled() calls today).

-- 
Eduardo

[PATCH] gitlab-CI: Test 32-bit builds with the fedora-i386-cross container

[Thomas Huth]

After adding some missing packages, it's possible to check 32-bit
builds and tests with the fedora-i386-cross container in the gitlab-CI,
too.

While we're at it, update it to Fedora 31. Unfortunately the gcc
from the later versions emits some very dubious format-truncation
warnings, so Fedora 32 and 33 are currently unsuitable for this job.

I also had to remove the gnutls-devel package since linking then
failed due to its dependency libtasn1 not being recognized (even
if I explicitely installed it into the container).

Signed-off-by: Thomas Huth 
---
 Test runs can be seen here:
  https://gitlab.com/huth/qemu/-/jobs/877519207
 and here:
  https://gitlab.com/huth/qemu/-/jobs/877519208

 .gitlab-ci.d/crossbuilds.yml | 16 ++--
 .../docker/dockerfiles/fedora-i386-cross.docker  | 14 --
 2 files changed, 26 insertions(+), 4 deletions(-)

diff --git a/.gitlab-ci.d/crossbuilds.yml b/.gitlab-ci.d/crossbuilds.yml
index 03ebfabb3f..283c42b100 100644
--- a/.gitlab-ci.d/crossbuilds.yml
+++ b/.gitlab-ci.d/crossbuilds.yml
@@ -11,7 +11,7 @@
 --target-list-exclude="aarch64-softmmu i386-softmmu microblaze-softmmu
   mips-softmmu mipsel-softmmu mips64-softmmu ppc64-softmmu sh4-softmmu
   xtensa-softmmu"
-- make -j$(expr $(nproc) + 1) all check-build
+- make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
 .cross_user_build_job_template: _user_build_job_definition
   stage: build
@@ -21,7 +21,7 @@
 - cd build
 - PKG_CONFIG_PATH=$PKG_CONFIG_PATH
   ../configure --enable-werror $QEMU_CONFIGURE_OPTS --disable-system
-- make -j$(expr $(nproc) + 1) all check-build
+- make -j$(expr $(nproc) + 1) all check-build $MAKE_CHECK_ARGS
 
 cross-armel-system:
   <<: *cross_system_build_job_definition
@@ -53,6 +53,18 @@ cross-arm64-user:
   variables:
 IMAGE: debian-arm64-cross
 
+cross-i386-system:
+  <<: *cross_system_build_job_definition
+  variables:
+IMAGE: fedora-i386-cross
+MAKE_CHECK_ARGS: check
+
+cross-i386-user:
+  <<: *cross_user_build_job_definition
+  variables:
+IMAGE: fedora-i386-cross
+MAKE_CHECK_ARGS: check
+
 cross-mips-system:
   <<: *cross_system_build_job_definition
   variables:
diff --git a/tests/docker/dockerfiles/fedora-i386-cross.docker 
b/tests/docker/dockerfiles/fedora-i386-cross.docker
index cd16cd1bfa..7824cc0da3 100644
--- a/tests/docker/dockerfiles/fedora-i386-cross.docker
+++ b/tests/docker/dockerfiles/fedora-i386-cross.docker
@@ -1,14 +1,24 @@
-FROM fedora:30
+FROM fedora:31
 ENV PACKAGES \
+bzip2 \
+diffutils \
+findutils \
 gcc \
+git \
+make \
+meson \
+ninja-build \
 glib2-devel.i686 \
 glibc-devel.i686 \
 glibc-static.i686 \
-gnutls-devel.i686 \
 nettle-devel.i686 \
+perl-Test-Harness \
 pixman-devel.i686 \
 zlib-devel.i686 \
 libzstd-devel.i686
 
+ENV QEMU_CONFIGURE_OPTS --extra-cflags=-m32
+ENV PKG_CONFIG_PATH /usr/lib/pkgconfig
+
 RUN dnf install -y $PACKAGES
 RUN rpm -q $PACKAGES | sort > /packages.txt
-- 
2.18.4

Re: [RFC v6 11/11] i386: split cpu accelerators from cpu.c, using AccelCPUClass

[Eduardo Habkost]

On Fri, Nov 27, 2020 at 06:53:23PM +0100, Claudio Fontana wrote:
> On 11/27/20 6:41 PM, Eduardo Habkost wrote:
> > On Thu, Nov 26, 2020 at 11:32:18PM +0100, Claudio Fontana wrote:
> >> i386 is the first user of AccelCPUClass, allowing to split
> >> cpu.c into:
> >>
> >> cpu.ccpuid and common x86 cpu functionality
> >> host-cpu.c   host x86 cpu functions and "host" cpu type
> >> kvm/cpu.cKVM x86 AccelCPUClass
> >> hvf/cpu.cHVF x86 AccelCPUClass
> >> tcg/cpu.cTCG x86 AccelCPUClass
> >>
> >> Signed-off-by: Claudio Fontana 
> >> ---
> > [...]
> >> +static void tcg_cpu_class_init(CPUClass *cc)
> > 
> > Is this the only case where we need to provide an
> > AccelCPUClass.cpu_class_init method?
> > 
> > 
> >> +{
> >> +cc->do_interrupt = x86_cpu_do_interrupt;
> >> +cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
> >> +cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
> >> +cc->cpu_exec_enter = x86_cpu_exec_enter;
> >> +cc->cpu_exec_exit = x86_cpu_exec_exit;
> >> +cc->tcg_initialize = tcg_x86_init;
> >> +cc->tlb_fill = x86_cpu_tlb_fill;
> >> +#ifndef CONFIG_USER_ONLY
> >> +cc->debug_excp_handler = breakpoint_handler;
> >> +#endif /* !CONFIG_USER_ONLY */
> > 
> > I find the need for these method overrides suspicious.
> 
> These mechanisms are preexistent. My refactoring only makes them more visible.
> 
> > 
> > Comparing this with the code on qemu.git master:
> > 
> > static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
> > {
> > [...]
> > #ifdef CONFIG_TCG
> > cc->do_interrupt = x86_cpu_do_interrupt;
> > cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
> > #endif
> > [...]
> > cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
> > [...]
> > #if defined(CONFIG_TCG) && !defined(CONFIG_USER_ONLY)
> > cc->debug_excp_handler = breakpoint_handler;
> > #endif
> > cc->cpu_exec_enter = x86_cpu_exec_enter;
> > cc->cpu_exec_exit = x86_cpu_exec_exit;
> > #ifdef CONFIG_TCG
> > cc->tcg_initialize = tcg_x86_init;
> > cc->tlb_fill = x86_cpu_tlb_fill;
> > #endif
> > [...]
> > }
> > 
> > So, we have two kinds of CPUClass fields above:
> > * Code that was never conditional on CONFIG_TCG, and now is
> >   conditional (synchronize_from_tb, cpu_exec_enter,
> >   cpu_exec_exit).
> 
> 
> 
> synchronize_from_tb, cpu_exec_enter and cpu_exec_exit only makes sense for 
> TCG,
> and their code should not be compiled in for non-TCG.
> 
> This is part of the effort to separate away non-pertinent code into 
> accelerator-specific builds (and in the future modules).
> 
> The fact that they were unconditionally compiled in before was a mistake, or 
> at least this is the assumption I am making when changing this.

Can you clarify why that was a mistake?  Patch 07/11 makes
existing functions TCG-specific, but doesn't explain why.

> 
> 
> > * Code that was conditional on CONFIG_TCG at compile time, and is
> >   now conditional on TCG being enabled at runtime.
> > 
> > On both cases, we are replacing static initialization of CPUClass
> > fields with dynamically initialization of CPUClass depending on
> > the chosen accelerator.  What makes this necessary?
> 
> 
> I am not sure about the definitions of "static" and "dynamic"
> you are using here, as all the above looks "dynamic" to me,
> before and after.

The existing code initializes all CPUClass fields as soon as
possible (at class_init), and we can be 100% sure the CPUClass
methods above are always pointing to the right function and will
never change.

This patch makes those CPUClass fields change at runtime, after
accelerator initialization.

This might be a reasonable approach if we have a good reason for
it, but what can we do to simplify this?  If those methods really
depend on the accelerator (and not the CPU class), why aren't
they part of the CpusAccel struct instead of CPUClass?


> 
> The overrides and additional fields in the CPUClass are only
> necessary for TCG, and all that code should not be compiled in
> if TCG is not built-in.
> 
> Ciao,
> 
> Claudio
> 
> > 
> > 
> >> +}
> >> +
> >> +/*
> >> + * TCG-specific defaults that override all CPU models when using TCG
> >> + */
> >> +static PropValue tcg_default_props[] = {
> >> +{ "vme", "off" },
> >> +{ NULL, NULL },
> >> +};
> >> +
> >> +static void tcg_cpu_instance_init(CPUState *cs)
> >> +{
> >> +X86CPU *cpu = X86_CPU(cs);
> >> +/* Special cases not set in the X86CPUDefinition structs: */
> >> +x86_cpu_apply_props(cpu, tcg_default_props);
> >> +}
> >> +
> >> +static void tcg_cpu_accel_class_init(ObjectClass *oc, void *data)
> >> +{
> >> +AccelCPUClass *acc = ACCEL_CPU_CLASS(oc);
> >> +
> >> +acc->cpu_realizefn = tcg_cpu_realizefn;
> >> +acc->cpu_class_init = tcg_cpu_class_init;
> >> +acc->cpu_instance_init = tcg_cpu_instance_init;
> >> +}
> >> +static const TypeInfo tcg_cpu_accel_type_info = {
> >> +.name = ACCEL_CPU_NAME("tcg"),
> >> +
> >> +.parent 

Re: [RFC v6 10/11] accel: introduce AccelCPUClass extending CPUClass

[Claudio Fontana]

On 11/27/20 6:06 PM, Eduardo Habkost wrote:
> On Thu, Nov 26, 2020 at 11:32:17PM +0100, Claudio Fontana wrote:
>> add a new optional interface to CPUClass,
>> which allows accelerators to extend the CPUClass
>> with additional accelerator-specific initializations.
>>
>> Signed-off-by: Claudio Fontana 
>> ---
> [...]
>> +static void accel_init_cpu_int_aux(ObjectClass *klass, void *opaque)
>> +{
>> +CPUClass *cc = CPU_CLASS(klass);
>> +AccelCPUClass *accel_cpu_interface = opaque;
>> +
>> +cc->accel_cpu_interface = accel_cpu_interface;
>> +if (accel_cpu_interface->cpu_class_init) {
>> +accel_cpu_interface->cpu_class_init(cc);
>> +}
>> +}
> 
> So, now that the approach we're following to trigger the
> accel_init_cpu*() call is less controversial (thanks for your
> patience!), we can try to address the monkey patching issue:
> 
> Monkey patching classes like this is acceptable as an initial
> solution, but I'd like us to have a plan to eventually get rid of
> it.  Monkey patching CPU classes makes querying of CPU model
> information less predictable and subtly dependent on QEMU
> initialization state.


The question of QEMU initialization state and the querying of supported 
functionality, also in relationship with the loadable modules, is I think a 
larger discussion.

Regardless of the amount of glue code and lipstick, this is hiding the fact 
that the fundamentals of the object hierarchy for cpus are wrong,
and are (unfortunately) codified as part of the external interface.


> 
> Removing CPUClass.accel_cpu_interface may be easy, because it
> should be possible to just call current_accel() when realizing
> CPUs.  Getting rid of CPUClass.cpu_class_init might be more
> difficult, depending on what the ->cpu_class_init() function is
> doing.


This seems to be for a next step to me.

Ciao,

Claudio


> 
>> +
>> +/* initialize the arch-specific accel CpuClass interfaces */
>> +static void accel_init_cpu_interfaces(AccelClass *ac, const char *cpu_type)
>> +{
>> +const char *ac_name; /* AccelClass name */
>> +char *acc_name;  /* AccelCPUClass name */
>> +ObjectClass *acc;/* AccelCPUClass */
>> +
>> +ac_name = object_class_get_name(OBJECT_CLASS(ac));
>> +g_assert(ac_name != NULL);
>> +
>> +acc_name = g_strdup_printf("%s-%s", ac_name, CPU_RESOLVING_TYPE);
>> +acc = object_class_by_name(acc_name);
>> +g_free(acc_name);
>> +
>> +if (acc) {
>> +object_class_foreach(accel_init_cpu_int_aux, cpu_type, false, acc);
>> +}
>> +}
>> +
>>  void accel_init_interfaces(AccelClass *ac, const char *cpu_type)
>>  {
>>  #ifndef CONFIG_USER_ONLY
>>  accel_init_ops_interfaces(ac);
>>  #endif /* !CONFIG_USER_ONLY */
>> +
>> +accel_init_cpu_interfaces(ac, cpu_type);
>>  }
> [...]
> 

Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Philippe Mathieu-Daudé]

On 11/27/20 6:47 PM, Thomas Huth wrote:
> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
>> We lately realized that the Avocado framework was not designed
>> to be regularly run on CI environments. Therefore, as of 5.2
>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
>> current users, it is possible to keep the current behavior by
>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
>> (see [*]).
>> From now on, using these jobs (or adding new tests to them)
>> is strongly discouraged.
>>
>> Tests based on Avocado will be ported to new job schemes during
>> the next releases, with better documentation and templates.
> 
> Why should we disable the jobs by default as long as there is no replacement
> available yet?

Why keep it enabled if it is failing randomly, if images hardcoded
in tests are being removed from public servers, etc...?


They are not disabled, they are still runnable manually or setting
QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE...

We realized by default Avocado runs all tests on the CI jobs,
triggering failures and complains. Developer stopped to contribute/
review integration tests because of that. We want developers be
able to contribute tests to the repository fearlessly.

If we don't change anything, we'll keep having CI failures due
to Avocado design issues (artifacts removed from remote servers,
etc...).

I haven't seen any attempt on this list to improve the current
fragile situation, but better suggestions are certainly welcome.

Thanks,

Phil.

Re: [RFC v6 11/11] i386: split cpu accelerators from cpu.c, using AccelCPUClass

[Claudio Fontana]

On 11/27/20 6:41 PM, Eduardo Habkost wrote:
> On Thu, Nov 26, 2020 at 11:32:18PM +0100, Claudio Fontana wrote:
>> i386 is the first user of AccelCPUClass, allowing to split
>> cpu.c into:
>>
>> cpu.ccpuid and common x86 cpu functionality
>> host-cpu.c   host x86 cpu functions and "host" cpu type
>> kvm/cpu.cKVM x86 AccelCPUClass
>> hvf/cpu.cHVF x86 AccelCPUClass
>> tcg/cpu.cTCG x86 AccelCPUClass
>>
>> Signed-off-by: Claudio Fontana 
>> ---
> [...]
>> +static void tcg_cpu_class_init(CPUClass *cc)
> 
> Is this the only case where we need to provide an
> AccelCPUClass.cpu_class_init method?
> 
> 
>> +{
>> +cc->do_interrupt = x86_cpu_do_interrupt;
>> +cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
>> +cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
>> +cc->cpu_exec_enter = x86_cpu_exec_enter;
>> +cc->cpu_exec_exit = x86_cpu_exec_exit;
>> +cc->tcg_initialize = tcg_x86_init;
>> +cc->tlb_fill = x86_cpu_tlb_fill;
>> +#ifndef CONFIG_USER_ONLY
>> +cc->debug_excp_handler = breakpoint_handler;
>> +#endif /* !CONFIG_USER_ONLY */
> 
> I find the need for these method overrides suspicious.

These mechanisms are preexistent. My refactoring only makes them more visible.

> 
> Comparing this with the code on qemu.git master:
> 
> static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
> {
> [...]
> #ifdef CONFIG_TCG
> cc->do_interrupt = x86_cpu_do_interrupt;
> cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
> #endif
> [...]
> cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
> [...]
> #if defined(CONFIG_TCG) && !defined(CONFIG_USER_ONLY)
> cc->debug_excp_handler = breakpoint_handler;
> #endif
> cc->cpu_exec_enter = x86_cpu_exec_enter;
> cc->cpu_exec_exit = x86_cpu_exec_exit;
> #ifdef CONFIG_TCG
> cc->tcg_initialize = tcg_x86_init;
> cc->tlb_fill = x86_cpu_tlb_fill;
> #endif
> [...]
> }
> 
> So, we have two kinds of CPUClass fields above:
> * Code that was never conditional on CONFIG_TCG, and now is
>   conditional (synchronize_from_tb, cpu_exec_enter,
>   cpu_exec_exit).



synchronize_from_tb, cpu_exec_enter and cpu_exec_exit only makes sense for TCG,
and their code should not be compiled in for non-TCG.

This is part of the effort to separate away non-pertinent code into 
accelerator-specific builds (and in the future modules).

The fact that they were unconditionally compiled in before was a mistake, or at 
least this is the assumption I am making when changing this.


> * Code that was conditional on CONFIG_TCG at compile time, and is
>   now conditional on TCG being enabled at runtime.
> 
> On both cases, we are replacing static initialization of CPUClass
> fields with dynamically initialization of CPUClass depending on
> the chosen accelerator.  What makes this necessary?


I am not sure about the definitions of "static" and "dynamic" you are using 
here, as all the above looks "dynamic" to me, before and after.

The overrides and additional fields in the CPUClass are only necessary for TCG, 
and all that code should not be compiled in if TCG is not built-in.

Ciao,

Claudio

> 
> 
>> +}
>> +
>> +/*
>> + * TCG-specific defaults that override all CPU models when using TCG
>> + */
>> +static PropValue tcg_default_props[] = {
>> +{ "vme", "off" },
>> +{ NULL, NULL },
>> +};
>> +
>> +static void tcg_cpu_instance_init(CPUState *cs)
>> +{
>> +X86CPU *cpu = X86_CPU(cs);
>> +/* Special cases not set in the X86CPUDefinition structs: */
>> +x86_cpu_apply_props(cpu, tcg_default_props);
>> +}
>> +
>> +static void tcg_cpu_accel_class_init(ObjectClass *oc, void *data)
>> +{
>> +AccelCPUClass *acc = ACCEL_CPU_CLASS(oc);
>> +
>> +acc->cpu_realizefn = tcg_cpu_realizefn;
>> +acc->cpu_class_init = tcg_cpu_class_init;
>> +acc->cpu_instance_init = tcg_cpu_instance_init;
>> +}
>> +static const TypeInfo tcg_cpu_accel_type_info = {
>> +.name = ACCEL_CPU_NAME("tcg"),
>> +
>> +.parent = TYPE_ACCEL_CPU,
>> +.class_init = tcg_cpu_accel_class_init,
>> +.abstract = true,
>> +};
>> +static void tcg_cpu_accel_register_types(void)
>> +{
>> +type_register_static(_cpu_accel_type_info);
>> +}
>> +type_init(tcg_cpu_accel_register_types);
>> diff --git a/target/i386/tcg/meson.build b/target/i386/tcg/meson.build
>> index 02794226c2..9e439df9c7 100644
>> --- a/target/i386/tcg/meson.build
>> +++ b/target/i386/tcg/meson.build
>> @@ -10,4 +10,5 @@ i386_ss.add(when: 'CONFIG_TCG', if_true: files(
>>'seg_helper.c',
>>'smm_helper.c',
>>'svm_helper.c',
>> -  'translate.c'), if_false: files('tcg-stub.c'))
>> +  'translate.c',
>> +  'cpu.c'), if_false: files('tcg-stub.c'))
>> -- 
>> 2.26.2
>>
>>
> 

Re: [PATCH 0/8] hvf: Implement Apple Silicon Support

[Philippe Mathieu-Daudé]

On 11/26/20 11:10 PM, Eduardo Habkost wrote:
> On Thu, Nov 26, 2020 at 10:50:09PM +0100, Alexander Graf wrote:
>> Now that Apple Silicon is widely available, people are obviously excited
>> to try and run virtualized workloads on them, such as Linux and Windows.
>>
>> This patch set implements a rudimentary, first version to get the ball
>> going on that. With this applied, I can successfully run both Linux and
>> Windows as guests, albeit with a few caveats:
>>
>>   * no WFI emulation, a vCPU always uses 100%
>>   * vtimer handling is a bit hacky
>>   * we handle most sysregs flying blindly, just returning 0
>>   * XHCI breaks in OVMF, works in Linux+Windows
>>
>> Despite those drawbacks, it's still an exciting place to start playing
>> with the power of Apple Silicon.
>>
>> Enjoy!
>>
>> Alex
>>
>> Alexander Graf (8):
>>   hvf: Add hypervisor entitlement to output binaries
>>   hvf: Move common code out
>>   arm: Set PSCI to 0.2 for HVF
>>   arm: Synchronize CPU on PSCI on
>>   hvf: Add Apple Silicon support
>>   hvf: Use OS provided vcpu kick function
>>   arm: Add Hypervisor.framework build target
>>   hw/arm/virt: Disable highmem when on hypervisor.framework
>>
>>  MAINTAINERS  |  14 +-
>>  accel/hvf/entitlements.plist |   8 +
>>  accel/hvf/hvf-all.c  |  56 
>>  accel/hvf/hvf-cpus.c | 484 +++
>>  accel/hvf/meson.build|   7 +
>>  accel/meson.build|   1 +
> 
> This seems to conflict with the accel cleanup work being done by
> Claudio[1].  Maybe Claudio could cherry-pick some of the code
> movement patches from this series, or this series could be
> rebased on top of his.

It seems easier for Claudio to cherry-pick patch 2/8
of this series ("hvf: Move common code out") and rebase
on top.

Claudio's series is still tagged RFC, but if you were
planing to queue it, you could take patch 2/8 out of
this series, as it is generic, and let the HVF/AA64
specific bits still being discussed.

> 
> [1] https://lore.kernel.org/qemu-devel/20201124162210.8796-1-cfont...@suse.de

Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Thomas Huth]

On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
> We lately realized that the Avocado framework was not designed
> to be regularly run on CI environments. Therefore, as of 5.2
> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
> current users, it is possible to keep the current behavior by
> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
> (see [*]).
> From now on, using these jobs (or adding new tests to them)
> is strongly discouraged.
> 
> Tests based on Avocado will be ported to new job schemes during
> the next releases, with better documentation and templates.

Why should we disable the jobs by default as long as there is no replacement
available yet?

 Thomas

Re: [RFC v6 11/11] i386: split cpu accelerators from cpu.c, using AccelCPUClass

[Eduardo Habkost]

On Thu, Nov 26, 2020 at 11:32:18PM +0100, Claudio Fontana wrote:
> i386 is the first user of AccelCPUClass, allowing to split
> cpu.c into:
> 
> cpu.ccpuid and common x86 cpu functionality
> host-cpu.c   host x86 cpu functions and "host" cpu type
> kvm/cpu.cKVM x86 AccelCPUClass
> hvf/cpu.cHVF x86 AccelCPUClass
> tcg/cpu.cTCG x86 AccelCPUClass
> 
> Signed-off-by: Claudio Fontana 
> ---
[...]
> +static void tcg_cpu_class_init(CPUClass *cc)

Is this the only case where we need to provide an
AccelCPUClass.cpu_class_init method?


> +{
> +cc->do_interrupt = x86_cpu_do_interrupt;
> +cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
> +cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
> +cc->cpu_exec_enter = x86_cpu_exec_enter;
> +cc->cpu_exec_exit = x86_cpu_exec_exit;
> +cc->tcg_initialize = tcg_x86_init;
> +cc->tlb_fill = x86_cpu_tlb_fill;
> +#ifndef CONFIG_USER_ONLY
> +cc->debug_excp_handler = breakpoint_handler;
> +#endif /* !CONFIG_USER_ONLY */

I find the need for these method overrides suspicious.

Comparing this with the code on qemu.git master:

static void x86_cpu_common_class_init(ObjectClass *oc, void *data)
{
[...]
#ifdef CONFIG_TCG
cc->do_interrupt = x86_cpu_do_interrupt;
cc->cpu_exec_interrupt = x86_cpu_exec_interrupt;
#endif
[...]
cc->synchronize_from_tb = x86_cpu_synchronize_from_tb;
[...]
#if defined(CONFIG_TCG) && !defined(CONFIG_USER_ONLY)
cc->debug_excp_handler = breakpoint_handler;
#endif
cc->cpu_exec_enter = x86_cpu_exec_enter;
cc->cpu_exec_exit = x86_cpu_exec_exit;
#ifdef CONFIG_TCG
cc->tcg_initialize = tcg_x86_init;
cc->tlb_fill = x86_cpu_tlb_fill;
#endif
[...]
}

So, we have two kinds of CPUClass fields above:
* Code that was never conditional on CONFIG_TCG, and now is
  conditional (synchronize_from_tb, cpu_exec_enter,
  cpu_exec_exit).
* Code that was conditional on CONFIG_TCG at compile time, and is
  now conditional on TCG being enabled at runtime.

On both cases, we are replacing static initialization of CPUClass
fields with dynamically initialization of CPUClass depending on
the chosen accelerator.  What makes this necessary?


> +}
> +
> +/*
> + * TCG-specific defaults that override all CPU models when using TCG
> + */
> +static PropValue tcg_default_props[] = {
> +{ "vme", "off" },
> +{ NULL, NULL },
> +};
> +
> +static void tcg_cpu_instance_init(CPUState *cs)
> +{
> +X86CPU *cpu = X86_CPU(cs);
> +/* Special cases not set in the X86CPUDefinition structs: */
> +x86_cpu_apply_props(cpu, tcg_default_props);
> +}
> +
> +static void tcg_cpu_accel_class_init(ObjectClass *oc, void *data)
> +{
> +AccelCPUClass *acc = ACCEL_CPU_CLASS(oc);
> +
> +acc->cpu_realizefn = tcg_cpu_realizefn;
> +acc->cpu_class_init = tcg_cpu_class_init;
> +acc->cpu_instance_init = tcg_cpu_instance_init;
> +}
> +static const TypeInfo tcg_cpu_accel_type_info = {
> +.name = ACCEL_CPU_NAME("tcg"),
> +
> +.parent = TYPE_ACCEL_CPU,
> +.class_init = tcg_cpu_accel_class_init,
> +.abstract = true,
> +};
> +static void tcg_cpu_accel_register_types(void)
> +{
> +type_register_static(_cpu_accel_type_info);
> +}
> +type_init(tcg_cpu_accel_register_types);
> diff --git a/target/i386/tcg/meson.build b/target/i386/tcg/meson.build
> index 02794226c2..9e439df9c7 100644
> --- a/target/i386/tcg/meson.build
> +++ b/target/i386/tcg/meson.build
> @@ -10,4 +10,5 @@ i386_ss.add(when: 'CONFIG_TCG', if_true: files(
>'seg_helper.c',
>'smm_helper.c',
>'svm_helper.c',
> -  'translate.c'), if_false: files('tcg-stub.c'))
> +  'translate.c',
> +  'cpu.c'), if_false: files('tcg-stub.c'))
> -- 
> 2.26.2
> 
> 

-- 
Eduardo

[RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

[Philippe Mathieu-Daudé]

We lately realized that the Avocado framework was not designed
to be regularly run on CI environments. Therefore, as of 5.2
we deprecate the gitlab-ci jobs using Avocado. To not disrupt
current users, it is possible to keep the current behavior by
setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
(see [*]).
>From now on, using these jobs (or adding new tests to them)
is strongly discouraged.

Tests based on Avocado will be ported to new job schemes during
the next releases, with better documentation and templates.

[*] 
https://docs.gitlab.com/ee/ci/variables/README.html#create-a-custom-variable-in-the-ui

Signed-off-by: Philippe Mathieu-Daudé 
---
 .gitlab-ci.yml | 9 +
 1 file changed, 9 insertions(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index d0173e82b16..2674407cd13 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -66,6 +66,15 @@ include:
 - cd build
 - python3 -c 'import json; r = 
json.load(open("tests/results/latest/results.json")); [print(t["logfile"]) for 
t in r["tests"] if t["status"] not in ("PASS", "SKIP", "CANCEL")]' | xargs cat
 - du -chs ${CI_PROJECT_DIR}/avocado-cache
+  rules:
+  # As of QEMU 5.2, Avocado is not yet ready to run in CI environments, 
therefore
+  # the jobs based on this template are not run automatically (except if the 
user
+  # explicitly sets the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE environment
+  # variable). Adding new jobs on top of this template is strongly discouraged.
+  - if: $QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE == null
+when: manual
+allow_failure: true
+  - when: always
 
 build-system-ubuntu:
   <<: *native_build_job_definition
-- 
2.26.2

Re: [PATCH v4 0/6] UFFD write-tracking migration/snapshots

[Andrey Gruzdev]

On 27.11.2020 18:45, Peter Xu wrote:

On Fri, Nov 27, 2020 at 01:00:48PM +0300, Andrey Gruzdev wrote:

On 27.11.2020 12:49, Peter Krempa wrote:

On Fri, Nov 27, 2020 at 11:21:39 +0300, Andrey Gruzdev wrote:

On 26.11.2020 18:47, Peter Krempa wrote:

On Thu, Nov 26, 2020 at 18:17:28 +0300, Andrey Gruzdev via wrote:

This patch series is a kind of 'rethinking' of Denis Plotnikov's ideas he's
implemented in his series '[PATCH v0 0/4] migration: add background snapshot'.

Currently the only way to make (external) live VM snapshot is using existing
dirty page logging migration mechanism. The main problem is that it tends to
produce a lot of page duplicates while running VM goes on updating already
saved pages. That leads to the fact that vmstate image size is commonly several
times bigger then non-zero part of virtual machine's RSS. Time required to
converge RAM migration and the size of snapshot image severely depend on the
guest memory write rate, sometimes resulting in unacceptably long snapshot
creation time and huge image size.

This series propose a way to solve the aforementioned problems. This is done
by using different RAM migration mechanism based on UFFD write protection
management introduced in v5.7 kernel. The migration strategy is to 'freeze'
guest RAM content using write-protection and iteratively release protection
for memory ranges that have already been saved to the migration stream.
At the same time we read in pending UFFD write fault events and save those
pages out-of-order with higher priority.


This sounds amazing! Based on your description I assume that the final
memory image contains state image from the beginning of the migration.

This would make it desirable for the 'migrate' qmp command to be used as
part of a 'transaction' qmp command so that we can do an instant disk
and memory snapshot without any extraneous pausing of the VM.

I'll have a look at using this mechanism in libvirt natively.



Correct, the final image contains state at the beginning of migration.

So far, if I'm not missing something about libvirt, for external snapshot
creation it performs a sequence like that:
migrate(fd)->transaction(blockdev-snapshot-all)->cont.

So, in case 'background-snapshot' capability is enabled, the sequence would
change to:
stop->transaction(blockdev-snapshot-all)->migrate(fd).
With background snapshot migration it will finish with VM running so there's
not need to 'cont' here.


Yes, that's correct.

The reason I've suggested that 'migrate' being part of a 'transaction'
is that it would remove the need to stop it for the disk snapshot part.



Hmm, I believe stopping VM for a short time is unavoidable to keep saved
device state consistent with blockdev snapshot base.. May be I've missed
something but it seems logical.


I guess PeterK meant an explicit stop vm command, rather than the stop in this
series that should be undetectable from the guest.  It would be definitely
great if PeterK could quickly follow up with this on libvirt. :)



Sorry if I misunderstood, sure it would be great if PeterK is really 
interested to follow up with this feature!



One overall comment (before go into details) is that please remember to start
using "migration: " prefixes in patch subjects from the next version.



Yes, I really missed it everywhere :)


Thanks,




--
Andrey Gruzdev, Principal Engineer
Virtuozzo GmbH  +7-903-247-6397
virtuzzo.com

Re: [PATCH v2] drivers/virt: vmgenid: add vm generation id driver

[Catangiu, Adrian Costin]

On 18/11/2020 12:30, Alexander Graf wrote:
>
>
> On 16.11.20 16:34, Catangiu, Adrian Costin wrote:
>> - Future improvements
>>
>> Ideally we would want the driver to register itself based on devices'
>> _CID and not _HID, but unfortunately I couldn't find a way to do that.
>> The problem is that ACPI device matching is done by
>> '__acpi_match_device()' which exclusively looks at
>> 'acpi_hardware_id *hwid'.
>>
>> There is a path for platform devices to match on _CID when _HID is
>> 'PRP0001' - but this is not the case for the Qemu vmgenid device.
>>
>> Guidance and help here would be greatly appreciated.
>
> That one is pretty important IMHO. How about the following (probably
> pretty mangled) patch? That seems to work for me. The ACPI change
> would obviously need to be its own stand alone change and needs proper
> assessment whether it could possibly break any existing systems.
>
> diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
> index 1682f8b454a2..452443d79d87 100644
> --- a/drivers/acpi/bus.c
> +++ b/drivers/acpi/bus.c
> @@ -748,7 +748,7 @@ static bool __acpi_match_device(struct acpi_device
> *device,
>  /* First, check the ACPI/PNP IDs provided by the caller. */
>  if (acpi_ids) {
>  for (id = acpi_ids; id->id[0] || id->cls; id++) {
> -    if (id->id[0] && !strcmp((char *)id->id, hwid->id))
> +    if (id->id[0] && !strncmp((char *)id->id, hwid->id,
> ACPI_ID_LEN - 1))
>  goto out_acpi_match;
>  if (id->cls && __acpi_match_device_cls(id, hwid))
>  goto out_acpi_match;
> diff --git a/drivers/virt/vmgenid.c b/drivers/virt/vmgenid.c
> index 75a787da8aad..0bfa422cf094 100644
> --- a/drivers/virt/vmgenid.c
> +++ b/drivers/virt/vmgenid.c
> @@ -356,7 +356,8 @@ static void vmgenid_acpi_notify(struct acpi_device
> *device, u32 event)
>  }
>
>  static const struct acpi_device_id vmgenid_ids[] = {
> -    {"QEMUVGID", 0},
> +    /* This really is VM_Gen_Counter, but we can only match 8
> characters */
> +    {"VM_GEN_C", 0},
>  {"", 0},
>  };
>

Looks legit. I can propose a patch with it, but how do we validate it
doesn't break any devices?


>> +2) ASYNC simplified example::
>> +
>> +    void handle_io_on_vmgenfd(int vmgenfd)
>> +    {
>> +    unsigned genid;
>> +
>> +    // because of VM generation change, we need to rebuild world
>> +    reseed_app_env();
>> +
>> +    // read new gen ID - we need it to confirm we've handled update
>> +    read(fd, , sizeof(genid));
>
> This is racy in case two consecutive snapshots happen. The read needs
> to go before the reseed.
>
Switched them around like you suggest to avoid confusion.

But I don't see a problem with this race. The idea here is to trigger
reseed_app_env() which doesn't depend on the generation counter value.
Whether it gets incremented once or N times is irrelevant, we're just
interested that we pause execution and reseed before resuming (in
between these, whether N or M generation changes is the same thing).

>> +3) Mapped memory polling simplified example::
>> +
>> +    /*
>> + * app/library function that provides cached secrets
>> + */
>> +    char * safe_cached_secret(app_data_t *app)
>> +    {
>> +    char *secret;
>> +    volatile unsigned *const genid_ptr = get_vmgenid_mapping(app);
>> +    again:
>> +    secret = __cached_secret(app);
>> +
>> +    if (unlikely(*genid_ptr != app->cached_genid)) {
>> +    // rebuild world then confirm the genid update (thru write)
>> +    rebuild_caches(app);
>> +
>> +    app->cached_genid = *genid_ptr;
>
> This is racy again. You need to read the genid before rebuild and set
> it here.
>
I don't see the race. Gen counter is read from volatile mapped mem, on
detected change we rebuild world, confirm the update back to the driver
then restart the loop. Loop will break when no more changes happen.

>> +    ack_vmgenid_update(app);
>> +
>> +    goto again;
>> +    }
>> +
>> +    return secret;
>> +    }
>> +

>> +
>> +static int vmgenid_close(struct inode *inode, struct file *file)
>> +{
>> +    struct file_data *file_data = file->private_data;
>> +    struct dev_data *priv = file_data->dev_data;
>> +
>> +    if (file_data->acked_gen_counter != priv->generation_counter)
>> +    vmgenid_put_outdated_watchers(priv);
>
> Is this racy? Could there be a snapshot notification coming between
> the branch and the put?
>
This is indeed racy, will fix it in patch v3.
>> +    atomic_dec(>watchers);
>> +    kfree(file_data);
>> +
>> +    return 0;
>> +}

>> +static ssize_t vmgenid_write(struct file *file, const char __user
>> *ubuf,
>> +    size_t count, loff_t *ppos)
>> +{
>> +    struct file_data *file_data = file->private_data;
>> +    struct dev_data *priv = file_data->dev_data;
>> +    unsigned int acked_gen_count;
>> +
>> +    /* disallow partial writes */
>> +    if (count != sizeof(acked_gen_count))

Re: [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework

[Eduardo Habkost]

On Fri, Nov 27, 2020 at 04:53:59PM +, Peter Maydell wrote:
> On Fri, 27 Nov 2020 at 16:47, Eduardo Habkost  wrote:
> > Do you know how the implicitly-accelerator-specific code is
> > implemented?  PARange is in id_aa64mmfr0, correct?  I don't see
> > any accel-specific code for initializing id_aa64mmfr0.
> 
> For TCG, the value of id_aa64mmfr0 is set by the per-cpu
> init functions aarch64_a57_initfn(), aarch64_a72_initfn(), etc.
> 
> For KVM, if we're using "-cpu cortex-a53" or "-cpu cortex-a57"
> these only work if the host CPU really is an A53 or A57, in
> which case the reset value set by the initfn is correct.
> In the more usual case of "-cpu host", we ask the kernel for
> the ID register values in kvm_arm_get_host_cpu_features(),
> which is part of the implementation of
> kvm_arm_set_cpu_features_from_host(), which gets called
> in aarch64_max_initfn() (inside a kvm_enabled() conditional).
> 
> So there is a *_enabled() check involved, which I hadn't
> realised until I worked back up to where this stuff is called.
> 

Thanks!  Is the data returned by kvm_arm_get_host_cpu_features()
supposed to eventually affect the value of id_aa64mmfr0?  I don't
see how that could happen.

-- 
Eduardo

[PULL 1/1] slirp: update to fix CVE-2020-29129 CVE-2020-29130

[marcandre . lureau]

From: Marc-André Lureau 

An out-of-bounds access issue was found in the SLIRP user networking
implementation of QEMU. It could occur while processing ARP/NCSI
packets, if the packet length was shorter than required to accommodate
respective protocol headers and payload. A privileged guest user may use
this flaw to potentially leak host information bytes.

Marc-André Lureau (1):
  Merge branch 'stable-4.2' into 'stable-4.2'

Prasad J Pandit (1):
  slirp: check pkt_len before reading protocol header

Signed-off-by: Marc-André Lureau 
---
 slirp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/slirp b/slirp
index ce94eba204..8f43a99191 16
--- a/slirp
+++ b/slirp
@@ -1 +1 @@
-Subproject commit ce94eba2042d52a0ba3d9e252ebce86715e94275
+Subproject commit 8f43a99191afb47ca3f3c6972f6306209f367ece
-- 
2.29.0

[PULL 0/1] Libslirp CVE-2020-29129 CVE-2020-29130

[marcandre . lureau]

From: Marc-André Lureau 

The following changes since commit ea8208249d1082eae0444934efb3b59cd3183f05:

  Merge remote-tracking branch 
'remotes/kraxel/tags/fixes-20201127-pull-request' into staging (2020-11-27 
11:11:43 +)

are available in the Git repository at:

  g...@github.com:elmarco/qemu.git tags/libslirp-pull-request

for you to fetch changes up to 37c0c885d19a4c2d69faed891b5c02aaffbdccfb:

  slirp: update to fix CVE-2020-29129 CVE-2020-29130 (2020-11-27 20:57:11 +0400)





Marc-André Lureau (1):
  slirp: update to fix CVE-2020-29129 CVE-2020-29130

 slirp | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
2.29.0

Re: [RFC v6 10/11] accel: introduce AccelCPUClass extending CPUClass

[Eduardo Habkost]

On Thu, Nov 26, 2020 at 11:32:17PM +0100, Claudio Fontana wrote:
> add a new optional interface to CPUClass,
> which allows accelerators to extend the CPUClass
> with additional accelerator-specific initializations.
> 
> Signed-off-by: Claudio Fontana 
> ---
[...]
> +static void accel_init_cpu_int_aux(ObjectClass *klass, void *opaque)
> +{
> +CPUClass *cc = CPU_CLASS(klass);
> +AccelCPUClass *accel_cpu_interface = opaque;
> +
> +cc->accel_cpu_interface = accel_cpu_interface;
> +if (accel_cpu_interface->cpu_class_init) {
> +accel_cpu_interface->cpu_class_init(cc);
> +}
> +}

So, now that the approach we're following to trigger the
accel_init_cpu*() call is less controversial (thanks for your
patience!), we can try to address the monkey patching issue:

Monkey patching classes like this is acceptable as an initial
solution, but I'd like us to have a plan to eventually get rid of
it.  Monkey patching CPU classes makes querying of CPU model
information less predictable and subtly dependent on QEMU
initialization state.

Removing CPUClass.accel_cpu_interface may be easy, because it
should be possible to just call current_accel() when realizing
CPUs.  Getting rid of CPUClass.cpu_class_init might be more
difficult, depending on what the ->cpu_class_init() function is
doing.

> +
> +/* initialize the arch-specific accel CpuClass interfaces */
> +static void accel_init_cpu_interfaces(AccelClass *ac, const char *cpu_type)
> +{
> +const char *ac_name; /* AccelClass name */
> +char *acc_name;  /* AccelCPUClass name */
> +ObjectClass *acc;/* AccelCPUClass */
> +
> +ac_name = object_class_get_name(OBJECT_CLASS(ac));
> +g_assert(ac_name != NULL);
> +
> +acc_name = g_strdup_printf("%s-%s", ac_name, CPU_RESOLVING_TYPE);
> +acc = object_class_by_name(acc_name);
> +g_free(acc_name);
> +
> +if (acc) {
> +object_class_foreach(accel_init_cpu_int_aux, cpu_type, false, acc);
> +}
> +}
> +
>  void accel_init_interfaces(AccelClass *ac, const char *cpu_type)
>  {
>  #ifndef CONFIG_USER_ONLY
>  accel_init_ops_interfaces(ac);
>  #endif /* !CONFIG_USER_ONLY */
> +
> +accel_init_cpu_interfaces(ac, cpu_type);
>  }
[...]

-- 
Eduardo

Re: [PATCH v2] tests/acceptance: add a test for devices on s390x

[Cornelia Huck]

On Fri, 27 Nov 2020 14:00:16 -0300
Wainer dos Santos Moschetta  wrote:

> Hi,
> 
> On 11/26/20 10:01 AM, Cornelia Huck wrote:
> > This adds a very basic test for checking that we present devices
> > in a way that Linux can consume: boot with both virtio-net-ccw and
> > virtio-net-pci attached and then verify that Linux is able to see
> > and detect these devices.
> >
> > Signed-off-by: Cornelia Huck 
> > ---
> > RFC->v2:
> >   - use a newer kernel that uses the uid in zpci address generation
> >   - add a zpci device to specify a uid
> >   - increase timeout
> >   - tweak naming
> >   - add a MAINTAINERS entry
> > ---
> >   MAINTAINERS |  1 +
> >   tests/acceptance/machine_s390_ccw_virtio.py | 70 +
> >   2 files changed, 71 insertions(+)
> >   create mode 100644 tests/acceptance/machine_s390_ccw_virtio.py
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index 68bc160f41bc..cc1c7c2ffed8 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -1426,6 +1426,7 @@ F: include/hw/s390x/
> >   F: hw/watchdog/wdt_diag288.c
> >   F: include/hw/watchdog/wdt_diag288.h
> >   F: default-configs/s390x-softmmu.mak
> > +F: tests/acceptance/machine_s390_ccw_virtio.py
> >   T: git https://github.com/cohuck/qemu.git s390-next
> >   T: git https://github.com/borntraeger/qemu.git s390-next
> >   L: qemu-s3...@nongnu.org
> > diff --git a/tests/acceptance/machine_s390_ccw_virtio.py 
> > b/tests/acceptance/machine_s390_ccw_virtio.py
> > new file mode 100644
> > index ..1f56be776c5f
> > --- /dev/null
> > +++ b/tests/acceptance/machine_s390_ccw_virtio.py
> > @@ -0,0 +1,70 @@
> > +# Functional test that boots an s390x Linux guest with ccw and PCI devices
> > +# attached and checks whether the devices are recognized by Linux
> > +#
> > +# Copyright (c) 2020 Red Hat, Inc.
> > +#
> > +# Author:
> > +#  Cornelia Huck 
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2 or
> > +# later.  See the COPYING file in the top-level directory.
> > +
> > +
> > +import os
> > +  
> 
> 
> Unused import.

Will remove.

> 
> 
> > +from avocado_qemu import Test
> > +from avocado_qemu import exec_command_and_wait_for_pattern
> > +from avocado_qemu import wait_for_console_pattern
> > +
> > +class S390CCWVirtioMachine(Test):
> > +KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
> > +
> > +def wait_for_console_pattern(self, success_message, vm=None):
> > +wait_for_console_pattern(self, success_message,
> > + failure_message='Kernel panic - not 
> > syncing',
> > + vm=vm)
> > +
> > +timeout = 120
> > +
> > +def test_s390x_devices(self):
> > +
> > +"""
> > +:avocado: tags=arch:s390x
> > +:avocado: tags=machine:s390-ccw-virtio
> > +"""
> > +
> > +kernel_url = ('https://snapshot.debian.org/archive/debian/'
> > +  '20201126T092837Z/dists/buster/main/installer-s390x/'
> > +  '20190702+deb10u6/images/generic/kernel.debian')
> > +kernel_hash = '5821fbee57d6220a067a8b967d24595621aa1eb6'
> > +kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
> > +
> > +initrd_url = ('https://snapshot.debian.org/archive/debian/'
> > +  '20201126T092837Z/dists/buster/main/installer-s390x/'
> > +  '20190702+deb10u6/images/generic/initrd.debian')
> > +initrd_hash = '81ba09c97bef46e8f4660ac25b4ac0a5be3a94d6'
> > +initrd_path = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
> > +
> > +self.vm.set_console()
> > +kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
> > +  'console=sclp0 root=/dev/ram0 BOOT_DEBUG=3')
> > +self.vm.add_args('-nographic',
> > + '-kernel', kernel_path,
> > + '-initrd', initrd_path,
> > + '-append', kernel_command_line,
> > + '-device', 'virtio-net-ccw,devno=fe.1.',
> > + '-device', 'zpci,uid=5,target=zzz',
> > + '-device', 'virtio-net-pci,id=zzz')
> > +self.vm.launch()
> > +
> > +shell_ready = "sh: can't access tty; job control turned off"
> > +self.wait_for_console_pattern(shell_ready)
> > +# first debug shell is too early, we need to wait for device 
> > detection
> > +exec_command_and_wait_for_pattern(self, 'exit', shell_ready)
> > +
> > +ccw_bus_id="0.1."
> > +pci_bus_id="0005:00:00.0"
> > +exec_command_and_wait_for_pattern(self, 'ls /sys/bus/ccw/devices/',
> > +  ccw_bus_id)
> > +exec_command_and_wait_for_pattern(self, 'ls /sys/bus/pci/devices/',
> > +  pci_bus_id)  
> 
> 
> I tested this test case on GitLab. It passed on the 
> 

Re: [PATCH v2] tests/acceptance: add a test for devices on s390x

[Wainer dos Santos Moschetta]

Hi,

On 11/26/20 10:01 AM, Cornelia Huck wrote:

This adds a very basic test for checking that we present devices
in a way that Linux can consume: boot with both virtio-net-ccw and
virtio-net-pci attached and then verify that Linux is able to see
and detect these devices.

Signed-off-by: Cornelia Huck 
---
RFC->v2:
  - use a newer kernel that uses the uid in zpci address generation
  - add a zpci device to specify a uid
  - increase timeout
  - tweak naming
  - add a MAINTAINERS entry
---
  MAINTAINERS |  1 +
  tests/acceptance/machine_s390_ccw_virtio.py | 70 +
  2 files changed, 71 insertions(+)
  create mode 100644 tests/acceptance/machine_s390_ccw_virtio.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 68bc160f41bc..cc1c7c2ffed8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1426,6 +1426,7 @@ F: include/hw/s390x/
  F: hw/watchdog/wdt_diag288.c
  F: include/hw/watchdog/wdt_diag288.h
  F: default-configs/s390x-softmmu.mak
+F: tests/acceptance/machine_s390_ccw_virtio.py
  T: git https://github.com/cohuck/qemu.git s390-next
  T: git https://github.com/borntraeger/qemu.git s390-next
  L: qemu-s3...@nongnu.org
diff --git a/tests/acceptance/machine_s390_ccw_virtio.py 
b/tests/acceptance/machine_s390_ccw_virtio.py
new file mode 100644
index ..1f56be776c5f
--- /dev/null
+++ b/tests/acceptance/machine_s390_ccw_virtio.py
@@ -0,0 +1,70 @@
+# Functional test that boots an s390x Linux guest with ccw and PCI devices
+# attached and checks whether the devices are recognized by Linux
+#
+# Copyright (c) 2020 Red Hat, Inc.
+#
+# Author:
+#  Cornelia Huck 
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+
+import os
+



Unused import.



+from avocado_qemu import Test
+from avocado_qemu import exec_command_and_wait_for_pattern
+from avocado_qemu import wait_for_console_pattern
+
+class S390CCWVirtioMachine(Test):
+KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
+
+def wait_for_console_pattern(self, success_message, vm=None):
+wait_for_console_pattern(self, success_message,
+ failure_message='Kernel panic - not syncing',
+ vm=vm)
+
+timeout = 120
+
+def test_s390x_devices(self):
+
+"""
+:avocado: tags=arch:s390x
+:avocado: tags=machine:s390-ccw-virtio
+"""
+
+kernel_url = ('https://snapshot.debian.org/archive/debian/'
+  '20201126T092837Z/dists/buster/main/installer-s390x/'
+  '20190702+deb10u6/images/generic/kernel.debian')
+kernel_hash = '5821fbee57d6220a067a8b967d24595621aa1eb6'
+kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
+
+initrd_url = ('https://snapshot.debian.org/archive/debian/'
+  '20201126T092837Z/dists/buster/main/installer-s390x/'
+  '20190702+deb10u6/images/generic/initrd.debian')
+initrd_hash = '81ba09c97bef46e8f4660ac25b4ac0a5be3a94d6'
+initrd_path = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
+
+self.vm.set_console()
+kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE +
+  'console=sclp0 root=/dev/ram0 BOOT_DEBUG=3')
+self.vm.add_args('-nographic',
+ '-kernel', kernel_path,
+ '-initrd', initrd_path,
+ '-append', kernel_command_line,
+ '-device', 'virtio-net-ccw,devno=fe.1.',
+ '-device', 'zpci,uid=5,target=zzz',
+ '-device', 'virtio-net-pci,id=zzz')
+self.vm.launch()
+
+shell_ready = "sh: can't access tty; job control turned off"
+self.wait_for_console_pattern(shell_ready)
+# first debug shell is too early, we need to wait for device detection
+exec_command_and_wait_for_pattern(self, 'exit', shell_ready)
+
+ccw_bus_id="0.1."
+pci_bus_id="0005:00:00.0"
+exec_command_and_wait_for_pattern(self, 'ls /sys/bus/ccw/devices/',
+  ccw_bus_id)
+exec_command_and_wait_for_pattern(self, 'ls /sys/bus/pci/devices/',
+  pci_bus_id)



I tested this test case on GitLab. It passed on the 
acceptance-system-centos job 
(https://gitlab.com/wainersm/qemu/-/jobs/877308808) and the pipeline as 
a whole passed (https://gitlab.com/wainersm/qemu/-/pipelines/77683).


So once the unused import is removed:

Reviewed-by: Wainer dos Santos Moschetta 

Re: [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework

[Peter Maydell]

On Fri, 27 Nov 2020 at 16:47, Eduardo Habkost  wrote:
> Do you know how the implicitly-accelerator-specific code is
> implemented?  PARange is in id_aa64mmfr0, correct?  I don't see
> any accel-specific code for initializing id_aa64mmfr0.

For TCG, the value of id_aa64mmfr0 is set by the per-cpu
init functions aarch64_a57_initfn(), aarch64_a72_initfn(), etc.

For KVM, if we're using "-cpu cortex-a53" or "-cpu cortex-a57"
these only work if the host CPU really is an A53 or A57, in
which case the reset value set by the initfn is correct.
In the more usual case of "-cpu host", we ask the kernel for
the ID register values in kvm_arm_get_host_cpu_features(),
which is part of the implementation of
kvm_arm_set_cpu_features_from_host(), which gets called
in aarch64_max_initfn() (inside a kvm_enabled() conditional).

So there is a *_enabled() check involved, which I hadn't
realised until I worked back up to where this stuff is called.

thanks
-- PMM

Re: [PATCH v2] tests/acceptance: add a test for devices on s390x

[Cornelia Huck]

On Thu, 26 Nov 2020 14:01:58 +0100
Cornelia Huck  wrote:

> This adds a very basic test for checking that we present devices
> in a way that Linux can consume: boot with both virtio-net-ccw and
> virtio-net-pci attached and then verify that Linux is able to see
> and detect these devices.
> 
> Signed-off-by: Cornelia Huck 
> ---
> RFC->v2:
>  - use a newer kernel that uses the uid in zpci address generation
>  - add a zpci device to specify a uid
>  - increase timeout
>  - tweak naming
>  - add a MAINTAINERS entry
> ---
>  MAINTAINERS |  1 +
>  tests/acceptance/machine_s390_ccw_virtio.py | 70 +
>  2 files changed, 71 insertions(+)
>  create mode 100644 tests/acceptance/machine_s390_ccw_virtio.py

Queued to s390-next.

I plan to add some more stuff on top (like checking the mac address, or
adding more devices), but that will have to wait until next week.

Re: [RFC] ich9:cpuhp: add support for cpu hot-unplug with SMI broadcast enabled

[Laszlo Ersek]

On 11/27/20 16:07, Igor Mammedov wrote:
> On Fri, 27 Nov 2020 15:48:34 +0100
> Laszlo Ersek  wrote:

>> The firmware logic needs to be aware of is_removing though, at least
>> understand the existence of this bit, as the "get pending" command
>> will report such CPUs too that only have is_removing set. Shouldn't
>> be a problem, we just have to recognize it.
>
> firmware shouldn't see bit #2 normally, it's cleared in AML during
> CSCN, right after remove Notify is sent to OSPM. I don't see a reason
> for firmware to use it, I'd just mask it out on firmware side if it
> messes logic.
>
> potentially if we have concurrent plug/unplug for several CPUs,
> firmware might see bit #2 which it should ignore, it's for OSPM
> consumption only.

Yes, that's what I meant.

Currently, inside the scanning loop of the QemuCpuhpCollectApicIds()
function in "OvmfPkg/CpuHotplugSmm/QemuCpuhp.c", there is no branch that
simply skips a CPU that was reported by QEMU_CPUHP_CMD_GET_PENDING. Now,
such a branch will be necessary.

This is what I mean (just for illustration):

$ git diff -b -U5

> diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h 
> b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
> index a34a6d3fae61..ddeef047c517 100644
> --- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
> +++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
> @@ -32,10 +32,11 @@
>
>  #define QEMU_CPUHP_R_CPU_STAT0x4
>  #define QEMU_CPUHP_STAT_ENABLEDBIT0
>  #define QEMU_CPUHP_STAT_INSERT BIT1
>  #define QEMU_CPUHP_STAT_REMOVE BIT2
> +#define QEMU_CPUHP_STAT_FIRMWARE_REMOVEBIT4
>
>  #define QEMU_CPUHP_RW_CMD_DATA   0x8
>
>  #define QEMU_CPUHP_W_CPU_SEL 0x0
>
> diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c 
> b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
> index 8d4a6693c8d6..9bff31628e61 100644
> --- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
> +++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
> @@ -258,35 +258,44 @@ QemuCpuhpCollectApicIds (
>DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: insert\n", 
> __FUNCTION__,
>  CurrentSelector));
>
>ExtendIds   = PluggedApicIds;
>ExtendCount = PluggedCount;
> -} else if ((CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
> -  DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: remove\n", 
> __FUNCTION__,
> -CurrentSelector));
> +} else if ((CpuStatus & QEMU_CPUHP_STAT_FIRMWARE_REMOVE) != 0) {
> +  DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: firmware remove\n",
> +__FUNCTION__, CurrentSelector));
>
>ExtendIds   = ToUnplugApicIds;
>ExtendCount = ToUnplugCount;
> +} else if ((CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
> +  DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: remove\n", 
> __FUNCTION__,
> +CurrentSelector));
> +
> +  ExtendIds   = NULL;
> +  ExtendCount = NULL;
>  } else {
>DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: no event\n",
>  __FUNCTION__, CurrentSelector));
>break;
>  }
>
> +ASSERT ((ExtendIds == NULL) == (ExtendCount == NULL));
> +if (ExtendIds != NULL) {
>//
> -// Save the APIC ID of the CPU with the pending event, to the 
> corresponding
> -// APIC ID array.
> +  // Save the APIC ID of the CPU with the pending event, to the
> +  // corresponding APIC ID array.
>//
>if (*ExtendCount == ApicIdCount) {
>  DEBUG ((DEBUG_ERROR, "%a: APIC ID array too small\n", __FUNCTION__));
>  return EFI_BUFFER_TOO_SMALL;
>}
>QemuCpuhpWriteCommand (MmCpuIo, QEMU_CPUHP_CMD_GET_ARCH_ID);
>NewApicId = QemuCpuhpReadCommandData (MmCpuIo);
>DEBUG ((DEBUG_VERBOSE, "%a: ApicId=" FMT_APIC_ID "\n", __FUNCTION__,
>  NewApicId));
>ExtendIds[(*ExtendCount)++] = NewApicId;
> +}
>
>  //
>  // We've processed the CPU with (known) pending events, but we must never
>  // clear events. Therefore we need to advance past this CPU manually;
>  // otherwise, QEMU_CPUHP_CMD_GET_PENDING would stick to the currently

Thanks
Laszlo

Re: ImageInfo oddities regarding compression

[Markus Armbruster]

Kevin Wolf  writes:

> Am 27.11.2020 um 13:21 hat Markus Armbruster geschrieben:
>> >> I fell down this (thankfully shallow) rabbit hole because we also have
>> >> 
>> >> { 'enum': 'MultiFDCompression',
>> >>   'data': [ 'none', 'zlib',
>> >> { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>> >> 
>> >> I wonder whether we could merge them into a common type.
>> 
>> Looks like we could: current code would never report the additional
>> value 'none'.  Introspection would show it, though.  Seems unlikely to
>> cause trouble.  Observation, not demand.
>
> Forgot to comment on this one...
>
> Technically we could probably, but would it make sense? Support for
> compression formats has to be implemented separately for both cases, so
> that they currently happen to support the same list is more of a
> coincidence.
>
> If we ever add a third compression format to qcow2, would we add the
> same format to migration, too, or would we split the schema into two
> types again?

I figure if a compression method is worth adding to one, it's probably
worth adding to the other.

Having two separate enums isn't too bad.  A third has been proposed[*],
but I hope we can reuse migration's existing enum there.


[*] [PATCH 1/6] migration: Add multi-thread compress method

Re: [PATCH v2 2/6] accel: accel_available() function

[Markus Armbruster]

Claudio Fontana  writes:

> On 11/27/20 3:45 PM, Markus Armbruster wrote:
>> Claudio Fontana  writes:
>> 
>>> On 11/26/20 10:48 PM, Eduardo Habkost wrote:
 On Thu, Nov 26, 2020 at 10:06:03PM +0100, Claudio Fontana wrote:
> On 11/26/20 3:25 PM, Paolo Bonzini wrote:
>> On 26/11/20 15:13, Claudio Fontana wrote:
>>> One option I see is simply to document the behavior where
>>> accel_available() is declared in accel.h (ie do not use in fast
>>> path), as well as in accel_find() actually, so that both accel_find()
>>> and accel_available() are avoided in fast path and avoid being called
>>> frequently at runtime.
>>>
>>> Another option could be to remove the allocation completely, and use
>>> for example accel_find(ACCEL_CLASS_NAME("tcg")), or another option
>>> again would be to remove the allocation and use either a fixed buffer
>>> + snprintf, or alloca -like builtin code to use the stack, ...
>>>
>>> Not a big deal, but with a general utility and short name like
>>> accel_available(name) it might be tempting to use this more in the
>>> future?
>>
>> I think it's just that the usecase is not that common.  "Is this 
>> accelerator compiled in the binary" is not something you need after 
>> startup (or if querying the monitor).
>>
>> Paolo
>>
>>
>
> A script that repeatedly uses the QMP interface to query for
> the status could generate fragmentation this way I think.

 Is this a problem?  Today, execution of a "query-kvm" command
 calls g_malloc() 37 times.

>>>
>>> Not ideal in my view, but not the end of the world either.
>> 
>> QMP's appetite for malloc is roughly comparable to a pig's for truffles.
>> 
>
> :-)
>
> Btw, do we have limits on the maximum size of these objects? I mean, a single 
> QMP command,
> a single QEMU object type name, etc?
>
> In this case we could do some overall improvement there, and might even avoid 
> some problems down the road..

We have limits, but they are not comprehensive.

The QMP client is trusted.  We don't try to guard against a malicious
QMP client.  We do try to guard against mistakes.

The JSON parser limits token size (in characters), expression size (in
tokens), and expression nesting depth.  This protects against a
malfunctioning QMP client.  The limits are ridiculously generous.

The QMP core limits the number of commands in flight per monitor to a
somewhat parsimonious 8-9 in-band commands, plus one out-of-band
command.  This protects against a QMP client sending commands faster
than we can execute them.

QMP output is buffered without limit.  When a (malfunctioning) QMP
client keeps sending commands without reading their output, QEMU keeps
buffering until it runs out of memory and crashes.

Re: [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework

[Peter Maydell]

On Fri, 27 Nov 2020 at 16:38, Peter Maydell  wrote:
> Having looked a bit more closely at some of the relevant target/arm
> code, I think the best approach is going to be that in virt.c
> we just check the PARange ID register field (probably via
> a convenience function that does the conversion of that to
> a nice number-of-bits return value; we might even have one
> already).

Ha, in fact we're already doing something quite close to this,
though instead of saying "decide whether to use highmem based
on the CPU's PA range" we go for "report error to user if PA
range is insufficient" and let the user pick some command line
options that disable highmem if they want:

if (aarch64 && vms->highmem) {
int requested_pa_size = 64 - clz64(vms->highest_gpa);
int pamax = arm_pamax(ARM_CPU(first_cpu));

if (pamax < requested_pa_size) {
error_report("VCPU supports less PA bits (%d) than "
 "requested by the memory map (%d)",
 pamax, requested_pa_size);
exit(1);
}
}

thanks
-- PMM

Re: [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework

[Eduardo Habkost]

On Fri, Nov 27, 2020 at 04:38:18PM +, Peter Maydell wrote:
> On Fri, 27 Nov 2020 at 16:26, Eduardo Habkost  wrote:
> >
> > On Thu, Nov 26, 2020 at 10:29:01PM +, Peter Maydell wrote:
> > > On Thu, 26 Nov 2020 at 22:14, Eduardo Habkost  wrote:
> > > > Direct checks for *_enabled() are a pain to clean up later when
> > > > we add support to new accelerators.  Can't this be implemented as
> > > > (e.g.) a AccelClass::max_physical_address_bits field?
> > >
> > > It's a property of the CPU (eg our emulated TCG CPUs may have
> > > varying supported numbers of physical address bits). So the
> > > virt board ought to look at the CPU, and the CPU should be
> > > set up with the right information for all of KVM, TCG, HVF
> > > (either a specific max_phys_addr_bits value or just ensure
> > > its ID_AA64MMFR0_EL1.PARange is right, not sure which would
> > > be easier/nicer).
> >
> > Agreed.
> >
> > My suggestion would still apply to the CPU code that will pick
> > the address size; ideally, accel-specific behaviour should be
> > represented as meaningful fields in AccelClass (either data or
> > virtual methods) instead of direct *_enabled() checks.
> 
> Having looked a bit more closely at some of the relevant target/arm
> code, I think the best approach is going to be that in virt.c
> we just check the PARange ID register field (probably via
> a convenience function that does the conversion of that to
> a nice number-of-bits return value; we might even have one
> already). KVM and TCG both already set that ID register field
> in the CPU struct correctly in their existing
> implicitly-accelerator-specific code; HVF needs to do the same.

Do you know how the implicitly-accelerator-specific code is
implemented?  PARange is in id_aa64mmfr0, correct?  I don't see
any accel-specific code for initializing id_aa64mmfr0.

-- 
Eduardo

Re: [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework

[Peter Maydell]

On Fri, 27 Nov 2020 at 16:26, Eduardo Habkost  wrote:
>
> On Thu, Nov 26, 2020 at 10:29:01PM +, Peter Maydell wrote:
> > On Thu, 26 Nov 2020 at 22:14, Eduardo Habkost  wrote:
> > > Direct checks for *_enabled() are a pain to clean up later when
> > > we add support to new accelerators.  Can't this be implemented as
> > > (e.g.) a AccelClass::max_physical_address_bits field?
> >
> > It's a property of the CPU (eg our emulated TCG CPUs may have
> > varying supported numbers of physical address bits). So the
> > virt board ought to look at the CPU, and the CPU should be
> > set up with the right information for all of KVM, TCG, HVF
> > (either a specific max_phys_addr_bits value or just ensure
> > its ID_AA64MMFR0_EL1.PARange is right, not sure which would
> > be easier/nicer).
>
> Agreed.
>
> My suggestion would still apply to the CPU code that will pick
> the address size; ideally, accel-specific behaviour should be
> represented as meaningful fields in AccelClass (either data or
> virtual methods) instead of direct *_enabled() checks.

Having looked a bit more closely at some of the relevant target/arm
code, I think the best approach is going to be that in virt.c
we just check the PARange ID register field (probably via
a convenience function that does the conversion of that to
a nice number-of-bits return value; we might even have one
already). KVM and TCG both already set that ID register field
in the CPU struct correctly in their existing
implicitly-accelerator-specific code; HVF needs to do the same.

thanks
-- PMM

Re: [PATCH for-6.0 6/6] qapi: Deprecate 'query-kvm'

[Peter Krempa]

On Fri, Nov 27, 2020 at 15:53:16 +, Daniel Berrange wrote:
> On Fri, Nov 27, 2020 at 01:18:09PM +0100, Peter Krempa wrote:
> > On Fri, Nov 27, 2020 at 14:45:12 +0300, Roman Bolshakov wrote:

[...]

> IIUC, this all relies on us importing a dump of the latest QEMU
> capabilities into the libvirt test suite.
> 
> Most of the capabilities we import are the release version, but
> we also periodically import the git snapshot capabilities and
> will refresh them until GA of QEMU.
> 
> Could we arrange it so that libvirt only reports an error for use
> of deprecated interfaces when testing against the GA capabilities
> dump. If testing against QEMU git snapshot capabilities, we can
> emit a warning only.
> 
> That way, we'll have a grace period in which libvirt can see the
> warning from tests, and we would only need to add an override
> to silence it once we import the GA capabilities, at which time
> it is safe to implement the new solution too.

As I've noted in my reply to Markus, in many cases we develop features
on top of unreleased qemu features. I don't really want to throw a
spanner into that process.

While deprecating old interfaces which are changed would lead to
regression in behaviour rather than releasing a broken feature if
changes are done in the API of a new feature the situation is slightly
worse, but it's also a very rare occurence.

I want to basically just an agreement that after such a thing is done
libvirt will be notified. We'll still try to develop replacements sooner
which can actually catch problems in the new features sooner than they
are released in qemu.

Re: [PATCH for-6.0 6/6] qapi: Deprecate 'query-kvm'

[Peter Krempa]

On Fri, Nov 27, 2020 at 16:44:05 +0100, Markus Armbruster wrote:
> Peter Krempa  writes:
> 
> > On Fri, Nov 27, 2020 at 14:45:12 +0300, Roman Bolshakov wrote:
> >> On Fri, Nov 27, 2020 at 12:21:54PM +0100, Peter Krempa wrote:

 [...]

> > As you can see this has an issue when we have to add support for a
> > unreleased interface, which may change during the dev cycle or plainly
> > forget that something got deprecated as we've already added an override.
> >
> > This mainly comes from libvirt trying to keep on top of the changes so
> > we refresh the QMP schema during qemu's dev cycle multiple times.
> >
> > Obviously the argument that we try to depend on unreleased functionality
> > can be used, but that would be to detrement of progress IMO.
> 
> I understand your concerns.
> 
> We have a somewhat similar problem in QEMU: there's nothing to remind us
> later on that the old interface still needs to be deprecated.

Oh, yes. That's basically the same thing.

The thing is that changes to the new interface are very rare, but very
real. Since I don't really want to increase the burden for any normal
scenario.

I'd also very much like to keep libvirt pulling in snapshots of qemu's
capabilities/qapi schema etc throughout the development cycle. It allows
us to adapt faster and develop features simultaneously.

This unfortunately undermines my own arguments partially as libvirt
regularly develops features on top of unreleased qemu features so we are
regularly risking very similar circumstances. The small but important
difference though is, that releasing a broken feature is not as bad as
breaking an existing feature.

As a conclusion of the above I'd basically prefer a sort of gentleman's
agreement, that new APIs become 'somewhat' stable at the moment the
commit deprecating the old interface hits upstream.

The 'somewhat'-stable API would mean that any changes to the new API
should be consulted with libvirt so that we can either give a go-ahead
that we didn't adapt yet, disable the adaptation until the changes can
be done, or another compromise depending on what's the state.

I know it's hard to enforce, but probably the cheapest in terms of
drawbacks any other solution would be.

I'll probably keep notifying patchsets which implement and deprecate old
api at the same time to keep in mind that we need to be kept in touch,
but I really don't want to impose any kind of extra process to
development on either side.

Re: [PATCH 8/8] hw/arm/virt: Disable highmem when on hypervisor.framework

[Eduardo Habkost]

On Thu, Nov 26, 2020 at 10:29:01PM +, Peter Maydell wrote:
> On Thu, 26 Nov 2020 at 22:14, Eduardo Habkost  wrote:
> >
> > On Thu, Nov 26, 2020 at 10:50:17PM +0100, Alexander Graf wrote:
> > > The Apple M1 only supports up to 36 bits of physical address space. That
> > > means we can not fit the 64bit MMIO BAR region into our address space.
> > >
> > > To fix this, let's not expose a 64bit MMIO BAR region when running on
> > > Apple Silicon.
> > >
> > > I have not been able to find a way to enumerate that easily, so let's
> > > just assume we always have that little PA space on hypervisor.framework
> > > systems.
> > >
> > > Signed-off-by: Alexander Graf 
> > > ---
> > >  hw/arm/virt.c | 9 +
> > >  1 file changed, 9 insertions(+)
> > >
> > > diff --git a/hw/arm/virt.c b/hw/arm/virt.c
> > > index 27dbeb549e..d74053ecd4 100644
> > > --- a/hw/arm/virt.c
> > > +++ b/hw/arm/virt.c
> > > @@ -45,6 +45,7 @@
> > >  #include "hw/display/ramfb.h"
> > >  #include "net/net.h"
> > >  #include "sysemu/device_tree.h"
> > > +#include "sysemu/hvf.h"
> > >  #include "sysemu/numa.h"
> > >  #include "sysemu/runstate.h"
> > >  #include "sysemu/sysemu.h"
> > > @@ -1746,6 +1747,14 @@ static void machvirt_init(MachineState *machine)
> > >  unsigned int smp_cpus = machine->smp.cpus;
> > >  unsigned int max_cpus = machine->smp.max_cpus;
> > >
> > > +/*
> > > + * On Hypervisor.framework capable systems, we only have 36 bits of 
> > > PA
> > > + * space, which is not enough to fit a 64bit BAR space
> > > + */
> > > +if (hvf_enabled()) {
> > > +vms->highmem = false;
> > > +}
> >
> > Direct checks for *_enabled() are a pain to clean up later when
> > we add support to new accelerators.  Can't this be implemented as
> > (e.g.) a AccelClass::max_physical_address_bits field?
> 
> It's a property of the CPU (eg our emulated TCG CPUs may have
> varying supported numbers of physical address bits). So the
> virt board ought to look at the CPU, and the CPU should be
> set up with the right information for all of KVM, TCG, HVF
> (either a specific max_phys_addr_bits value or just ensure
> its ID_AA64MMFR0_EL1.PARange is right, not sure which would
> be easier/nicer).

Agreed.

My suggestion would still apply to the CPU code that will pick
the address size; ideally, accel-specific behaviour should be
represented as meaningful fields in AccelClass (either data or
virtual methods) instead of direct *_enabled() checks.

-- 
Eduardo

Re: [PATCH for-6.0 6/6] qapi: Deprecate 'query-kvm'

[Daniel P . Berrangé]

On Fri, Nov 27, 2020 at 01:18:09PM +0100, Peter Krempa wrote:
> On Fri, Nov 27, 2020 at 14:45:12 +0300, Roman Bolshakov wrote:
> > On Fri, Nov 27, 2020 at 12:21:54PM +0100, Peter Krempa wrote:
> > > On Fri, Nov 27, 2020 at 10:50:59 +, Daniel Berrange wrote:
> > > > Copying libvir-list for the deprecation warning.
> > > > 
> > > > 
> > > > On Mon, Nov 16, 2020 at 04:10:11PM +0300, Roman Bolshakov wrote:
> > > > > 'query-accel' QMP command should be used instead.
> > > > > 
> > > > > Signed-off-by: Roman Bolshakov 
> > > > > ---
> 
> [...]
> 
> > > We try hard to stay on top of such changes by using the new interface as
> > > soon as possible, but that is very hard if the replacement changes
> > > during the dev cycle.
> > > 
> > 
> > I see, thanks for the explanation! Perhaps I'll drop deprecation from
> > the series to avoid the issue.
> > 
> > Then as soon as libvirt gets support for queyring accels we might
> > consider deprecation again.
> 
> I don't want to imply that it's entirely necessary to postpone it, but
> in such cases the new API which was added to replace the old one needs
> to be considered a bit more strongly until the release.
> 
> The main reason for this is that libvirt has tests whether it uses any
> deprecated interface. If anything is marked as deprecated and our tests
> flag it, we add an override. Usually the override is added in the same
> patchset which actually implements the new approach.
> 
> We obviously can add an override and then wait for the supported
> interface, but once the override is added there's nothing to remind us
> later on, so I generally like to have everything in one series.

IIUC, this all relies on us importing a dump of the latest QEMU
capabilities into the libvirt test suite.

Most of the capabilities we import are the release version, but
we also periodically import the git snapshot capabilities and
will refresh them until GA of QEMU.

Could we arrange it so that libvirt only reports an error for use
of deprecated interfaces when testing against the GA capabilities
dump. If testing against QEMU git snapshot capabilities, we can
emit a warning only.

That way, we'll have a grace period in which libvirt can see the
warning from tests, and we would only need to add an override
to silence it once we import the GA capabilities, at which time
it is safe to implement the new solution too.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

QEMU Advent Calendar 2020 Call for Images

[Eldon Stegall]

Hi,
QEMU Advent Calendar 2020 is around the corner and we are looking for
volunteers to contribute disk images that showcase something cool, bring
back retro computing memories, or simply entertain with a puzzle or game.

QEMU Advent Calendar publishes a QEMU disk image each day from
December 1-24. Each image is a surprise designed to delight an audience
consisting of the QEMU community and beyond. You can see previous
years here:

  https://www.qemu-advent-calendar.org/

You can help us make this year's calendar awesome by:
 * Sending disk images ( or links to larger images )
 * Replying with ideas for disk images (reply off-list to avoid spoilers!)

If you have an idea after the start of the advent, go ahead and send it. We may
find space to include it, or go ahead and get a jump on 2021!

Here are the requirements for disk images:
 * Content must be freely redistributable (i.e. no proprietary
   license that prevents distribution). For GPL based software,
   you need to provide the source code, too.
 * Provide a name and a short description of the disk image
   (e.g. with hints on what to try)
 * Provide a ./run shell script that prints out the name and
   description/hints and launches QEMU
 * Provide a 320x240 screenshot/image/logo for the website
 * Size should be ideally under 100 MB per disk image
   (but if some few images are bigger, that should be OK, too)

Check out this disk image as an example of how to distribute an image:
https://www.qemu-advent-calendar.org/2018/download/day24.tar.xz

PS: QEMU Advent Calendar is a secular calendar (not
religious). The idea is to create a fun experience for the QEMU
community which can be shared with everyone. You don't need
to celebrate Christmas or another religious festival to participate!

Thanks, and best wishes!
Eldon

Re: [PATCH v4 0/6] UFFD write-tracking migration/snapshots

[Peter Xu]

On Fri, Nov 27, 2020 at 01:00:48PM +0300, Andrey Gruzdev wrote:
> On 27.11.2020 12:49, Peter Krempa wrote:
> > On Fri, Nov 27, 2020 at 11:21:39 +0300, Andrey Gruzdev wrote:
> > > On 26.11.2020 18:47, Peter Krempa wrote:
> > > > On Thu, Nov 26, 2020 at 18:17:28 +0300, Andrey Gruzdev via wrote:
> > > > > This patch series is a kind of 'rethinking' of Denis Plotnikov's 
> > > > > ideas he's
> > > > > implemented in his series '[PATCH v0 0/4] migration: add background 
> > > > > snapshot'.
> > > > > 
> > > > > Currently the only way to make (external) live VM snapshot is using 
> > > > > existing
> > > > > dirty page logging migration mechanism. The main problem is that it 
> > > > > tends to
> > > > > produce a lot of page duplicates while running VM goes on updating 
> > > > > already
> > > > > saved pages. That leads to the fact that vmstate image size is 
> > > > > commonly several
> > > > > times bigger then non-zero part of virtual machine's RSS. Time 
> > > > > required to
> > > > > converge RAM migration and the size of snapshot image severely depend 
> > > > > on the
> > > > > guest memory write rate, sometimes resulting in unacceptably long 
> > > > > snapshot
> > > > > creation time and huge image size.
> > > > > 
> > > > > This series propose a way to solve the aforementioned problems. This 
> > > > > is done
> > > > > by using different RAM migration mechanism based on UFFD write 
> > > > > protection
> > > > > management introduced in v5.7 kernel. The migration strategy is to 
> > > > > 'freeze'
> > > > > guest RAM content using write-protection and iteratively release 
> > > > > protection
> > > > > for memory ranges that have already been saved to the migration 
> > > > > stream.
> > > > > At the same time we read in pending UFFD write fault events and save 
> > > > > those
> > > > > pages out-of-order with higher priority.
> > > > 
> > > > This sounds amazing! Based on your description I assume that the final
> > > > memory image contains state image from the beginning of the migration.
> > > > 
> > > > This would make it desirable for the 'migrate' qmp command to be used as
> > > > part of a 'transaction' qmp command so that we can do an instant disk
> > > > and memory snapshot without any extraneous pausing of the VM.
> > > > 
> > > > I'll have a look at using this mechanism in libvirt natively.
> > > > 
> > > 
> > > Correct, the final image contains state at the beginning of migration.
> > > 
> > > So far, if I'm not missing something about libvirt, for external snapshot
> > > creation it performs a sequence like that:
> > > migrate(fd)->transaction(blockdev-snapshot-all)->cont.
> > > 
> > > So, in case 'background-snapshot' capability is enabled, the sequence 
> > > would
> > > change to:
> > > stop->transaction(blockdev-snapshot-all)->migrate(fd).
> > > With background snapshot migration it will finish with VM running so 
> > > there's
> > > not need to 'cont' here.
> > 
> > Yes, that's correct.
> > 
> > The reason I've suggested that 'migrate' being part of a 'transaction'
> > is that it would remove the need to stop it for the disk snapshot part.
> > 
> 
> Hmm, I believe stopping VM for a short time is unavoidable to keep saved
> device state consistent with blockdev snapshot base.. May be I've missed
> something but it seems logical.

I guess PeterK meant an explicit stop vm command, rather than the stop in this
series that should be undetectable from the guest.  It would be definitely
great if PeterK could quickly follow up with this on libvirt. :)

One overall comment (before go into details) is that please remember to start
using "migration: " prefixes in patch subjects from the next version.

Thanks,

-- 
Peter Xu

[RFC PATCH-for-5.2 2/2] net: Assert no packet bigger than NET_BUFSIZE is queued

[Philippe Mathieu-Daudé]

Ensure no packet bigger then NET_BUFSIZE is queued via
qemu_net_queue_append*() by adding assertions.

Signed-off-by: Philippe Mathieu-Daudé 
---
 net/queue.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/net/queue.c b/net/queue.c
index 221a1c87961..94b98b19ef9 100644
--- a/net/queue.c
+++ b/net/queue.c
@@ -102,6 +102,8 @@ static void qemu_net_queue_append(NetQueue *queue,
 if (queue->nq_count >= queue->nq_maxlen && !sent_cb) {
 return; /* drop if queue full and no callback */
 }
+
+assert(size <= NET_BUFSIZE);
 packet = g_malloc(sizeof(NetPacket) + size);
 packet->sender = sender;
 packet->flags = flags;
@@ -131,6 +133,7 @@ void qemu_net_queue_append_iov(NetQueue *queue,
 max_len += iov[i].iov_len;
 }
 
+assert(max_len <= NET_BUFSIZE);
 packet = g_malloc(sizeof(NetPacket) + max_len);
 packet->sender = sender;
 packet->sent_cb = sent_cb;
-- 
2.26.2

Re: [RFC PATCH-for-5.2 0/2] net: Do not accept packets with invalid huge size

[Philippe Mathieu-Daudé]

On 11/27/20 4:45 PM, Philippe Mathieu-Daudé wrote:
> Hi,
> 
> This is a simple attempt to avoid the following pattern:
> 
>   ssize_t pkt_size = get_pkt_size(); // returns errno

Sorry, I meant: returns "-errno" (< 0).

> 
>   // no check
> 
>   send_packet(size_t size=pkt_size); // size casted to unsigned
>  // -> overflow
> 
> Regards,
> 
> Phil.
> 
> Philippe Mathieu-Daudé (2):
>   net: Do not accept packets bigger then NET_BUFSIZE
>   net: Assert no packet bigger than NET_BUFSIZE is queued
> 
>  net/net.c   | 4 
>  net/queue.c | 7 +++
>  2 files changed, 11 insertions(+)
> 

[RFC PATCH-for-5.2 1/2] net: Do not accept packets bigger then NET_BUFSIZE

[Philippe Mathieu-Daudé]

Do not allow qemu_send_packet*() and qemu_net_queue_send()
functions to accept packets bigger then NET_BUFSIZE.

Signed-off-by: Philippe Mathieu-Daudé 
---
We have to put a limit somewhere. NET_BUFSIZE is defined as:

 /* Maximum GSO packet size (64k) plus plenty of room for
  * the ethernet and virtio_net headers
  */
 #define NET_BUFSIZE (4096 + 65536)

If we do want to accept bigger packets (i.e. multiple GSO packets
in a IOV), we could use INT32_MAX as limit...
---
 net/net.c   | 4 
 net/queue.c | 4 
 2 files changed, 8 insertions(+)

diff --git a/net/net.c b/net/net.c
index 6a2c3d95670..f29bfac2b11 100644
--- a/net/net.c
+++ b/net/net.c
@@ -644,6 +644,10 @@ static ssize_t 
qemu_send_packet_async_with_flags(NetClientState *sender,
 qemu_hexdump(stdout, "net", buf, size);
 #endif
 
+if (size > NET_BUFSIZE) {
+return -1;
+}
+
 if (sender->link_down || !sender->peer) {
 return size;
 }
diff --git a/net/queue.c b/net/queue.c
index 19e32c80fda..221a1c87961 100644
--- a/net/queue.c
+++ b/net/queue.c
@@ -191,6 +191,10 @@ ssize_t qemu_net_queue_send(NetQueue *queue,
 {
 ssize_t ret;
 
+if (size > NET_BUFSIZE) {
+return -1;
+}
+
 if (queue->delivering || !qemu_can_send_packet(sender)) {
 qemu_net_queue_append(queue, sender, flags, data, size, sent_cb);
 return 0;
-- 
2.26.2

Re: [RFC PATCH 00/27] vDPA software assisted live migration

[Stefano Garzarella]

On Fri, Nov 20, 2020 at 07:50:38PM +0100, Eugenio Pérez wrote:

This series enable vDPA software assisted live migration for vhost-net
devices. This is a new method of vhost devices migration: Instead of
relay on vDPA device's dirty logging capability, SW assisted LM
intercepts dataplane, forwarding the descriptors between VM and device.

In this migration mode, qemu offers a new vring to the device to
read and write into, and disable vhost notifiers, processing guest and
vhost notifications in qemu. On used buffer relay, qemu will mark the
dirty memory as with plain virtio-net devices. This way, devices does
not need to have dirty page logging capability.

This series is a POC doing SW LM for vhost-net devices, which already
have dirty page logging capabilities. None of the changes have actual
effect with current devices until last two patches (26 and 27) are
applied, but they can be rebased on top of any other. These checks the
device to meet all requirements, and disable vhost-net devices logging
so migration goes through SW LM. This last patch is not meant to be
applied in the final revision, it is in the series just for testing
purposes.

For use SW assisted LM these vhost-net devices need to be instantiated:
* With IOMMU (iommu_platform=on,ats=on)
* Without event_idx (event_idx=off)

Just the notification forwarding (with no descriptor relay) can be
achieved with patches 7 and 9, and starting migration. Partial applies
between 13 and 24 will not work while migrating on source, and patch
25 is needed for the destination to resume network activity.

It is based on the ideas of DPDK SW assisted LM, in the series of
DPDK's https://patchwork.dpdk.org/cover/48370/ .

Comments are welcome.


Hi Eugenio,
I took a look and the idea of the shadow queue I think is the right way.
It's very similar to what we thought with Stefan for io_uring 
passthrough and vdpa-blk.


IIUC, when the migrations starts, the notifications from the guest to 
vhost are disabled, so QEMU starts to intercept them through the 
custom_handler installed in virtio-net (we need to understand how to 
generalize this).
At this point QEMU starts to use the shadows queues and exposes them to 
vhost.
The opposite is done for vhost to guest notifications, where 
vhost_handle_call is installed to masked_notifier to intercept the 
notification.


I hope to give better feedback when I get a complete overview ;-)

Anyway, as Jason suggested, we should split this series, so maybe we can 
merge some preparations patches (e.g. 1, 11, 21, 22) regardless the 
other patches.


Thanks,
Stefano

[RFC PATCH-for-5.2 0/2] net: Do not accept packets with invalid huge size

[Philippe Mathieu-Daudé]

Hi,

This is a simple attempt to avoid the following pattern:

  ssize_t pkt_size = get_pkt_size(); // returns errno

  // no check

  send_packet(size_t size=pkt_size); // size casted to unsigned
 // -> overflow

Regards,

Phil.

Philippe Mathieu-Daudé (2):
  net: Do not accept packets bigger then NET_BUFSIZE
  net: Assert no packet bigger than NET_BUFSIZE is queued

 net/net.c   | 4 
 net/queue.c | 7 +++
 2 files changed, 11 insertions(+)

-- 
2.26.2

Re: [PATCH for-6.0 6/6] qapi: Deprecate 'query-kvm'

[Markus Armbruster]

Peter Krempa  writes:

> On Fri, Nov 27, 2020 at 14:45:12 +0300, Roman Bolshakov wrote:
>> On Fri, Nov 27, 2020 at 12:21:54PM +0100, Peter Krempa wrote:
>> > On Fri, Nov 27, 2020 at 10:50:59 +, Daniel Berrange wrote:
>> > > Copying libvir-list for the deprecation warning.
>> > > 
>> > > 
>> > > On Mon, Nov 16, 2020 at 04:10:11PM +0300, Roman Bolshakov wrote:
>> > > > 'query-accel' QMP command should be used instead.
>> > > > 
>> > > > Signed-off-by: Roman Bolshakov 
>> > > > ---
>
> [...]
>
>> > We try hard to stay on top of such changes by using the new interface as
>> > soon as possible, but that is very hard if the replacement changes
>> > during the dev cycle.
>> > 
>> 
>> I see, thanks for the explanation! Perhaps I'll drop deprecation from
>> the series to avoid the issue.
>> 
>> Then as soon as libvirt gets support for queyring accels we might
>> consider deprecation again.
>
> I don't want to imply that it's entirely necessary to postpone it, but
> in such cases the new API which was added to replace the old one needs
> to be considered a bit more strongly until the release.
>
> The main reason for this is that libvirt has tests whether it uses any
> deprecated interface. If anything is marked as deprecated and our tests
> flag it, we add an override. Usually the override is added in the same
> patchset which actually implements the new approach.
>
> We obviously can add an override and then wait for the supported
> interface, but once the override is added there's nothing to remind us
> later on, so I generally like to have everything in one series.
>
> As you can see this has an issue when we have to add support for a
> unreleased interface, which may change during the dev cycle or plainly
> forget that something got deprecated as we've already added an override.
>
> This mainly comes from libvirt trying to keep on top of the changes so
> we refresh the QMP schema during qemu's dev cycle multiple times.
>
> Obviously the argument that we try to depend on unreleased functionality
> can be used, but that would be to detrement of progress IMO.

I understand your concerns.

We have a somewhat similar problem in QEMU: there's nothing to remind us
later on that the old interface still needs to be deprecated.

Here's an idea.  Keep a list of things to deprecate in the repository.
Instead of deprecating right away, we add to the list.  When soft freeze
comes, we go through the list and decide: either deprecate now (and
delete from the list), or postpone deprecation to the next release (and
keep it on the list).

Would that work for libvirt?

There's still a risk of us forgetting about the list.  Perhaps keeping a
reminder on the Planning/x.y wiki page could help.  Peter (Maydell), do
you have a check list for the various milestones?