[Qemu-devel] [PATCH] tap: close fd conditionally when error occured

[Jay Zhou]

If netdev_add tap,id=net0,vhost=on failed in net_init_tap_one(),
the followed up device_add virtio-net-pci,netdev=net0 will fail
too, prints:

  TUNSETOFFLOAD ioctl() failed: Bad file descriptor TUNSETOFFLOAD
  ioctl() failed: Bad file descriptor

The reason is that the fd of tap is closed when error occured after
calling net_init_tap_one().

I think the fd should be closed in these two case:
  1.tap_set_sndbuf() failed
  2.tap_set_sndbuf() succeeded but vhost failed to initialize with
vhostforce flag on
Meanwhile, the fd should not be closed just because vhost failed to
initialize but without vhostforce flag. So the followed up device_add
can fall back to userspace virtio successfully.

Suggested-by: Michael S. Tsirkin 
Suggested-by: Igor Mammedov 
Suggested-by: Jason Wang 
Signed-off-by: Jay Zhou 
---
 net/tap.c | 18 +-
 1 file changed, 13 insertions(+), 5 deletions(-)

diff --git a/net/tap.c b/net/tap.c
index 979e622..3ed72eb 100644
--- a/net/tap.c
+++ b/net/tap.c
@@ -651,6 +651,9 @@ static void net_init_tap_one(const NetdevTapOptions *tap, 
NetClientState *peer,
 tap_set_sndbuf(s->fd, tap, );
 if (err) {
 error_propagate(errp, err);
+if (!tap->has_fd && !tap->has_fds) {
+close(fd);
+}
 return;
 }
 
@@ -687,14 +690,14 @@ static void net_init_tap_one(const NetdevTapOptions *tap, 
NetClientState *peer,
 vhostfd = monitor_fd_param(cur_mon, vhostfdname, );
 if (vhostfd == -1) {
 error_propagate(errp, err);
-return;
+goto cleanup;
 }
 } else {
 vhostfd = open("/dev/vhost-net", O_RDWR);
 if (vhostfd < 0) {
 error_setg_errno(errp, errno,
  "tap: open vhost char device failed");
-return;
+goto cleanup;
 }
 fcntl(vhostfd, F_SETFL, O_NONBLOCK);
 }
@@ -704,11 +707,18 @@ static void net_init_tap_one(const NetdevTapOptions *tap, 
NetClientState *peer,
 if (!s->vhost_net) {
 error_setg(errp,
"vhost-net requested but could not be initialized");
-return;
+goto cleanup;
 }
 } else if (vhostfdname) {
 error_setg(errp, "vhostfd(s)= is not valid without vhost");
 }
+
+cleanup:
+if (!tap->has_fd && !tap->has_fds && tap->has_vhostforce &&
+tap->vhostforce) {
+close(fd);
+}
+return;
 }
 
 static int get_fds(char *str, char *fds[], int max)
@@ -877,7 +887,6 @@ free_fail:
  vnet_hdr, fd, );
 if (err) {
 error_propagate(errp, err);
-close(fd);
 return -1;
 }
 } else {
@@ -916,7 +925,6 @@ free_fail:
  vhostfdname, vnet_hdr, fd, );
 if (err) {
 error_propagate(errp, err);
-close(fd);
 return -1;
 }
 }
-- 
1.8.3.1

Re: [Qemu-devel] [RESEND PATCH for-2.12 0/2] ACPI/unit-test: Add a testcase

[Dou Liyang]

At 01/12/2018 03:23 PM, Dou Liyang wrote:

[...]


BTW, I used "make TEST_ACPI_REBUILD_AML=1 check" to create the DSDT
file. Is it correct?


This sounds correct to me.  Igor, can you confirm?

It's will rebuild/update reference tables.

I'm for merging it early and tracking/fixing issue Peter saw during 
this dev cycle.





Send an ack then?


Hi Michael,

I saw Igor gave the Reviewed-by at 21 Dec 2017

                                     ^^ Oops! Sorry, it's 27 Dec.


What is the state of this patchset?

I rebase and retest it again, Do I need to send it again?

Thanks,
 dou

Re: [Qemu-devel] [RESEND PATCH for-2.12 0/2] ACPI/unit-test: Add a testcase

[Dou Liyang]

[...]


BTW, I used "make TEST_ACPI_REBUILD_AML=1 check" to create the DSDT
file. Is it correct?


This sounds correct to me.  Igor, can you confirm?

It's will rebuild/update reference tables.

I'm for merging it early and tracking/fixing issue Peter saw during this dev 
cycle.




Send an ack then?


Hi Michael,

I saw Igor gave the Reviewed-by at 21 Dec 2017.

What is the state of this patchset?

I rebase and retest it again, Do I need to send it again?

Thanks,
dou

Re: [Qemu-devel] [RFC PATCH v3 30/30] replay: don't process async events when warping the clock

[Pavel Dovgalyuk]

> From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo 
> Bonzini
> On 11/01/2018 09:27, Pavel Dovgalyuk wrote:
> > Virtual clock is wapred from iothread and vcpu thread. When the hardware
> > events associated with warp checkpoint, then interrupt delivering may be
> > non-deterministic if checkpoint is processed in different threads in record
> > and replay.
> > This patch disables event processing for clock warp checkpoint and leaves
> > all hardware events to other checkpoints (e.g., virtual clock).
> >
> > Signed-off-by: Pavel Dovgalyuk 
> > ---
> >  replay/replay.c |7 ++-
> >  1 file changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/replay/replay.c b/replay/replay.c
> > index b9c496a..cc43c38 100644
> > --- a/replay/replay.c
> > +++ b/replay/replay.c
> > @@ -211,7 +211,12 @@ bool replay_checkpoint(ReplayCheckpoint checkpoint)
> >  } else if (replay_mode == REPLAY_MODE_RECORD) {
> >  g_assert(replay_mutex_locked());
> >  replay_put_event(EVENT_CHECKPOINT + checkpoint);
> > -replay_save_events(checkpoint);
> > +/* This checkpoint belongs to several threads.
> > +   Processing events from different threads is
> > +   non-deterministic */
> > +if (checkpoint != CHECKPOINT_CLOCK_WARP_START) {
> > +replay_save_events(checkpoint);
> > +}
> >  res = true;
> >  }
> >  out:
> >
> 
> Please add an assertion in replay_read_event that the read_checkpoint is
> never CHECKPOINT_CLOCK_WARP_START. 

Ok.
 
> Why is the checkpoint still needed?

Because it synchronizes the warp itself.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC v6 25/27] docs: update QMP documents for OOB commands

[Peter Xu]

On Tue, Jan 09, 2018 at 02:52:25PM +, Stefan Hajnoczi wrote:
> On Tue, Dec 19, 2017 at 04:45:55PM +0800, Peter Xu wrote:
> > +To add OOB execution support for a command, we need to make sure the
> > +command handler satisfies at least the following:
> 
> It would help to rephrase this in the imperative mood (telling the
> reader what to do rather than describing what would need to be done):
> 
> "OOB command handlers must satisfy the following conditions:"
> 
> That way it's shorter and easier to read, and it communicates that these
> conditions are absolutely necessary.
> 
> > +
> > +- It executes extremely fast,
> > +- It does not take any lock (or, it can take very small locks, but in
> > +  very predictable ways),
> 
> "it can take very small locks, but in very predictable ways" does not
> explain what is allowed and what isn't.
> 
> I suggest: "it can take very small locks if all critical regions also
> follow the rules for OOB command handler code".
> 
> > +- It does not invoke system calls that may block,
> > +- It does not access guest RAM that may block when userfaultfd is
> > +  enabled for postcopy live migration.
> > +
> > +If someone is unsure about whether a command handler can be run in OOB
> > +way, then it possibly means that it does not suite for OOB execution.
> 
> "If in doubt, do not implement OOB execution support."

All fixed.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] vhost-user graceful connect/disconnect

[Wei Wang]

On 01/09/2018 12:09 AM, Stefan Hajnoczi wrote:

On Mon, Jan 08, 2018 at 07:22:37PM +0800, Wei Wang wrote:

On 01/05/2018 11:49 PM, Stefan Hajnoczi wrote:

On Thu, Jan 04, 2018 at 07:15:38PM +0800, Wei Wang wrote:

On 01/04/2018 06:47 PM, Stefan Hajnoczi wrote:

On Thu, Dec 21, 2017 at 06:01:29AM -0500, Marc-André Lureau wrote:

I'm not going to prototype this yet, I'm working on virtio-vhost-user
first, but eventually I might get back to -object vhost-user(-backend).


Hi Stefan, are you implementing the guest slave and vhost-pci driver (we've
posted to the dpdk mailinglist) as well? and do you have an estimation when
would the prototype be ready?

I'm implementing the "[RFC virtio-dev] vhost-user-slave: add vhost-user
slave device type" device in QEMU and DPDK in order to show how the
ideas we've discussed work.

Here is the VIRTIO spec link again:
https://stefanha.github.io/virtio/vhost-user-slave.html#x1-2830007

There are four virtqueues documented in the spec, would two suffice? Request
and Response can be distinguished by VHOST_USER_REPLY_MASK.

That's a good idea and I'll do it for the master-to-slave virtqueue.

Unfortunately virtqueues are asymmetric so a single virtqueue cannot
support the slave-to-master message flow (VHOST_USER_SET_SLAVE_REQ_FD):
1. Master puts an empty buffer onto vq.
2. Slave takes buffer and fills in a request.  The buffer is now used.
3. Master pops the used buffer but there is no way to reply back to the
slave!



Essentially, I think two virtqueues should be sufficient for 
guest-to-host and host-to-guest communication.


Host-to-guest virtqueue: used for master request/response messages 
passed to the guest slave (driver). The buffer can be primed by the 
guest driver (like virtio-net rxq).
Guest-to-host virtqueue: used for guest slave request/response messages, 
like virtio-net txq, it doesn't need to prime buffers in advance.



Best,
Wei

Re: [Qemu-devel] [RFC v6 22/27] qmp: isolate responses into io thread

[Peter Xu]

On Tue, Jan 09, 2018 at 02:24:35PM +, Stefan Hajnoczi wrote:
> On Tue, Dec 19, 2017 at 04:45:52PM +0800, Peter Xu wrote:
> > +static void monitor_qmp_bh_responder(void *opaque)
> > +{
> > +QMPResponse response;
> > +
> > +while (true) {
> > +response = monitor_qmp_response_pop_one();
> > +if (!response.data) {
> > +break;
> > +}
> > +monitor_json_emitter_raw(response.mon, response.data);
> 
> Have you audited all mon->out_lock users to ensure that guest memory is
> never touched while the lock is held?
> 
> If guest memory is touched then the main loop could be blocked due to
> postcopy and when the IOThread executes monitor_qmp_bh_responder() ->
> monitor_json_emitter_raw() -> monitor_puts() it will also hang!

Yes, I am mostly certain that it never touches guest memory.  IMHO
it's a pure lock for monitor's output buffer since day one of its
occurence in 6cff3e8594c.

An extreme case I can think of is when someone wants to dump some data
from guest memory in QMP (I suspect who will use it though...).  In
that case we will still be safe, since the guest memory access should
be happening in g_strdup_vprintf() rather than monitor_puts(), so even
if it happens we will hang at g_strdup_vprintf() without the lock taken.

> 
> Please add a comment above the out_lock declaration letting users know
> that they must not touch guest memory while holding the lock.

I'm not sure whether adding a comment like this for the lock would be
a bit strange, but I agree it should be better than nothing.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH] net: Allow netdevs to be used with 'hostfwd_add' and 'hostfwd_remove'

[Fam Zheng]

On Fri, 01/12 07:23, Thomas Huth wrote:
> On 11.01.2018 21:41, no-re...@patchew.org wrote:
> > Hi,
> > 
> > This series failed build test on ppc host. Please find the details below.
> [...]
> > Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
> > Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 
> > 'dtc'
> > Submodule 'pixman' (git://anongit.freedesktop.org/pixman) registered for 
> > path 'pixman'
> > Submodule 'roms/SLOF' (git://git.qemu-project.org/SLOF.git) registered for 
> > path 'roms/SLOF'
> > Submodule 'roms/ipxe' (git://git.qemu-project.org/ipxe.git) registered for 
> > path 'roms/ipxe'
> > Submodule 'roms/openbios' (git://git.qemu-project.org/openbios.git) 
> > registered for path 'roms/openbios'
> > Submodule 'roms/openhackware' (git://git.qemu-project.org/openhackware.git) 
> > registered for path 'roms/openhackware'
> > Submodule 'roms/qemu-palcode' (git://github.com/rth7680/qemu-palcode.git) 
> > registered for path 'roms/qemu-palcode'
> > Submodule 'roms/seabios' (git://git.qemu-project.org/seabios.git/) 
> > registered for path 'roms/seabios'
> > Submodule 'roms/sgabios' (git://git.qemu-project.org/sgabios.git) 
> > registered for path 'roms/sgabios'
> > Submodule 'roms/u-boot' (git://git.qemu-project.org/u-boot.git) registered 
> > for path 'roms/u-boot'
> > Submodule 'roms/vgabios' (git://git.qemu-project.org/vgabios.git/) 
> > registered for path 'roms/vgabios'
> > Cloning into 'dtc'...
> > Submodule path 'dtc': checked out '65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf'
> > fatal: remote error: access denied or repository not exported: /pixman
> > Cloning into 'pixman'...
> > Clone of 'git://anongit.freedesktop.org/pixman' into submodule path 
> > 'pixman' failed
> 
>  Hi Fam,
> 
> why is patchew still trying to clone pixman? I thought we only use the
> system pixman library nowadays?

There is a bug in patchew tester that the git cache repo is not updated, so
every time it starts from a very old commit. Will fix it soon.

Fam

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Wei Wang]

On 01/11/2018 05:56 PM, Stefan Hajnoczi wrote:

On Thu, Jan 11, 2018 at 6:31 AM, Wei Wang  wrote:

On 01/11/2018 12:14 AM, Stefan Hajnoczi wrote:

2) requires the driver to join the vhost-user negotiation.

The driver must participate in vhost-user negotiation.  The vhost-pci
patches try to avoid this by taking features bits on the QEMU
command-line and hardcoding the number of supported virtqueues.  That
doesn't work in production environments because:
1. What if the driver inside the VM has been updated and now supports
different features?
2. What if the user isn't allowed to modify the VM configuration?
3. What if the management software doesn't expose the feature bits
command-line parameter?
4. What if the number of virtqueues must be different from QEMU's
default value to limit resource consumption?

Users will find it incovenient to manually enter feature bits for the
driver they are using.  The driver needs to be part of negotiation so
it can indicate which features are supported, how many virtqueues,
etc.



OK, the advantages of letting the guest join the vhost-user negotiation 
sounds convincing to me. Thanks.





Without above two, the solution already works well, so I'm not sure why would 
we need the above two from functionality point of view.

The "[PATCH v3 0/7] Vhost-pci for inter-VM communication" series is
incomplete.  It is a subset of vhost-user-net and it works only for
poll-mode drivers.  It's the requirements that haven't been covered by
the vhost-pci patch series yet that make me prefer the
virtio-vhost-user approach.

The virtio device design needs to be capable of supporting the rest of
vhost-user functionality in the future.  Once the code is merged in
QEMU and DPDK it will be very difficult to make changes to the virtio
device.


This is how virtio works. A new feature with a new feature bit. Now, we 
let the guest driver join the vhost-user negotiation (including feature 
negotiation), the default device/driver feature negotiation is free to use.
I'm thinking if it is worthwhile to do some kind of mediated 
passthrough, which passes the selected messages only. Because many 
messages are not necessary to be passed down (e.g. VHOST_USER_SEND_RARP 
is not needed for simple VM-to-VM communication), though might be safe 
to do. I plan to see your full passthrough code first, and see if 
changing to mediated passthrough would be simpler.


Best,
Wei

Re: [Qemu-devel] [RFC v6 21/27] qmp: support out-of-band (oob) execution

[Peter Xu]

On Tue, Jan 09, 2018 at 02:08:10PM +, Stefan Hajnoczi wrote:
> On Tue, Dec 19, 2017 at 04:45:51PM +0800, Peter Xu wrote:
> > diff --git a/monitor.c b/monitor.c
> > index b571866659..505db439d8 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -1090,6 +1090,44 @@ static void qmp_caps_apply(Monitor *mon, 
> > QMPCapabilityList *list)
> >  }
> >  }
> >  
> > +/*
> > + * Return true if check successful, or false otherwise.  When false is
> > + * returned, detailed error will be in errp if provided.
> > + */
> > +static bool qmp_cmd_oob_check(Monitor *mon, QDict *req, Error **errp)
> > +{
> > +const char *command;
> > +QmpCommand *cmd;
> > +
> > +command = qdict_get_try_str(req, "execute");
> > +if (!command) {
> > +error_setg(errp, "Command field 'execute' missing");
> > +return false;
> > +}
> > +
> > +cmd = qmp_find_command(mon->qmp.commands, command);
> > +if (!cmd) {
> > +error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
> > +  "The command %s has not been found", command);
> > +return false;
> > +}
> > +
> > +if (qmp_is_oob(req)) {
> > +if (!qmp_oob_enabled(mon)) {
> > +error_setg(errp, "Please enable Out-Of-Band first "
> > +   "for the session during handshake");
> 
> The qmp-spec.txt document calls it "capabilities negotiation".  I don't
> see "handshake" mentioned.  Please use the terminology from the QMP spec
> so it's clear what this error message means.
> 
> > +return false;
> > +}
> > +if (!(cmd->options & QCO_ALLOW_OOB)) {
> > +error_setg(errp, "The command %s does not support OOB",
> > +   command);
> > +return false;
> > +}
> > +}
> > +
> > +return true;
> > +}
> > +
> >  void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
> >Error **errp)
> >  {
> > @@ -4032,6 +4070,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
> >  QMPRequest *req_obj = monitor_qmp_requests_pop_one();
> >  
> >  if (req_obj) {
> > +trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id));
> 
> Not all tracers on all platforms support NULL string pointers.  Please
> use qobject_get_try_str(req_obj->id) ?: "".
> 
> >  monitor_qmp_dispatch_one(req_obj);
> >  /* Reschedule instead of looping so the main loop stays responsive 
> > */
> >  qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> > @@ -4055,17 +4094,31 @@ static void handle_qmp_command(JSONMessageParser 
> > *parser, GQueue *tokens)
> >  error_setg(, QERR_JSON_PARSING);
> >  }
> >  if (err) {
> > -monitor_qmp_respond(mon, NULL, err, NULL);
> > -qobject_decref(req);
> > -return;
> > +goto err;
> > +}
> > +
> > +/* Check against the request in general layout */
> > +qdict = qmp_dispatch_check_obj(req, );
> > +if (!qdict) {
> > +goto err;
> >  }
> >  
> > -qdict = qobject_to_qdict(req);
> > -if (qdict) {
> > -id = qdict_get(qdict, "id");
> > -qobject_incref(id);
> > -qdict_del(qdict, "id");
> > -} /* else will fail qmp_dispatch() */
> > +/* Check against OOB specific */
> > +if (!qmp_cmd_oob_check(mon, qdict, )) {
> > +goto err;
> > +}
> > +
> > +id = qdict_get(qdict, "id");
> > +
> > +/* When OOB is enabled, the "id" field is mandatory. */
> > +if (qmp_oob_enabled(mon) && !id) {
> > +error_setg(, "Out-Of-Band capability requires that "
> > +   "every command contains an 'id' field.");
> 
> error_setg() messages shouldn't end with a period ('.').
> 
> > +goto err;
> > +}
> > +
> > +qobject_incref(id);
> > +qdict_del(qdict, "id");
> >  
> >  req_obj = g_new0(QMPRequest, 1);
> >  req_obj->mon = mon;
> > @@ -4073,6 +4126,13 @@ static void handle_qmp_command(JSONMessageParser 
> > *parser, GQueue *tokens)
> >  req_obj->req = req;
> >  req_obj->need_resume = false;
> >  
> > +if (qmp_is_oob(qdict)) {
> > +/* Out-Of-Band (OOB) requests are executed directly in parser. */
> > +
> > trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id));
> 
> Same NULL "%s" issue here.

Fixing them all.

Good to know the "?:" syntax...  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH] net: Allow netdevs to be used with 'hostfwd_add' and 'hostfwd_remove'

[Thomas Huth]

On 11.01.2018 21:41, no-re...@patchew.org wrote:
> Hi,
> 
> This series failed build test on ppc host. Please find the details below.
[...]
> Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
> Submodule 'dtc' (git://git.qemu-project.org/dtc.git) registered for path 'dtc'
> Submodule 'pixman' (git://anongit.freedesktop.org/pixman) registered for path 
> 'pixman'
> Submodule 'roms/SLOF' (git://git.qemu-project.org/SLOF.git) registered for 
> path 'roms/SLOF'
> Submodule 'roms/ipxe' (git://git.qemu-project.org/ipxe.git) registered for 
> path 'roms/ipxe'
> Submodule 'roms/openbios' (git://git.qemu-project.org/openbios.git) 
> registered for path 'roms/openbios'
> Submodule 'roms/openhackware' (git://git.qemu-project.org/openhackware.git) 
> registered for path 'roms/openhackware'
> Submodule 'roms/qemu-palcode' (git://github.com/rth7680/qemu-palcode.git) 
> registered for path 'roms/qemu-palcode'
> Submodule 'roms/seabios' (git://git.qemu-project.org/seabios.git/) registered 
> for path 'roms/seabios'
> Submodule 'roms/sgabios' (git://git.qemu-project.org/sgabios.git) registered 
> for path 'roms/sgabios'
> Submodule 'roms/u-boot' (git://git.qemu-project.org/u-boot.git) registered 
> for path 'roms/u-boot'
> Submodule 'roms/vgabios' (git://git.qemu-project.org/vgabios.git/) registered 
> for path 'roms/vgabios'
> Cloning into 'dtc'...
> Submodule path 'dtc': checked out '65cc4d2748a2c2e6f27f1cf39e07a5dbabd80ebf'
> fatal: remote error: access denied or repository not exported: /pixman
> Cloning into 'pixman'...
> Clone of 'git://anongit.freedesktop.org/pixman' into submodule path 'pixman' 
> failed

 Hi Fam,

why is patchew still trying to clone pixman? I thought we only use the
system pixman library nowadays?

 Thomas

Re: [Qemu-devel] [RFC v6 20/27] qmp: export qmp_dispatch_check_obj and allow "id"

[Peter Xu]

On Tue, Jan 09, 2018 at 01:45:28PM +, Stefan Hajnoczi wrote:
> On Tue, Dec 19, 2017 at 04:45:50PM +0800, Peter Xu wrote:
> > We need this for earlier check for OOB, before reaching
> > do_qmp_dispatch().  Meanwhile, a small touch-up to allow "id" field,
> > after all we allow that for sure in the spec (in any form).
> 
> This change would be better as part of the patch that needs it.  I don't
> understand the commit description and there is no context in this patch
> showing how this change is used.

Maybe I should mention it explicitly that "it'll be used in the
following up patch".

Logically I think this has nothing to do with OOB so I splitted it
out.  I would prefer it to be a separate patch (though I can enrich
the commit message).  Or, I'm also fine if you think squashing this
into next would be nicer.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC PATCH v3 29/30] replay: improve replay performance

[Pavel Dovgalyuk]

> From: Paolo Bonzini [mailto:pbonz...@redhat.com]
> On 11/01/2018 14:12, Pavel Dovgalyuk wrote:
> >> From: Paolo Bonzini [mailto:paolo.bonz...@gmail.com] On Behalf Of Paolo 
> >> Bonzini
> >> On 11/01/2018 09:27, Pavel Dovgalyuk wrote:
> >>> +} else {
> >>> +qemu_notify_event();
> >>> +}
> >>
> >> Before this patch, what would do the qemu_notify_event?
> >
> > We tried using qemu_notify_event to wake up the iothread and speedup the 
> > replay,
> > when there is no work for vcpu thread to do.
> 
> But why doesn't the iothread wake up on its own?

It will wake up by timeout. But with this change this happens sooner and replay 
goes faster.

Pavel Dovgalyuk

Re: [Qemu-devel] [RFC PATCH v3 03/30] This patch adds a condition before overwriting exception_index fields.

[Pavel Dovgalyuk]

> From: Paolo Bonzini [mailto:pbonz...@redhat.com]
> On 11/01/2018 09:25, Pavel Dovgalyuk wrote:
> > It is needed when exception_index is already set to some meaningful value.
> >
> > Signed-off-by: Pavel Dovgalyuk 
> > Signed-off-by: Paolo Bonzini 
> > ---
> >  accel/tcg/cpu-exec.c |5 -
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> >
> > diff --git a/accel/tcg/cpu-exec.c b/accel/tcg/cpu-exec.c
> > index 280200f..9cc6972 100644
> > --- a/accel/tcg/cpu-exec.c
> > +++ b/accel/tcg/cpu-exec.c
> > @@ -585,6 +585,7 @@ static inline bool cpu_handle_interrupt(CPUState *cpu,
> >  else {
> >  if (cc->cpu_exec_interrupt(cpu, interrupt_request)) {
> >  replay_interrupt();
> > +cpu->exception_index = -1;
> >  *last_tb = NULL;
> >  }
> >  /* The target hook may have updated the 
> > 'cpu->interrupt_request';
> > @@ -606,7 +607,9 @@ static inline bool cpu_handle_interrupt(CPUState *cpu,
> >  if (unlikely(atomic_read(>exit_request)
> >  || (use_icount && cpu->icount_decr.u16.low + cpu->icount_extra == 
> > 0))) {
> >  atomic_set(>exit_request, 0);
> > -cpu->exception_index = EXCP_INTERRUPT;
> > +if (cpu->exception_index == -1) {
> > +cpu->exception_index = EXCP_INTERRUPT;
> > +}
> >  return true;
> >  }
> >
> >
> 
> Where does my S-o-b come from? :)

Because you proposed first version of this code, I think.


Pavel Dovgalyuk

Re: [Qemu-devel] [RFC v6 17/27] qmp: add new event "command-dropped"

[Peter Xu]

On Tue, Jan 09, 2018 at 01:20:20PM +, Stefan Hajnoczi wrote:
> On Tue, Dec 19, 2017 at 04:45:47PM +0800, Peter Xu wrote:
> > This event will be emitted if one QMP command is dropped.  Along,
> > declare an enum for the reasons.
> > 
> > Signed-off-by: Peter Xu 
> > ---
> >  qapi-schema.json | 37 +
> >  1 file changed, 37 insertions(+)
> > 
> > diff --git a/qapi-schema.json b/qapi-schema.json
> > index 531fd4c0db..447ab25572 100644
> > --- a/qapi-schema.json
> > +++ b/qapi-schema.json
> > @@ -3222,3 +3222,40 @@
> >  # Since: 2.11
> >  ##
> >  { 'command': 'watchdog-set-action', 'data' : {'action': 'WatchdogAction'} }
> > +
> > +##
> > +# @CommandDropReason:
> > +#
> > +# Reasons that caused one command to be dropped.
> > +#
> > +# @queue-full: the queue of command is full. This can only occur when
> 
> s/the queue of command is full/the command queue is full/

Fixed.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v6 16/27] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Mon, Jan 08, 2018 at 05:09:16PM +, Stefan Hajnoczi wrote:
> On Tue, Dec 19, 2017 at 04:45:46PM +0800, Peter Xu wrote:
> > Originally QMP goes throw these steps:
> 
> s/throw/through/
> 
> > 
> >   JSON Parser --> QMP Dispatcher --> Respond
> >   /|\(2)(3) |
> >(1) |   \|/ (4)
> >+-  main thread  +
> > 
> > This patch does this:
> > 
> >   JSON Parser QMP Dispatcher --> Respond
> >   /|\ |   /|\   (4) |
> >|  | (2)| (3)|  (5)
> >(1) |  +->  |   \|/
> >+-  main thread  <---+
> > 
> > So the parsing job and the dispatching job is isolated now.  It gives us
> > a chance in following up patches to totally move the parser outside.
> > 
> > The isloation is done using one QEMUBH. Only one dispatcher QEMUBH is
> 
> s/isloation/isolation/
> 
> > @@ -3897,30 +3920,39 @@ static void monitor_qmp_respond(Monitor *mon, 
> > QObject *rsp,
> >  qobject_decref(rsp);
> >  }
> >  
> > -static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> > +struct QMPRequest {
> > +/* Owner of the request */
> > +Monitor *mon;
> > +/* "id" field of the request */
> > +QObject *id;
> > +/* Request object to be handled */
> > +QObject *req;
> > +/*
> > + * Whether we need to resume the monitor afterward.  This flag is
> > + * used to emulate the old QMP server behavior that only one
> > + * command is executed for each time.
> 
> s/only one command is executed for each time/the current command must
> complete before the next one executes/

Will address all the grammar issues.

> 
> > + */
> > +bool need_resume;
> 
> This isn't really a per-request decision so a QMPRequest field is not
> necessary.  If "oob" is enabled then we dispatch commands without
> waiting.  If "oob" is disabled then we complete the current command
> before dispatching the next one.

I explicitly added this to make sure the suspend and resume will be
paired up.  Actually there is at least one exception, which is the
initial "qmp_capabilities" command especially when used to enable the
OOB capability. For that case, OOB is not enabled before execution,
while OOB is enabled during the execution and after running the
command.  If without this field, monitor can hang.

> 
> If you want to keep this, I don't mind, but the field isn't necessary.

According to above, I would still like to keep it.

> 
> > @@ -4150,6 +4292,15 @@ static void monitor_iothread_init(void)
> >  {
> >  mon_global.mon_iothread = iothread_create("mon_iothread",
> >_abort);
> > +
> > +/*
> > + * This MUST be on main loop thread since we have commands that
> > + * have assumption to be run on main loop thread (Yeah, we'd
> > + * better remove this assumption in the future).
> 
> "we'd better ..." usually means that it will be necessary.  It is
> stronger than "it would be nice to ...".  I'm not sure which one you
> mean here.
> 
> There doesn't seem to be any immediate need or plan to run regular
> commands outside the main loop.  I'm not aware of active work to do
> that.  So what is this comment trying to say?

Yeah I mean "it would be nice to", I'll switch.

This comment explains why dispatcher bottom half is bound to the
default main thread and it cannot be bound to anything else.  Thanks,

-- 
Peter Xu

[Qemu-devel] [QEMU-PPC] [PATCH 3/3] target/ppc: Add H-Call H_GET_CPU_CHARACTERISTICS

[Suraj Jitindar Singh]

The new H-Call H_GET_CPU_CHARACTERISTICS is used by the guest to query
behaviours and available characteristics of the cpu.

Implement the handler for this new H-Call which formulates its response
based on the setting of the new capabilities added in the previous
patch.
---
 hw/ppc/spapr_hcall.c   | 78 ++
 include/hw/ppc/spapr.h |  1 +
 2 files changed, 79 insertions(+)

diff --git a/hw/ppc/spapr_hcall.c b/hw/ppc/spapr_hcall.c
index 51eba52e86..b167554771 100644
--- a/hw/ppc/spapr_hcall.c
+++ b/hw/ppc/spapr_hcall.c
@@ -1654,6 +1654,81 @@ static target_ulong 
h_client_architecture_support(PowerPCCPU *cpu,
 return H_SUCCESS;
 }
 
+/* H_GET_CPU_CHARACTERISTICS return values */
+#define H_CPU_CHAR_SPEC_BAR_ORI31   PPC_BIT(0)
+#define H_CPU_CHAR_BCCTRL_SERIALISEDPPC_BIT(1)
+#define H_CPU_CHAR_L1D_FLUSH_ORI30  PPC_BIT(2)
+#define H_CPU_CHAR_L1D_FLUSH_TRIG2  PPC_BIT(3)
+#define H_CPU_CHAR_L1D_THREAD_PRIV  PPC_BIT(4)
+#define H_CPU_CHAR_HON_BRANCH_HINTS PPC_BIT(5)
+#define H_CPU_CHAR_THR_RECONF_TRIG  PPC_BIT(6)
+#define H_CPU_BEHAV_FAVOUR_SECURITY PPC_BIT(0)
+#define H_CPU_BEHAV_L1D_FLUSH_PRPPC_BIT(1)
+#define H_CPU_BEHAV_BNDS_CHK_SPEC_BAR   PPC_BIT(2)
+
+static target_ulong h_get_cpu_characteristics(PowerPCCPU *cpu,
+  sPAPRMachineState *spapr,
+  target_ulong opcode,
+  target_ulong *args)
+{
+uint64_t characteristics = H_CPU_CHAR_HON_BRANCH_HINTS &
+  ~H_CPU_CHAR_THR_RECONF_TRIG;
+uint64_t behaviour = H_CPU_BEHAV_FAVOUR_SECURITY;
+uint8_t safe_cache = spapr_get_cap(spapr, SPAPR_CAP_CFPC);
+uint8_t safe_bounds_check = spapr_get_cap(spapr, SPAPR_CAP_SBBC);
+uint8_t safe_indirect_branch = spapr_get_cap(spapr, SPAPR_CAP_IBS);
+
+switch (safe_cache) {
+case SPAPR_CAP_WORKAROUND:
+characteristics |= H_CPU_CHAR_L1D_FLUSH_ORI30;
+characteristics |= H_CPU_CHAR_L1D_FLUSH_TRIG2;
+characteristics |= H_CPU_CHAR_L1D_THREAD_PRIV;
+behaviour |= H_CPU_BEHAV_L1D_FLUSH_PR;
+break;
+case SPAPR_CAP_FIXED:
+break;
+default: /* broken */
+if (safe_cache != SPAPR_CAP_BROKEN) {
+error_report("Invalid value for cap-cfpc (%d), assuming broken",
+ safe_cache);
+}
+behaviour |= H_CPU_BEHAV_L1D_FLUSH_PR;
+break;
+}
+
+switch (safe_bounds_check) {
+case SPAPR_CAP_WORKAROUND:
+characteristics |= H_CPU_CHAR_SPEC_BAR_ORI31;
+behaviour |= H_CPU_BEHAV_BNDS_CHK_SPEC_BAR;
+break;
+case SPAPR_CAP_FIXED:
+break;
+default: /* broken */
+if (safe_bounds_check != SPAPR_CAP_BROKEN) {
+error_report("Invalid value for cap-sbbc (%d), assuming broken",
+ safe_bounds_check);
+}
+behaviour |= H_CPU_BEHAV_BNDS_CHK_SPEC_BAR;
+break;
+}
+
+switch (safe_indirect_branch) {
+case SPAPR_CAP_FIXED:
+characteristics |= H_CPU_CHAR_BCCTRL_SERIALISED;
+default: /* broken */
+if (safe_indirect_branch != SPAPR_CAP_BROKEN) {
+error_report("Invalid value for cap-ibs (%d), assuming broken",
+ safe_indirect_branch);
+}
+break;
+}
+
+args[0] = characteristics;
+args[1] = behaviour;
+
+return H_SUCCESS;
+}
+
 static spapr_hcall_fn papr_hypercall_table[(MAX_HCALL_OPCODE / 4) + 1];
 static spapr_hcall_fn kvmppc_hypercall_table[KVMPPC_HCALL_MAX - 
KVMPPC_HCALL_BASE + 1];
 
@@ -1733,6 +1808,9 @@ static void hypercall_register_types(void)
 spapr_register_hypercall(H_INVALIDATE_PID, h_invalidate_pid);
 spapr_register_hypercall(H_REGISTER_PROC_TBL, h_register_process_table);
 
+/* hcall-get-cpu-characteristics */
+spapr_register_hypercall(H_GET_CPU_CHARACTERISTICS, 
h_get_cpu_characteristics);
+
 /* "debugger" hcalls (also used by SLOF). Note: We do -not- differenciate
  * here between the "CI" and the "CACHE" variants, they will use whatever
  * mapping attributes qemu is using. When using KVM, the kernel will
diff --git a/include/hw/ppc/spapr.h b/include/hw/ppc/spapr.h
index 63ffd422b5..98e76d8b9a 100644
--- a/include/hw/ppc/spapr.h
+++ b/include/hw/ppc/spapr.h
@@ -392,6 +392,7 @@ struct sPAPRMachineState {
 #define H_GET_HCA_INFO  0x1B8
 #define H_GET_PERF_COUNT0x1BC
 #define H_MANAGE_TRACE  0x1C0
+#define H_GET_CPU_CHARACTERISTICS 0x1C8
 #define H_FREE_LOGICAL_LAN_BUFFER 0x1D4
 #define H_QUERY_INT_STATE   0x1E4
 #define H_POLL_PENDING  0x1D8
-- 
2.13.6

[Qemu-devel] [QEMU-PPC] [PATCH 1/3] hw/ppc/spapr_caps: Rework spapr_caps to use uint8 internal representation

[Suraj Jitindar Singh]

Currently spapr_caps are tied to boolean values (on or off). This patch
reworks the caps so that they can have any uint8 value. This allows more
capabilities with various values to be represented in the same way
internally. Capabilities are numbered in ascending order. The internal
representation of capability values is an array of uint8s in the
sPAPRMachineState, indexed by capability number.

Capabilities can have their own name, description, options, getter and
setter functions, type and allow functions. They also each have their own
section in the migration stream. Capabilities are only migrated if they
were explictly set on the command line, with the assumption that
otherwise the default will match.

On migration we ensure that the capability value on the destination
is greater than or equal to the capability value from the source. So
long at this remains the case then the migration is considered
compatible and allowed to continue.

This patch implements generic getter and setter functions for boolean
capabilities. It also converts the existings cap-htm, cap-vsx and
cap-dfp capabilities to this new format.
---
 hw/ppc/spapr.c |  45 +--
 hw/ppc/spapr_caps.c| 322 +
 include/hw/ppc/spapr.h |  45 +++
 3 files changed, 225 insertions(+), 187 deletions(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index d1acfe8858..3e528fe91e 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -320,7 +320,7 @@ static void spapr_populate_pa_features(sPAPRMachineState 
*spapr,
  */
 pa_features[3] |= 0x20;
 }
-if (spapr_has_cap(spapr, SPAPR_CAP_HTM) && pa_size > 24) {
+if ((spapr_get_cap(spapr, SPAPR_CAP_HTM) != 0) && pa_size > 24) {
 pa_features[24] |= 0x80;/* Transactional memory support */
 }
 if (legacy_guest && pa_size > 40) {
@@ -563,7 +563,7 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, 
int offset,
  *
  * Only CPUs for which we create core types in spapr_cpu_core.c
  * are possible, and all of those have VMX */
-if (spapr_has_cap(spapr, SPAPR_CAP_VSX)) {
+if (spapr_get_cap(spapr, SPAPR_CAP_VSX) != 0) {
 _FDT((fdt_setprop_cell(fdt, offset, "ibm,vmx", 2)));
 } else {
 _FDT((fdt_setprop_cell(fdt, offset, "ibm,vmx", 1)));
@@ -572,7 +572,7 @@ static void spapr_populate_cpu_dt(CPUState *cs, void *fdt, 
int offset,
 /* Advertise DFP (Decimal Floating Point) if available
  *   0 / no property == no DFP
  *   1   == DFP available */
-if (spapr_has_cap(spapr, SPAPR_CAP_DFP)) {
+if (spapr_get_cap(spapr, SPAPR_CAP_DFP) != 0) {
 _FDT((fdt_setprop_cell(fdt, offset, "ibm,dfp", 1)));
 }
 
@@ -1586,6 +1586,18 @@ static bool spapr_vga_init(PCIBus *pci_bus, Error **errp)
 }
 }
 
+static int spapr_pre_load(void *opaque)
+{
+int rc;
+
+rc = spapr_caps_pre_load(opaque);
+if (rc) {
+return rc;
+}
+
+return 0;
+}
+
 static int spapr_post_load(void *opaque, int version_id)
 {
 sPAPRMachineState *spapr = (sPAPRMachineState *)opaque;
@@ -1627,6 +1639,18 @@ static int spapr_post_load(void *opaque, int version_id)
 return err;
 }
 
+static int spapr_pre_save(void *opaque)
+{
+int rc;
+
+rc = spapr_caps_pre_save(opaque);
+if (rc) {
+return rc;
+}
+
+return 0;
+}
+
 static bool version_before_3(void *opaque, int version_id)
 {
 return version_id < 3;
@@ -1747,7 +1771,9 @@ static const VMStateDescription vmstate_spapr = {
 .name = "spapr",
 .version_id = 3,
 .minimum_version_id = 1,
+.pre_load = spapr_pre_load,
 .post_load = spapr_post_load,
+.pre_save = spapr_pre_save,
 .fields = (VMStateField[]) {
 /* used to be @next_irq */
 VMSTATE_UNUSED_BUFFER(version_before_3, 0, 4),
@@ -1762,7 +1788,9 @@ static const VMStateDescription vmstate_spapr = {
 _spapr_ov5_cas,
 _spapr_patb_entry,
 _spapr_pending_events,
-_spapr_caps,
+_spapr_cap_htm,
+_spapr_cap_vsx,
+_spapr_cap_dfp,
 NULL
 }
 };
@@ -2323,8 +2351,6 @@ static void spapr_machine_init(MachineState *machine)
 char *filename;
 Error *resize_hpt_err = NULL;
 
-spapr_caps_validate(spapr, _fatal);
-
 msi_nonbroken = true;
 
 QLIST_INIT(>phbs);
@@ -3834,7 +3860,9 @@ static void spapr_machine_class_init(ObjectClass *oc, 
void *data)
  */
 mc->numa_mem_align_shift = 28;
 
-smc->default_caps = spapr_caps(SPAPR_CAP_VSX | SPAPR_CAP_DFP);
+smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
+smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
+smc->default_caps.caps[SPAPR_CAP_DFP] = SPAPR_CAP_ON;
 spapr_caps_add_properties(smc, _abort);
 }
 
@@ -3916,8 +3944,7 @@ static void spapr_machine_2_11_class_options(MachineClass 
*mc)
 sPAPRMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
 
 spapr_machine_2_12_class_options(mc);
-smc->default_caps = 

[Qemu-devel] [QEMU-PPC] [PATCH 0/3] target/ppc: Rework spapr_caps

[Suraj Jitindar Singh]

The following patch series reworks the implementation of spapr_caps
to allow for a increased number of possible values in the internal
representation.

It also adds 3 new tristate capabilities.

A new H-Call is implemented which a guest will use to query the 
requirement for and availability of workarounds for certain cpu 
behaviours.

Applies on top of David's series - spapr: Add optional capabilities
Based on ppc-for-2.12

The patches could be split up more, but in the interest of getting
them out there they are as they are.

Suraj Jitindar Singh (3):
  hw/ppc/spapr_caps: Rework spapr_caps to use uint8 internal
representation
  hw/spapr/spapr_caps: Add new caps
safe_[cache/bounds_check/indirect_branch]
  target/ppc: Add H-Call H_GET_CPU_CHARACTERISTICS

 hw/ppc/spapr.c|  51 -
 hw/ppc/spapr_caps.c   | 483 --
 hw/ppc/spapr_hcall.c  |  78 
 include/hw/ppc/spapr.h|  61 +++---
 linux-headers/linux/kvm.h |   3 +
 target/ppc/kvm.c  |  28 +++
 target/ppc/kvm_ppc.h  |  18 ++
 7 files changed, 544 insertions(+), 178 deletions(-)

-- 
2.13.6

[Qemu-devel] [QEMU-PPC] [PATCH 2/3] hw/spapr/spapr_caps: Add new caps safe_[cache/bounds_check/indirect_branch]

[Suraj Jitindar Singh]

This patch adds three new capabilities:
cap-cfpc -> safe_cache
cap-sbbc -> safe_bounds_check
cap-ibs  -> safe_indirect_branch

Each capability is tristate with the possible values "broken",
"workaround" or "fixed". Add generic getter and setter functions for
this new capability type. Add these new capabilities to the capabilities
list. The maximum value for the capabilities is queried from kvm through
new kvm capabilities. The requested values are considered to be
compatible if kvm can support an equal or higher value for each
capability.

Note: For TCG we only allow broken for each of these until we decide on
  a mitigation technique.
---
 hw/ppc/spapr.c|   6 ++
 hw/ppc/spapr_caps.c   | 181 ++
 include/hw/ppc/spapr.h|  15 +++-
 linux-headers/linux/kvm.h |   3 +
 target/ppc/kvm.c  |  28 +++
 target/ppc/kvm_ppc.h  |  18 +
 6 files changed, 250 insertions(+), 1 deletion(-)

diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 3e528fe91e..269c1c7857 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -1791,6 +1791,9 @@ static const VMStateDescription vmstate_spapr = {
 _spapr_cap_htm,
 _spapr_cap_vsx,
 _spapr_cap_dfp,
+_spapr_cap_cfpc,
+_spapr_cap_sbbc,
+_spapr_cap_ibs,
 NULL
 }
 };
@@ -3863,6 +3866,9 @@ static void spapr_machine_class_init(ObjectClass *oc, 
void *data)
 smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
 smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
 smc->default_caps.caps[SPAPR_CAP_DFP] = SPAPR_CAP_ON;
+smc->default_caps.caps[SPAPR_CAP_CFPC] = SPAPR_CAP_BROKEN;
+smc->default_caps.caps[SPAPR_CAP_SBBC] = SPAPR_CAP_BROKEN;
+smc->default_caps.caps[SPAPR_CAP_IBS] = SPAPR_CAP_BROKEN;
 spapr_caps_add_properties(smc, _abort);
 }
 
diff --git a/hw/ppc/spapr_caps.c b/hw/ppc/spapr_caps.c
index f95a78547d..1c0c2c8253 100644
--- a/hw/ppc/spapr_caps.c
+++ b/hw/ppc/spapr_caps.c
@@ -74,6 +74,66 @@ static void spapr_cap_set_bool(Object *obj, Visitor *v, 
const char *name,
 spapr->eff.caps[cap->index] = value ? SPAPR_CAP_ON : SPAPR_CAP_OFF;
 }
 
+static void spapr_cap_get_tristate(Object *obj, Visitor *v, const char *name,
+   void *opaque, Error **errp)
+{
+sPAPRCapabilityInfo *cap = opaque;
+sPAPRMachineState *spapr = SPAPR_MACHINE(obj);
+char *val = NULL;
+uint8_t value = spapr_get_cap(spapr, cap->index);
+
+switch (value) {
+case SPAPR_CAP_BROKEN:
+val = g_strdup("broken");
+break;
+case SPAPR_CAP_WORKAROUND:
+val = g_strdup("workaround");
+break;
+case SPAPR_CAP_FIXED:
+val = g_strdup("fixed");
+break;
+default:
+error_setg(errp, "Invalid value (%d) for cap-%s", value, cap->name);
+return;
+}
+
+visit_type_str(v, name, , errp);
+g_free(val);
+}
+
+static void spapr_cap_set_tristate(Object *obj, Visitor *v, const char *name,
+   void *opaque, Error **errp)
+{
+sPAPRCapabilityInfo *cap = opaque;
+sPAPRMachineState *spapr = SPAPR_MACHINE(obj);
+char *val;
+Error *local_err = NULL;
+uint8_t value;
+
+visit_type_str(v, name, , _err);
+if (local_err) {
+error_propagate(errp, local_err);
+return;
+}
+
+if (!strcasecmp(val, "broken")) {
+value = SPAPR_CAP_BROKEN;
+} else if (!strcasecmp(val, "workaround")) {
+value = SPAPR_CAP_WORKAROUND;
+} else if (!strcasecmp(val, "fixed")) {
+value = SPAPR_CAP_FIXED;
+} else {
+error_setg(errp, "Invalid capability mode \"%s\" for cap-%s", val,
+   cap->name);
+goto out;
+}
+
+spapr->cmd_line_caps[cap->index] = true;
+spapr->eff.caps[cap->index] = value;
+out:
+g_free(val);
+}
+
 static void cap_htm_apply(sPAPRMachineState *spapr, uint8_t val, Error **errp)
 {
 if (!val) {
@@ -121,6 +181,40 @@ static void cap_dfp_apply(sPAPRMachineState *spapr, 
uint8_t val, Error **errp)
 }
 }
 
+static void cap_safe_cache_apply(sPAPRMachineState *spapr, uint8_t val,
+ Error **errp)
+{
+if (tcg_enabled() && val) {
+/* TODO - for now only allow broken for TCG */
+error_setg(errp, "Requested safe cache capability level not supported 
by tcg, try a different value for cap-cfpc");
+} else if (kvm_enabled() && (val > kvmppc_get_cap_safe_cache())) {
+error_setg(errp, "Requested safe cache capability level not supported 
by kvm, try a different value for cap-cfpc");
+}
+}
+
+static void cap_safe_bounds_check_apply(sPAPRMachineState *spapr, uint8_t val,
+Error **errp)
+{
+if (tcg_enabled() && val) {
+/* TODO - for now only allow broken for TCG */
+error_setg(errp, "Requested safe bounds check capability level not 
supported by tcg, try a different value for cap-cfpc");
+  

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Yang, Zhiyong]

Hi Jason,

> -Original Message-
> From: Qemu-devel [mailto:qemu-devel-
> bounces+zhiyong.yang=intel@nongnu.org] On Behalf Of Jason Wang
> Sent: Friday, January 12, 2018 11:33 AM
> To: Stefan Hajnoczi 
> Cc: Wang, Wei W ; qemu-devel@nongnu.org
> Subject: Re: [Qemu-devel] vhost-pci and virtio-vhost-user
> 
> 
> 
> On 2018年01月11日 23:23, Stefan Hajnoczi wrote:
> > On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:
> >>
> >> On 2018年01月11日 00:14, Stefan Hajnoczi wrote:
> >>> Hi Wei,
> >>> I wanted to summarize the differences between the vhost-pci and
> >>> virtio-vhost-user approaches because previous discussions may have
> >>> been confusing.
> >>>
> >>> vhost-pci defines a new virtio device type for each vhost device
> >>> type (net, scsi, blk).  It therefore requires a virtio device driver
> >>> for each device type inside the slave VM.
> >>>
> >>> Adding a new device type requires:
> >>> 1. Defining a new virtio device type in the VIRTIO specification.
> >>> 3. Implementing a new QEMU device model.
> >>> 2. Implementing a new virtio driver.
> >>>
> >>> virtio-vhost-user is a single virtio device that acts as a
> >>> vhost-user protocol transport for any vhost device type.  It
> >>> requires one virtio driver inside the slave VM and device types are
> >>> implemented using existing vhost-user slave libraries (librte_vhost
> >>> in DPDK and libvhost-user in QEMU).
> >>>
> >>> Adding a new device type to virtio-vhost-user involves:
> >>> 1. Adding any new vhost-user protocol messages to the QEMU
> >>>  virtio-vhost-user device model.
> >>> 2. Adding any new vhost-user protocol messages to the vhost-user slave
> >>>  library.
> >>> 3. Implementing the new device slave.
> >>>
> >>> The simplest case is when no new vhost-user protocol messages are
> >>> required for the new device.  Then all that's needed for
> >>> virtio-vhost-user is a device slave implementation (#3).  That slave
> >>> implementation will also work with AF_UNIX because the vhost-user
> >>> slave library hides the transport (AF_UNIX vs virtio-vhost-user).
> >>> Even better, if another person has already implemented that device
> >>> slave to use with AF_UNIX then no new code is needed for
> >>> virtio-vhost-user support at all!
> >>>
> >>> If you compare this to vhost-pci, it would be necessary to design a
> >>> new virtio device, implement it in QEMU, and implement the virtio driver.
> >>> Much of virtio driver is more or less the same thing as the
> >>> vhost-user device slave but it cannot be reused because the
> >>> vhost-user protocol isn't being used by the virtio device.  The
> >>> result is a lot of duplication in DPDK and other codebases that
> >>> implement vhost-user slaves.
> >>>
> >>> The way that vhost-pci is designed means that anyone wishing to
> >>> support a new device type has to become a virtio device designer.
> >>> They need to map vhost-user protocol concepts to a new virtio device
> >>> type.  This will be time-consuming for everyone involved (e.g. the
> >>> developer, the VIRTIO community, etc).
> >>>
> >>> The virtio-vhost-user approach stays at the vhost-user protocol
> >>> level as much as possible.  This way there are fewer concepts that
> >>> need to be mapped by people adding new device types.  As a result,
> >>> it will allow virtio-vhost-user to keep up with AF_UNIX vhost-user
> >>> and grow because it's easier to work with.
> >>>
> >>> What do you think?
> >>>
> >>> Stefan
> >> So a question is what's the motivation here?
> >>
> >> Form what I'm understanding, vhost-pci tries to build a scalable V2V
> >> private datapath. But according to what you describe here,
> >> virito-vhost-user tries to make it possible to implement the device
> >> inside another VM. I understand the goal of vhost-pci could be done
> >> on top, but it looks to me it would then rather similar to the design
> >> of Xen driver domain. So I can not figure out how it can be done in a high
> performance way.
> > vhost-pci and virtio-vhost-user both have the same goal.  They allow a
> > VM to implement a vhost device (net, scsi, blk, etc).
> 
> Looks not, if I read the code correctly, vhost-pci has a device implementation
> in qemu, and in slave VM it only have a vhost-pci-net driver.
> 
> > This allows
> > software defined network or storage appliances running inside a VM to
> > provide I/O services to other VMs.
> 
> Well, I think we can do it even with the existed virtio or whatever other
> emulated device which should not be bounded to any specific kind of
> device.  And what's more important, according to the kvm 2016 slides of
> vhost-pci, the motivation of vhost-pci is not building SDN but a chain of 
> VNFs.
> So bypassing the central vswitch through a private VM2VM path does make
> sense. (Though whether or not vhost-pci is the best choice is still
> questionable).
> 
> >   To the other VMs the devices look
> > like regular virtio devices.
> >
> > I'm not sure I 

Re: [Qemu-devel] [RFC v6 18/27] monitor: send event when command queue full

[Peter Xu]

On Tue, Jan 09, 2018 at 01:42:08PM +, Stefan Hajnoczi wrote:
> On Tue, Dec 19, 2017 at 04:45:48PM +0800, Peter Xu wrote:
> > @@ -4071,6 +4073,9 @@ static void handle_qmp_command(JSONMessageParser 
> > *parser, GQueue *tokens)
> >  req_obj->req = req;
> >  req_obj->need_resume = false;
> >  
> > +/* Protect qmp_requests and fetching its length. */
> > +qemu_mutex_lock(>qmp.qmp_queue_lock);
> > +
> >  /*
> >   * If OOB is not enabled on current monitor, we'll emulate the old
> >   * behavior that we won't process current monitor any more until
> > @@ -4080,6 +4085,17 @@ static void handle_qmp_command(JSONMessageParser 
> > *parser, GQueue *tokens)
> >  if (!qmp_oob_enabled(mon)) {
> >  monitor_suspend(mon);
> >  req_obj->need_resume = true;
> > +} else {
> > +/* Drop the request if queue is full. */
> > +if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
> > +qapi_event_send_command_dropped(id,
> > +COMMAND_DROP_REASON_QUEUE_FULL,
> > +NULL);
> > +qobject_decref(id);
> > +qobject_decref(req);
> > +g_free(req_obj);
> > +return;
> 
> qmp_queue_lock is still locked!  I suggest releasing the lock before
> calling qapi_event_send_command_dropped().

Yes.  It took me some time to rebase and move that lock out, even if
so I still got one more bug...  Thanks for catching it.

And yes, I agree I should unlock before sending the event.

-- 
Peter Xu

Re: [Qemu-devel] [RFC v6 15/27] monitor: let suspend/resume work even with QMPs

[Peter Xu]

On Mon, Jan 08, 2018 at 04:49:36PM +, Stefan Hajnoczi wrote:
> On Mon, Dec 25, 2017 at 11:26:13AM +0800, Peter Xu wrote:
> > On Thu, Dec 21, 2017 at 07:27:38PM +0800, Fam Zheng wrote:
> > > On Tue, 12/19 16:45, Peter Xu wrote:
> > > > One thing to mention is that for QMPs that are using IOThreads, we need
> > > > an explicit kick for the IOThread in case it is sleeping.
> > > > 
> > > > Since at it, add traces for the operations.
> > > > 
> > > > Signed-off-by: Peter Xu 
> > > > ---
> > > >  monitor.c| 26 +-
> > > >  trace-events |  1 +
> > > >  2 files changed, 22 insertions(+), 5 deletions(-)
> > > > 
> > > > diff --git a/monitor.c b/monitor.c
> > > > index 844508d134..5f05f2e9da 100644
> > > > --- a/monitor.c
> > > > +++ b/monitor.c
> > > > @@ -3992,19 +3992,35 @@ static void monitor_command_cb(void *opaque, 
> > > > const char *cmdline,
> > > >  
> > > >  int monitor_suspend(Monitor *mon)
> > > >  {
> > > > -if (!mon->rs)
> > > > -return -ENOTTY;
> > > 
> > > Please add to the commit message why the mon->rs check is dropped, for 
> > > this and
> > > the other one.
> > 
> > I thought it would be clear enough since mon->rs is only used by HMP
> > and the subject tells us that this patch is adding support for QMP.
> > But... sure I can add one more sentence for that!
> 
> This change breaks hmp.c:hmp_migrate():
> 
>   if (monitor_suspend(mon) < 0) {
>   monitor_printf(mon, "terminal does not allow synchronous "
>  "migration, continuing detached\n");
>   return;
>   }
> 
> mon->rs is used by HMP code to determine whether this is an interactive
> terminal.  Both live migration and password prompting rely on this
> because they must be forbidden when doing so would be impossible (e.g.
> GDB stub tunneling HMP commands).
> 
> Suspending the QMP monitor is useful, but please don't break the HMP
> code.

Could you elaborate a bit on how this patch breaks anything?

IMHO HMP monitors always have Monitor.rs setup, so here in
hmp_migrate() I don't really understand why we need this check, since
IIUC monitor_suspend() for a HMP monitor will never fail before.

Thanks,

-- 
Peter Xu

[Qemu-devel] [Bug 1086782] Re: HPET time drift windows 7 64bits guest

[Launchpad Bug Tracker]

[Expired for QEMU because there has been no activity for 60 days.]

** Changed in: qemu
   Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1086782

Title:
  HPET time drift windows 7 64bits guest

Status in QEMU:
  Expired

Bug description:
  Using latest qemu-kvm (1.2.0), time drift (clock slow in guest) in Windows 7 
64 bits guest when HPET is enabled (default).
  Disabling HPET (-no-hpet) solves the time drift.

  UsePlatformClock enable/disable doesn't make a difference in the guest.
  bcdedit /set useplatformclock true

  Using driftfix slew doesn't make a difference too.

  
  # qemu-system-x86_64 --version
  QEMU emulator version 1.2.0 (qemu-kvm-1.2.0), Copyright (c) 2003-2008 Fabrice 
Bellard

  Kernel is 3.6.8:
  # uname -a
  Linux pulsar 3.6.8 #1 SMP Sat Dec 1 16:26:10 CET 2012 x86_64 x86_64 x86_64 
GNU/Linux

  TSC is stable in the host:
  ===
  # cat /sys/devices/system/clocksource/clocksource0/current_clocksource
  tsc

  Dmesg:
  [0.00] hpet clockevent registered
  [0.00] tsc: Fast TSC calibration using PIT
  [0.00] tsc: Detected 2660.096 MHz processor
  [0.001002] Calibrating delay loop (skipped), value calculated using timer 
frequency.. 5320.19 BogoMIPS (lpj=2660096)
  [0.001138] pid_max: default: 32768 minimum: 301
  ...
  [1.492019] tsc: Refined TSC clocksource calibration: 2659.973 MHz
  [1.492093] Switching to clocksource tsc

  
  CPUinfo, constant_tsc:
  vendor_id   : GenuineIntel
  cpu family  : 6
  model   : 23
  model name  : Intel(R) Core(TM)2 Quad CPUQ8400  @ 2.66GHz
  stepping: 10
  microcode   : 0xa0b
  cpu MHz : 2667.000
  cache size  : 2048 KB
  flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm 
constant_tsc arch_perfmon pebs bts rep_good nopl aperfmperf pni dtes64 monitor 
ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm sse4_1 xsave lahf_lm dtherm tpr_shadow 
vnmi flexpriority
  bogomips: 5320.19

  # grep -i hpet .config
  CONFIG_HPET_TIMER=y
  CONFIG_HPET_EMULATE_RTC=y
  CONFIG_HPET=y
  # CONFIG_HPET_MMAP is not set
  ===

  Qemu command line:
  /usr/bin/qemu-system-x86_64 -drive 
file=/dev/vol0/KVMORION01,cache=none,aio=native,if=virtio \
-drive file=/dev/vol0/KVMORION02,cache=none,aio=native,if=virtio \
-cpu host \
-m 2048 \
-smp 4,maxcpus=4,cores=4,threads=1,sockets=1 \
-rtc base=localtime,driftfix=slew \
-vnc 10.124.241.211:0,password -k es \
-monitor telnet:localhost:37200,server,nowait \
-netdev 
tap,id=kvmorion,ifname=kvmorion,script=/etc/qemu-ifup-br0,downscript=/etc/qemu-ifdown-br0
 \
-device virtio-net-pci,netdev=kvmorion,id=virtio-nic0,mac=02:85:64:02:c2:aa 
\
-device virtio-balloon-pci,id=balloon0 \
-boot menu=on \
-pidfile /var/run/kvmorion.pid \
-daemonize

  Using 1 CPU doesn't make a difference.
  Only workaround is disabling hpet (-no-hpet)

  Sample time drift in guest:
  >ntpdate -q 10.124.241.211
   5 Dec 13:36:06 ntpdate[3464]: Raised to realtime priority class
  server 10.124.241.211, stratum 2, offset 3.694184, delay 0.02551
   5 Dec 13:36:12 ntpdate[3464]: step time server 10.124.241.211 offset 
3.694184 s
  ec

  >ntpdate -q 10.124.241.211
   5 Dec 13:52:02 ntpdate[1964]: Raised to realtime priority class
  server 10.124.241.211, stratum 2, offset 4.719968, delay 0.02554
   5 Dec 13:52:08 ntpdate[1964]: step time server 10.124.241.211 offset 
4.719968 s
  ec

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1086782/+subscriptions

Re: [Qemu-devel] [RFC v6 11/27] qmp: introduce QMPCapability

[Peter Xu]

On Thu, Jan 11, 2018 at 05:07:11PM -0600, Eric Blake wrote:
> On 12/19/2017 02:45 AM, Peter Xu wrote:
> > There was no QMP capabilities defined.  Define the first "oob" as
> 
> s/was/were/

Fixed.

Just to confirm: is "There was no QMP capability" also correct?

> 
> > capability to allow out-of-band messages.
> > 
> > Also, touch up qmp-test.c to test the new bits.
> > 
> > Signed-off-by: Peter Xu 
> > ---
> >  monitor.c| 10 --
> >  qapi-schema.json | 13 +
> >  tests/qmp-test.c | 10 +-
> >  3 files changed, 30 insertions(+), 3 deletions(-)
> 
> I'm assuming later patches will document this?
> 
> I'm somewhat a fan of documentation alongside or before implementation,
> as getting the general overview right and then checking that the code
> matches is a bit nicer than random coding then documenting what we ended
> up with.  But I don't know if reordering patches in your series is
> necessary, as long as the end product is properly documented.

Yes, I put the whole document at the end of the series.  I can put it
as the first patch too.

> 
> > +++ b/qapi-schema.json
> > @@ -118,6 +118,19 @@
> >  ##
> >  { 'command': 'qmp_capabilities' }
> 
> The client can't request a particular feature alongside the command?  Or
> is that in later patches?  With just this patch, the enum QMPCapability
> is not introspected, because it is not referenced by any command
> (although introspection is a bit moot, since the client will learn what
> the host advertises from the initial handshake before the client can
> even request introspection).

Yes, client can't if without further patches.

> 
> >  
> > +##
> > +# @QMPCapability:
> > +#
> > +# QMP supported capabilities to be broadcasted to the clients.
> 
> 'broadcast' is one of those weird verbs that doesn't change spelling
> when constructing its past tense (there is no 'broadcasted').  However,
> I think this description is a bit nicer (and avoids the problematic word
> altogether):
> 
> Enumeration of capabilities to be advertised during initial client
> connection, used for agreeing on particular QMP extension behaviors.

I'll take your advise.

> 
> > +#
> > +# @oob:   QMP ability to support Out-Of-Band requests.
> 
> Rather terse (it doesn't say what Out-Of-Band requests are); even a
> pointer to the QMP spec (where OOB is more fully documented) might be
> nice (of course, that means we need a patch to docs/interop/qmp-spec.txt
> somewhere in the series, especially since this patch renders 2.2.1 in
> that document obsolete...)

Sorry for the inconvenience.  Please refer to the last doc patch for
details.

I thought the doc patch would explain itself but obviously I should be
more careful on the ordering next time to ease reviewers.  I'll move
the doc patch to front when repost, and I'll note here to refer to
qmp-spec.txt for more information.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH 1/1] spapr: Check SMT based on KVM_CAP_PPC_SMT_POSSIBLE

[David Gibson]

On Fri, Jan 12, 2018 at 01:16:22AM +1100, David Gibson wrote:
> On Fri, Jan 05, 2018 at 10:47:22PM -0200, Jose Ricardo Ziviani wrote:
> > Power9 supports 4 HW threads/core but it's possible to emulate
> > doorbells to implement virtual SMT. KVM has the KVM_CAP_PPC_SMT_POSSIBLE
> > which returns a bitmap with all SMT modes supported by the host.
> > 
> > Today, QEMU forces the SMT mode based on PVR compat table, this is
> > silently done in spapr_fixup_cpu_dt. Then, if user passes thread=8 the
> > guest will end up with 4 threads/core without any feedback to the user.
> > It is confusing and will crash QEMU if a cpu is hotplugged in that
> > guest.
> > 
> > This patch makes use of KVM_CAP_PPC_SMT_POSSIBLE to check if the host
> > supports the SMT mode so it allows Power9 guests to have 8 threads/core
> > if desired.
> > 
> > Reported-by: Satheesh Rajendran 
> > Signed-off-by: Jose Ricardo Ziviani 
> > ---
> >  hw/ppc/spapr.c   | 14 +-
> >  hw/ppc/trace-events  |  1 +
> >  target/ppc/kvm.c |  5 +
> >  target/ppc/kvm_ppc.h |  6 ++
> >  4 files changed, 25 insertions(+), 1 deletion(-)
> > 
> > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > index d1acfe8858..ea2503cd2f 100644
> > --- a/hw/ppc/spapr.c
> > +++ b/hw/ppc/spapr.c
> > @@ -345,7 +345,19 @@ static int spapr_fixup_cpu_dt(void *fdt, 
> > sPAPRMachineState *spapr)
> >  PowerPCCPU *cpu = POWERPC_CPU(cs);
> >  DeviceClass *dc = DEVICE_GET_CLASS(cs);
> >  int index = spapr_vcpu_id(cpu);
> > -int compat_smt = MIN(smp_threads, ppc_compat_max_threads(cpu));
> > +
> > +/* set smt to maximum for this current pvr if the number
> > + * passed is higher than defined by PVR compat mode AND
> > + * if KVM cannot emulate it.*/
> > +int compat_smt = smp_threads;
> > +if ((kvmppc_cap_smt_possible() & smp_threads) != smp_threads &&
> > +smp_threads > ppc_compat_max_threads(cpu)) {
> > +compat_smt = ppc_compat_max_threads(cpu);
> 
> I don't think this is the right approach.  We've been trying to remove
> places where host properties (such as those read from KVM
> capabilities) affect guest visible properties of the VM - like vsmt.
> Places like that break migration and often libvirt expectations as
> well.
> 
> This is putting one back in, and so a step in the wrong direction.

Following on from this with some more investigation.

I think the discussion is going off the rails because the reason for
this compat threads limit has been forgotten.

*It has nothing to do with the host*.  It's been recoded a bunch, but
the logic was originally introduced by 2a48d993 "spapr: Limit threads
per core according to current compatibility mode".  It states that it
was to avoid confusing the *guest* by presenting more threads than it
expects on an compatible-with-old CPU.

This also explains why it's done in this otherwise weird way - just
truncating the presented threads in the device tree, while the other
threads still "exist" in the qemu and KVM model:

It's because this happens at CAS time, for the benefit of guests which
don't understand newer CPUs, at which point it's really too late to
report an error to the user or renumber the CPUs.

So I think what we actually want to do here is just *remove* the
compat limit for POWER9 cpus.  Strictly it's to do with the host
kernel rather than the cpu, but in practice we can count on POWER9
guests not being confused by >4 threads (since they POWER8 compat
guests running on POWER9 have to anyway).


As a separate matter, we need to validate guest threads-per-core
against what the host's capable of, but that doesn't belong here.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH v2 2/2] hostmem-file: add 'sync' option

[Haozhong Zhang]

On 01/11/18 20:06 +, Dr. David Alan Gilbert wrote:
> * Haozhong Zhang (haozhong.zh...@intel.com) wrote:
> > This option controls whether QEMU mmap(2) the memory backend file with
> > MAP_SYNC flag, which can fully guarantee the guest write persistence
> > to the backend, if MAP_SYNC flag is supported by the host kernel
> > (Linux kernel 4.15 and later) and the backend is a file supporting
> > DAX (e.g., file on ext4/xfs file system mounted with '-o dax').
> > 
> > It can take one of following values:
> >  - on:  try to pass MAP_SYNC to mmap(2); if MAP_SYNC is not supported or
> > 'share=off', QEMU will abort
> >  - off: never pass MAP_SYNC to mmap(2)
> >  - auto (default): if MAP_SYNC is supported and 'share=on', work as if
> > 'sync=on'; otherwise, work as if 'sync=off'
> 
> I don't know anything about MAP_SYNC but see two minor comments below:
> 
> > Signed-off-by: Haozhong Zhang 
> > Sugguested-by: Eduardo Habkost 
>   ^ typo!
>
[..]
> > +static void file_memory_backend_set_sync(
> > +Object *obj, Visitor *v, const char *name, void *opaque, Error **errp)
> > +{
> > +HostMemoryBackend *backend = MEMORY_BACKEND(obj);
> > +HostMemoryBackendFile *fb = MEMORY_BACKEND_FILE(obj);
> > +Error *local_err = NULL;
> > +OnOffAuto value;
> > +
> > +if (host_memory_backend_mr_inited(backend)) {
> > +error_setg(_err, "cannot change property value");
> 
> Please make this error clearer; if I was helping someone and saw
> this error message in a log, I wouldn't know what type of property it
> was on what device.   Please at least include something saying it's
> memory backend related and sync related; even just including a %s and
> __func__ would help;  even better if you include the name of the object
> so if you had multiple backend files and one failed you could see which
> one.

I'll fix the typo, and include the backend id and the property name in
the error messages in the next version.

Thanks,
Haozhong

Re: [Qemu-devel] vhost-pci and virtio-vhost-user

[Jason Wang]

On 2018年01月11日 23:23, Stefan Hajnoczi wrote:

On Thu, Jan 11, 2018 at 06:57:03PM +0800, Jason Wang wrote:


On 2018年01月11日 00:14, Stefan Hajnoczi wrote:

Hi Wei,
I wanted to summarize the differences between the vhost-pci and
virtio-vhost-user approaches because previous discussions may have been
confusing.

vhost-pci defines a new virtio device type for each vhost device type
(net, scsi, blk).  It therefore requires a virtio device driver for each
device type inside the slave VM.

Adding a new device type requires:
1. Defining a new virtio device type in the VIRTIO specification.
3. Implementing a new QEMU device model.
2. Implementing a new virtio driver.

virtio-vhost-user is a single virtio device that acts as a vhost-user
protocol transport for any vhost device type.  It requires one virtio
driver inside the slave VM and device types are implemented using
existing vhost-user slave libraries (librte_vhost in DPDK and
libvhost-user in QEMU).

Adding a new device type to virtio-vhost-user involves:
1. Adding any new vhost-user protocol messages to the QEMU
 virtio-vhost-user device model.
2. Adding any new vhost-user protocol messages to the vhost-user slave
 library.
3. Implementing the new device slave.

The simplest case is when no new vhost-user protocol messages are
required for the new device.  Then all that's needed for
virtio-vhost-user is a device slave implementation (#3).  That slave
implementation will also work with AF_UNIX because the vhost-user slave
library hides the transport (AF_UNIX vs virtio-vhost-user).  Even
better, if another person has already implemented that device slave to
use with AF_UNIX then no new code is needed for virtio-vhost-user
support at all!

If you compare this to vhost-pci, it would be necessary to design a new
virtio device, implement it in QEMU, and implement the virtio driver.
Much of virtio driver is more or less the same thing as the vhost-user
device slave but it cannot be reused because the vhost-user protocol
isn't being used by the virtio device.  The result is a lot of
duplication in DPDK and other codebases that implement vhost-user
slaves.

The way that vhost-pci is designed means that anyone wishing to support
a new device type has to become a virtio device designer.  They need to
map vhost-user protocol concepts to a new virtio device type.  This will
be time-consuming for everyone involved (e.g. the developer, the VIRTIO
community, etc).

The virtio-vhost-user approach stays at the vhost-user protocol level as
much as possible.  This way there are fewer concepts that need to be
mapped by people adding new device types.  As a result, it will allow
virtio-vhost-user to keep up with AF_UNIX vhost-user and grow because
it's easier to work with.

What do you think?

Stefan

So a question is what's the motivation here?

Form what I'm understanding, vhost-pci tries to build a scalable V2V private
datapath. But according to what you describe here, virito-vhost-user tries
to make it possible to implement the device inside another VM. I understand
the goal of vhost-pci could be done on top, but it looks to me it would then
rather similar to the design of Xen driver domain. So I can not figure out
how it can be done in a high performance way.

vhost-pci and virtio-vhost-user both have the same goal.  They allow
a VM to implement a vhost device (net, scsi, blk, etc).


Looks not, if I read the code correctly, vhost-pci has a device 
implementation in qemu, and in slave VM it only have a vhost-pci-net driver.



This allows
software defined network or storage appliances running inside a VM to
provide I/O services to other VMs.


Well, I think we can do it even with the existed virtio or whatever 
other emulated device which should not be bounded to any specific kind 
of device.  And what's more important, according to the kvm 2016 slides 
of vhost-pci, the motivation of vhost-pci is not building SDN but a 
chain of VNFs. So bypassing the central vswitch through a private VM2VM 
path does make sense. (Though whether or not vhost-pci is the best 
choice is still questionable).



  To the other VMs the devices look
like regular virtio devices.

I'm not sure I understand your reference to the Xen driver domain or
performance.


So what proposed here is basically memory sharing and event notification 
through eventfd, this model have been used by Xen for many years through 
grant table and event channel. Xen use this to move the backend 
implementation from dom0 to a driver domain which has direct access to 
some hardwares. Consider the case of network, it can then implement xen 
netback inside driver domain which can access hardware NIC directly.


This makes sense for Xen and for performance since driver domain 
(backend) can access hardware directly and event was triggered through 
lower overhead hypercall (or it can do busypolling). But for 
virtio-vhost-user, unless you want SRIOV based solutions inside the 
slave VM, I believe we won't want to go 

Re: [Qemu-devel] [RFC v6 10/27] monitor: allow to use IO thread for parsing

[Peter Xu]

On Fri, Jan 05, 2018 at 05:22:26PM +, Stefan Hajnoczi wrote:
> On Tue, Dec 19, 2017 at 04:45:40PM +0800, Peter Xu wrote:
> >  if (monitor_is_qmp(mon)) {
> > -qemu_chr_fe_set_handlers(>chr, monitor_can_read, 
> > monitor_qmp_read,
> > - monitor_qmp_event, NULL, mon, NULL, true);
> >  qemu_chr_fe_set_echo(>chr, true);
> >  json_message_parser_init(>qmp.parser, handle_qmp_command);
> > +qemu_chr_fe_set_handlers(>chr, monitor_can_read, 
> > monitor_qmp_read,
> > + monitor_qmp_event, NULL, mon, context, 
> > true);
> >  } else {
> > +assert(!context);   /* HMP does not support IOThreads */
> >  qemu_chr_fe_set_handlers(>chr, monitor_can_read, monitor_read,
> >   monitor_event, NULL, mon, NULL, true);
> >  }
> 
> qemu_chr_fe_set_handlers() isn't thread-safe.  It looks like
> monitor_can_read()/monitor_qmp_read() can run in the IOThread at the
> same time as monitor_qmp_event() runs in the main loop:
> 
>   void qemu_chr_fe_set_handlers(CharBackend *b,
> IOCanReadHandler *fd_can_read,
> IOReadHandler *fd_read,
> IOEventHandler *fd_event,
> BackendChangeHandler *be_change,
> void *opaque,
> GMainContext *context,
> bool set_open)
>   {
>   ...
> 
>   qemu_chr_be_update_read_handlers(s, context);
> ^-- The IOThread may invoke handler functions from now on!
> 
>   <-- Everything below races with the IOThread!
> 
>   if (set_open) {
>   qemu_chr_fe_set_open(b, fe_open);
>   }
> 
>   if (fe_open) {
>   qemu_chr_fe_take_focus(b);
>   /* We're connecting to an already opened device, so let's make sure 
> we
>  also get the open event */
>   if (s->be_open) {
>   qemu_chr_be_event(s, CHR_EVENT_OPENED);
>   }
>   }
> 
>   if (CHARDEV_IS_MUX(s)) {
>   mux_chr_set_handlers(s, context);
>   }
>   }
> 
> It looks like chr_*() functions must be called from the event loop
> thread that monitors the chardev.  You can use aio_bh_schedule_oneshot()
> or g_idle_source_new() to execute the last part of monitor_init() in the
> GMainContext.
> 
> That way there is no race condition because qemu_chr_fe_set_handlers()
> is called from within the event loop thread.

Good catch.  I'll take your suggestion.  Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC v6 10/27] monitor: allow to use IO thread for parsing

[Peter Xu]

On Tue, Jan 09, 2018 at 05:37:39PM -0600, Eric Blake wrote:
> On 12/19/2017 02:45 AM, Peter Xu wrote:
> 
> Grammar in the subject:
> 
> "allow to $VERB" is not idiomatic English; correct is either "allow
> ${VERB}ing" or "allow $SUBJECT to $VERB".  Concretely, s/to use/using/
> 
> > For each Monitor, add one field "use_io_thr" to show whether it will be
> > using the dedicated monitor IO thread to handle input/output.  When set,
> > monitor IO parsing work will be offloaded to dedicated monitor IO
> 
> s/to/to the/
> 
> > thread, rather than the original main loop thread.
> > 
> > This only works for QMP.  HMP will always be run on main loop thread.
> 
> s/on/on the/
> 
> > 
> > Currently we're still keeping use_io_thr to off always.  Will turn it on
> 
> s/to off/off/
> 
> > later at some point.
> > 
> > One thing to mention is that we cannot set use_io_thr for every QMP
> > monitors.  The problem is that MUXed typed chardevs may not work well
> 
> s/monitors/monitor/
> 
> > with it now. When MUX is used, frontend of chardev can be the monitor
> > plus something else.  The only thing we know would be safe to be run
> > outside main thread so far is the monitor frontend. All the rest of the
> > frontends should still be run in main thread only.
> > 
> > Signed-off-by: Peter Xu 
> > ---
> >  monitor.c | 41 -
> >  1 file changed, 32 insertions(+), 9 deletions(-)
> > 
> I have nothing to add on the code review, but it looks like Stefan had
> valid comments that need addressing.

Yes, will address it with all the grammar fixes above.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v4 9/9] qapi: Add NVMe driver options to the schema

[Fam Zheng]

On Wed, 01/10 08:41, Eric Blake wrote:
> On 01/10/2018 03:18 AM, Fam Zheng wrote:
> > Signed-off-by: Fam Zheng 
> > ---
> >  qapi/block-core.json | 16 +++-
> >  1 file changed, 15 insertions(+), 1 deletion(-)
> > 
> > diff --git a/qapi/block-core.json b/qapi/block-core.json
> > index e94a6881b2..fd6c94454d 100644
> > --- a/qapi/block-core.json
> > +++ b/qapi/block-core.json
> > @@ -2237,7 +2237,7 @@
> >'data': [ 'blkdebug', 'blkverify', 'bochs', 'cloop',
> >  'dmg', 'file', 'ftp', 'ftps', 'gluster', 'host_cdrom',
> >  'host_device', 'http', 'https', 'iscsi', 'luks', 'nbd', 'nfs',
> > -'null-aio', 'null-co', 'parallels', 'qcow', 'qcow2', 'qed',
> > +'null-aio', 'null-co', 'nvme', 'parallels', 'qcow', 'qcow2', 
> > 'qed',
> >  'quorum', 'raw', 'rbd', 'replication', 'sheepdog', 'ssh',
> >  'throttle', 'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat', 'vxhs' ] }
> 
> Missing a doc line:
> @nvme: Since 2.12
> 

Will add, thanks!

Fam

Re: [Qemu-devel] [PATCH v4 3/9] block: Add VFIO based NVMe driver

[Fam Zheng]

On Wed, 01/10 08:44, Eric Blake wrote:
> On 01/10/2018 03:18 AM, Fam Zheng wrote:
> > This is a new protocol driver that exclusively opens a host NVMe
> > controller through VFIO. It achieves better latency than linux-aio by
> > completely bypassing host kernel vfs/block layer.
> > 
> 
> > +static BlockDriver bdrv_nvme = {
> > +.format_name  = "nvme",
> > +.protocol_name= "nvme",
> > +.instance_size= sizeof(BDRVNVMeState),
> > +
> > +.bdrv_parse_filename  = nvme_parse_filename,
> > +.bdrv_file_open   = nvme_file_open,
> > +.bdrv_close   = nvme_close,
> > +.bdrv_getlength   = nvme_getlength,
> > +
> > +.bdrv_co_preadv   = nvme_co_preadv,
> > +.bdrv_co_pwritev  = nvme_co_pwritev,
> > +.bdrv_co_flush_to_disk= nvme_co_flush,
> > +.bdrv_reopen_prepare  = nvme_reopen_prepare,
> > +
> > +.bdrv_co_get_block_status = nvme_co_get_block_status,
> 
> Semantic conflict with my pending patches to switch to byte-based block
> status in the drivers.

Should be very easy to rebase for either of us. :)

Fam

Re: [Qemu-devel] [PATCH v4 3/9] block: Add VFIO based NVMe driver

[Fam Zheng]

On Wed, 01/10 18:33, Stefan Hajnoczi wrote:
> > +ret = event_notifier_init(>irq_notifier, 0);
> > +if (ret) {
> > +error_setg(errp, "Failed to init event notifier");
> > +return ret;
> 
> dma_map_lock should be destroyed.

CoMutexes are initialized by memset so I don't think destroying is necessary:

void qemu_co_mutex_init(CoMutex *mutex)
{
memset(mutex, 0, sizeof(*mutex));
}

(I agree to and will fix other issues you pointed out. Thanks for the quick
review!)

Fam

Re: [Qemu-devel] [PATCH v3 03/21] RISC-V CPU Core Definition

[Palmer Dabbelt]

On Thu, 11 Jan 2018 09:55:36 PST (-0800), Michael Clark wrote:

On Fri, Jan 12, 2018 at 3:37 AM, Richard Henderson <
richard.hender...@linaro.org> wrote:


On 01/10/2018 06:21 PM, Michael Clark wrote:
> +static inline void cpu_get_tb_cpu_state(CPURISCVState *env,
target_ulong *pc,
> +target_ulong *cs_base, uint32_t
*flags)
> +{
> +*pc = env->pc;
> +*cs_base = 0;
> +*flags = 0; /* necessary to avoid compiler warning */
> +}

Actually, at minimum you have to put enough into flags to differentiate
machine
vs supervisor vs user mode.  Otherwise you can wind up running a previously
translated block with the wrong permissions.

The patch I saw from Stefan O'Rear would do nicely.


Thanks for pointing this out.  I've just hopped on board here (and I'm way 
oversubscribed, so I might not be that responsive), so I'm sure to have missed 
a lot.






OK. I didn't see the patch. I'll trawl through my email for the last couple
of weeks before the next spin.


I think this is it

   
https://github.com/riscv/riscv-qemu/commit/a038a2874a3eba27650c164f4622e47a3fe95199.patch

which I've blindly applied to our repo here:

   https://github.com/riscv/riscv-qemu/pull/93

We'll be sure to fix it for the v3.

Thanks!


So any CPU state that can effect translation should be in flags?


That's the first comment in the patch -- though that about exhausts my 
knowledge of it :)


   +/* tb_flags must contain all information that affects execution of ordinary
   + * instructions (helpers can look at the CPURISCVState) */


Assuming that is the case, we could probably join mode with the lower 26
bits of misa and we probably should do a tb_flush on misa CSR writes if any
bits are changed. i.e. if extensions such as MAFDC are added or
removed (muldiv, atomics, fp-single, fp-double, compressed).


I agree: I think any MISA bit change would result in different translation 
behavior.

[Qemu-devel] [resend][PATCH] qga: unset frozen state if no mount points are frozen

[Chen Hanxiao]

From: Chen Hanxiao 

If we set mountpoints to qmp_guest_fsfreeze_freeze_list,
we may got nothing to freeze as all mountpoints are
not valid.
Call ga_unset_frozen in this senario.

Cc: Michael Roth 
Signed-off-by: Chen Hanxiao 
---
Rebase on master

 qga/commands-posix.c | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/qga/commands-posix.c b/qga/commands-posix.c
index e809e382eb..9fd51f1d7a 100644
--- a/qga/commands-posix.c
+++ b/qga/commands-posix.c
@@ -1273,6 +1273,12 @@ int64_t qmp_guest_fsfreeze_freeze_list(bool 
has_mountpoints,
 }
 
 free_fs_mount_list();
+/* We may not issue any FIFREEZE here when had mountpoints.
+ * Just unset ga_state here and ready for the next call.
+ */
+if (has_mountpoints && i == 0) {
+ga_unset_frozen(ga_state);
+}
 return i;
 
 error:
-- 
2.14.3

[Qemu-devel] [PATCH v6 2/3] vhost: fix memslot limit check

[Jay Zhou]

Since used_memslots will be updated to the actual value after
registering memory listener for the first time, move the
memslots limit checking to the right place.

Reviewed-by: Igor Mammedov 
Signed-off-by: Jay Zhou 
---
 hw/virtio/vhost.c | 19 ---
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index e4290ce..69b3599 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1251,13 +1251,6 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
 goto fail;
 }
 
-if (used_memslots > hdev->vhost_ops->vhost_backend_memslots_limit(hdev)) {
-error_report("vhost backend memory slots limit is less"
-" than current number of present memory slots");
-r = -1;
-goto fail;
-}
-
 r = hdev->vhost_ops->vhost_set_owner(hdev);
 if (r < 0) {
 VHOST_OPS_DEBUG("vhost_set_owner failed");
@@ -1339,6 +1332,18 @@ int vhost_dev_init(struct vhost_dev *hdev, void *opaque,
 hdev->memory_changed = false;
 memory_listener_register(>memory_listener, _space_memory);
 QLIST_INSERT_HEAD(_devices, hdev, entry);
+
+if (used_memslots > hdev->vhost_ops->vhost_backend_memslots_limit(hdev)) {
+error_report("vhost backend memory slots limit is less"
+" than current number of present memory slots");
+r = -1;
+if (busyloop_timeout) {
+goto fail_busyloop;
+} else {
+goto fail;
+}
+}
+
 return 0;
 
 fail_busyloop:
-- 
1.8.3.1

[Qemu-devel] [PATCH v6 1/3] vhost: remove assertion to prevent crash

[Jay Zhou]

QEMU will assert on vhost-user backed virtio device hotplug if QEMU is
using more RAM regions than VHOST_MEMORY_MAX_NREGIONS (for example if
it were started with a lot of DIMM devices).

Fix it by returning error instead of asserting and let callers of
vhost_set_mem_table() handle error condition gracefully.

Cc: qemu-sta...@nongnu.org
Signed-off-by: Igor Mammedov 
Signed-off-by: Jay Zhou 
---
 hw/virtio/vhost-user.c | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index 093675e..8500562 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -317,11 +317,14 @@ static int vhost_user_set_mem_table(struct vhost_dev *dev,
  );
 fd = memory_region_get_fd(mr);
 if (fd > 0) {
+if (fd_num == VHOST_MEMORY_MAX_NREGIONS) {
+error_report("Failed preparing vhost-user memory table msg");
+return -1;
+}
 msg.payload.memory.regions[fd_num].userspace_addr = 
reg->userspace_addr;
 msg.payload.memory.regions[fd_num].memory_size  = reg->memory_size;
 msg.payload.memory.regions[fd_num].guest_phys_addr = 
reg->guest_phys_addr;
 msg.payload.memory.regions[fd_num].mmap_offset = offset;
-assert(fd_num < VHOST_MEMORY_MAX_NREGIONS);
 fds[fd_num++] = fd;
 }
 }
-- 
1.8.3.1

[Qemu-devel] [PATCH v6 0/3] vhost: two fixes and used_memslots refactoring

[Jay Zhou]

Jay Zhou (3):
  vhost: remove assertion to prevent crash
  vhost: fix memslot limit check
  vhost: used_memslots refactoring

 hw/virtio/vhost-backend.c | 15 +++-
 hw/virtio/vhost-user.c| 74 +++
 hw/virtio/vhost.c | 30 +---
 include/hw/virtio/vhost-backend.h |  6 ++--
 4 files changed, 86 insertions(+), 39 deletions(-)

--
1.8.3.1

[Qemu-devel] [PATCH v6 3/3] vhost: used_memslots refactoring

[Jay Zhou]

Used_memslots is shared by vhost kernel and user, it is equal to
dev->mem->nregions, which is correct for vhost kernel, but not for
vhost user, the latter one uses memory regions that have file
descriptor. E.g. a VM has a vhost-user NIC and 8(vhost user memslot
upper limit) memory slots, it will be failed to hotplug a new DIMM
device since vhost_has_free_slot() finds no free slot left. It
should be successful if only part of memory slots have file
descriptor, so setting used memslots for vhost-user and
vhost-kernel respectively.

Signed-off-by: Igor Mammedov 
Signed-off-by: Jay Zhou 
Signed-off-by: Liuzhe 
---
 hw/virtio/vhost-backend.c | 15 +++-
 hw/virtio/vhost-user.c| 77 ++-
 hw/virtio/vhost.c | 13 +++
 include/hw/virtio/vhost-backend.h |  6 ++-
 4 files changed, 75 insertions(+), 36 deletions(-)

diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c
index 7f09efa..59def69 100644
--- a/hw/virtio/vhost-backend.c
+++ b/hw/virtio/vhost-backend.c
@@ -15,6 +15,8 @@
 #include "hw/virtio/vhost-backend.h"
 #include "qemu/error-report.h"
 
+static unsigned int vhost_kernel_used_memslots;
+
 static int vhost_kernel_call(struct vhost_dev *dev, unsigned long int request,
  void *arg)
 {
@@ -62,6 +64,11 @@ static int vhost_kernel_memslots_limit(struct vhost_dev *dev)
 return limit;
 }
 
+static bool vhost_kernel_has_free_memslots(struct vhost_dev *dev)
+{
+return vhost_kernel_used_memslots < vhost_kernel_memslots_limit(dev);
+}
+
 static int vhost_kernel_net_set_backend(struct vhost_dev *dev,
 struct vhost_vring_file *file)
 {
@@ -233,11 +240,16 @@ static void vhost_kernel_set_iotlb_callback(struct 
vhost_dev *dev,
 qemu_set_fd_handler((uintptr_t)dev->opaque, NULL, NULL, NULL);
 }
 
+static void vhost_kernel_set_used_memslots(struct vhost_dev *dev)
+{
+vhost_kernel_used_memslots = dev->mem->nregions;
+}
+
 static const VhostOps kernel_ops = {
 .backend_type = VHOST_BACKEND_TYPE_KERNEL,
 .vhost_backend_init = vhost_kernel_init,
 .vhost_backend_cleanup = vhost_kernel_cleanup,
-.vhost_backend_memslots_limit = vhost_kernel_memslots_limit,
+.vhost_backend_has_free_memslots = vhost_kernel_has_free_memslots,
 .vhost_net_set_backend = vhost_kernel_net_set_backend,
 .vhost_scsi_set_endpoint = vhost_kernel_scsi_set_endpoint,
 .vhost_scsi_clear_endpoint = vhost_kernel_scsi_clear_endpoint,
@@ -264,6 +276,7 @@ static const VhostOps kernel_ops = {
 #endif /* CONFIG_VHOST_VSOCK */
 .vhost_set_iotlb_callback = vhost_kernel_set_iotlb_callback,
 .vhost_send_device_iotlb_msg = vhost_kernel_send_device_iotlb_msg,
+.vhost_set_used_memslots = vhost_kernel_set_used_memslots,
 };
 
 int vhost_set_backend_type(struct vhost_dev *dev, VhostBackendType 
backend_type)
diff --git a/hw/virtio/vhost-user.c b/hw/virtio/vhost-user.c
index 8500562..11c7d52 100644
--- a/hw/virtio/vhost-user.c
+++ b/hw/virtio/vhost-user.c
@@ -122,6 +122,8 @@ static VhostUserMsg m __attribute__ ((unused));
 /* The version of the protocol we support */
 #define VHOST_USER_VERSION(0x1)
 
+static bool vhost_user_free_memslots = true;
+
 struct vhost_user {
 CharBackend *chr;
 int slave_fd;
@@ -289,12 +291,43 @@ static int vhost_user_set_log_base(struct vhost_dev *dev, 
uint64_t base,
 return 0;
 }
 
+static int vhost_user_prepare_msg(struct vhost_dev *dev, VhostUserMemory *mem,
+  int *fds)
+{
+int i, fd;
+
+vhost_user_free_memslots = true;
+for (i = 0, mem->nregions = 0; i < dev->mem->nregions; ++i) {
+struct vhost_memory_region *reg = dev->mem->regions + i;
+ram_addr_t offset;
+MemoryRegion *mr;
+
+assert((uintptr_t)reg->userspace_addr == reg->userspace_addr);
+mr = memory_region_from_host((void *)(uintptr_t)reg->userspace_addr,
+ );
+fd = memory_region_get_fd(mr);
+if (fd > 0) {
+if (mem->nregions == VHOST_MEMORY_MAX_NREGIONS) {
+vhost_user_free_memslots = false;
+return -1;
+}
+
+mem->regions[mem->nregions].userspace_addr = reg->userspace_addr;
+mem->regions[mem->nregions].memory_size = reg->memory_size;
+mem->regions[mem->nregions].guest_phys_addr = reg->guest_phys_addr;
+mem->regions[mem->nregions].mmap_offset = offset;
+fds[mem->nregions++] = fd;
+}
+}
+
+return 0;
+}
+
 static int vhost_user_set_mem_table(struct vhost_dev *dev,
 struct vhost_memory *mem)
 {
 int fds[VHOST_MEMORY_MAX_NREGIONS];
-int i, fd;
-size_t fd_num = 0;
+size_t fd_num;
 bool reply_supported = 

Re: [Qemu-devel] [QEMU-PPC] [RFC 1/3] hw/ppc/spapr_caps: Rework spapr_caps to use uint8 internal representation

[Suraj Jitindar Singh]

On Wed, 2018-01-10 at 15:13 +1100, David Gibson wrote:
> On Tue, Jan 09, 2018 at 08:21:01PM +1100, Suraj Jitindar Singh wrote:
> > Currently spapr_caps are tied to boolean values (on or off). This
> > patch
> > reworks the caps so that they can have any value between 0 and 127,
> > inclusive. This allows more capabilities with various values to be
> > represented in the same way internally. Capabilities are numbered
> > in
> > ascending order. The internal representation of capability values
> > is an
> > array of uint8s in the sPAPRMachineState, indexed by capability
> > number.
> > Note: The MSB (0x80) of a capability is reserved to track whether
> > the
> >   capability was set from the command line.
> > 
> > Capabilities can have their own name, description, options, getter
> > and
> > setter functions, type and allow functions. They also each have
> > their own
> > section in the migration stream. Capabilities are only migrated if
> > they
> > were explictly set on the command line, with the assumption that
> > otherwise the default will match.
> > 
> > On migration we ensure that the capability value on the destination
> > is greater than or equal to the capability value from the source.
> > So
> > long at this remains the case then the migration is considered
> > compatible and allowed to continue.
> > 
> > This patch implements generic getter and setter functions for
> > boolean
> > capabilities. It also converts the existings cap-htm, cap-vsx and
> > cap-dfp capabilities to this new format.
> > ---
> >  hw/ppc/spapr.c |  19 +--
> >  hw/ppc/spapr_caps.c| 335 -
> > 
> >  include/hw/ppc/spapr.h |  41 +++---
> >  3 files changed, 222 insertions(+), 173 deletions(-)
> > 
> > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > index d1acfe8858..7fa45729ba 100644
> > --- a/hw/ppc/spapr.c
> > +++ b/hw/ppc/spapr.c
> > @@ -320,7 +320,7 @@ static void
> > spapr_populate_pa_features(sPAPRMachineState *spapr,
> >   */
> >  pa_features[3] |= 0x20;
> >  }
> > -if (spapr_has_cap(spapr, SPAPR_CAP_HTM) && pa_size > 24) {
> > +if (spapr_get_cap(spapr, SPAPR_CAP_HTM) && pa_size > 24) {
> 
> I'd prefer to see explicit >0 or !=0 to emphasise that the returned
> value is now an integer not a bool.

Sure

> 
> >  pa_features[24] |= 0x80;/* Transactional memory
> > support */
> >  }
> >  if (legacy_guest && pa_size > 40) {
> > @@ -563,7 +563,7 @@ static void spapr_populate_cpu_dt(CPUState *cs,
> > void *fdt, int offset,
> >   *
> >   * Only CPUs for which we create core types in
> > spapr_cpu_core.c
> >   * are possible, and all of those have VMX */
> > -if (spapr_has_cap(spapr, SPAPR_CAP_VSX)) {
> > +if (spapr_get_cap(spapr, SPAPR_CAP_VSX)) {
> >  _FDT((fdt_setprop_cell(fdt, offset, "ibm,vmx", 2)));
> >  } else {
> >  _FDT((fdt_setprop_cell(fdt, offset, "ibm,vmx", 1)));
> > @@ -572,7 +572,7 @@ static void spapr_populate_cpu_dt(CPUState *cs,
> > void *fdt, int offset,
> >  /* Advertise DFP (Decimal Floating Point) if available
> >   *   0 / no property == no DFP
> >   *   1   == DFP available */
> > -if (spapr_has_cap(spapr, SPAPR_CAP_DFP)) {
> > +if (spapr_get_cap(spapr, SPAPR_CAP_DFP)) {
> >  _FDT((fdt_setprop_cell(fdt, offset, "ibm,dfp", 1)));
> >  }
> >  
> > @@ -1762,7 +1762,9 @@ static const VMStateDescription vmstate_spapr
> > = {
> >  _spapr_ov5_cas,
> >  _spapr_patb_entry,
> >  _spapr_pending_events,
> > -_spapr_caps,
> > +_spapr_cap_htm,
> > +_spapr_cap_vsx,
> > +_spapr_cap_dfp,
> >  NULL
> >  }
> >  };
> > @@ -2323,8 +2325,6 @@ static void spapr_machine_init(MachineState
> > *machine)
> >  char *filename;
> >  Error *resize_hpt_err = NULL;
> >  
> > -spapr_caps_validate(spapr, _fatal);
> > -
> >  msi_nonbroken = true;
> >  
> >  QLIST_INIT(>phbs);
> > @@ -3834,7 +3834,9 @@ static void
> > spapr_machine_class_init(ObjectClass *oc, void *data)
> >   */
> >  mc->numa_mem_align_shift = 28;
> >  
> > -smc->default_caps = spapr_caps(SPAPR_CAP_VSX | SPAPR_CAP_DFP);
> > +smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_OFF;
> > +smc->default_caps.caps[SPAPR_CAP_VSX] = SPAPR_CAP_ON;
> > +smc->default_caps.caps[SPAPR_CAP_DFP] = SPAPR_CAP_ON;
> >  spapr_caps_add_properties(smc, _abort);
> >  }
> >  
> > @@ -3916,8 +3918,7 @@ static void
> > spapr_machine_2_11_class_options(MachineClass *mc)
> >  sPAPRMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
> >  
> >  spapr_machine_2_12_class_options(mc);
> > -smc->default_caps = spapr_caps(SPAPR_CAP_HTM | SPAPR_CAP_VSX
> > -   | SPAPR_CAP_DFP);
> > +smc->default_caps.caps[SPAPR_CAP_HTM] = SPAPR_CAP_ON;
> >  SET_MACHINE_COMPAT(mc, SPAPR_COMPAT_2_11);
> >  }
> >  
> > diff --git a/hw/ppc/spapr_caps.c b/hw/ppc/spapr_caps.c
> > 

Re: [Qemu-devel] [PATCH] spapr: Correct compatibility mode setting for hotplugged CPUs

[David Gibson]

On Thu, Jan 11, 2018 at 01:24:30PM +0100, Greg Kurz wrote:
> On Fri, 5 Jan 2018 14:07:29 +1100
> David Gibson  wrote:
> 
> > On Thu, Jan 04, 2018 at 06:47:18PM +0100, Greg Kurz wrote:
> > > On Thu,  4 Jan 2018 15:24:05 +1100
> > > David Gibson  wrote:
> > >   
> > > > Currently the pseries machine sets the compatibility mode for the
> > > > guest's cpus in two places: 1) at machine reset and 2) after CAS
> > > > negotiation.
> > > > 
> > > > This means that if we set or negotiate a compatiblity mode, then
> > > > hotplug a cpu, the hotplugged cpu doesn't get the right mode set and
> > > > will incorrectly have the full native features.
> > > > 
> > > > To correct this, we set the compatibility mode on a cpu when it is
> > > > brought online with the 'start-cpu' RTAS call.  Given that we no
> > > > longer need to set the compatibility mode on all CPUs at machine
> > > > reset, so we change that to only set the mode for the boot cpu.
> > > > 
> > > > Signed-off-by: David Gibson 
> > > > ---
> > > >  hw/ppc/spapr.c  | 2 +-
> > > >  hw/ppc/spapr_rtas.c | 8 
> > > >  2 files changed, 9 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > > > index e22888ba06..d1acfe8858 100644
> > > > --- a/hw/ppc/spapr.c
> > > > +++ b/hw/ppc/spapr.c
> > > > @@ -1510,7 +1510,7 @@ static void spapr_machine_reset(void)
> > > >  spapr_ovec_cleanup(spapr->ov5_cas);
> > > >  spapr->ov5_cas = spapr_ovec_new();
> > > >  
> > > > -ppc_set_compat_all(spapr->max_compat_pvr, _fatal);
> > > > +ppc_set_compat(first_ppc_cpu, spapr->max_compat_pvr, 
> > > > _fatal);
> > > >  }
> > > >  
> > > >  fdt = spapr_build_fdt(spapr, rtas_addr, spapr->rtas_size);
> > > > diff --git a/hw/ppc/spapr_rtas.c b/hw/ppc/spapr_rtas.c
> > > > index 4bb939d3d1..2ed00548c1 100644
> > > > --- a/hw/ppc/spapr_rtas.c
> > > > +++ b/hw/ppc/spapr_rtas.c
> > > > @@ -163,6 +163,7 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, 
> > > > sPAPRMachineState *spapr,
> > > >  CPUState *cs = CPU(cpu);
> > > >  CPUPPCState *env = >env;
> > > >  PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
> > > > +Error *local_err = NULL;
> > > >  
> > > >  if (!cs->halted) {
> > > >  rtas_st(rets, 0, RTAS_OUT_HW_ERROR);
> > > > @@ -174,6 +175,13 @@ static void rtas_start_cpu(PowerPCCPU *cpu_, 
> > > > sPAPRMachineState *spapr,
> > > >   * new cpu enters */
> > > >  kvm_cpu_synchronize_state(cs);
> > > >  
> > > > +/* Set compatibility mode to match existing cpus */
> > > > +ppc_set_compat(cpu, POWERPC_CPU(first_cpu)->compat_pvr, 
> > > > _err);  
> > > 
> > > Is it okay to report a simple HW error to the guest here instead of 
> > > aborting
> > > like we do with first_cpu at reset time ?  
> > 
> > Should be: this happens before we turn the cpu on, so the effect will
> > be that the guest fails to online the cpu.  That seems like a better
> > failure mode than killing the already running guest.
> > 
> 
> Of course, I generally agree with the "better failure mode". My point was
> just that if we managed to set the compat mode with the first cpu but we
> fail to propagate the same compat mode to subsequent cpus, then this is
> a bug in QEMU or KVM.

Ah, I see your point.  Yes, I guess that's true.  Nonetheless, I think
the better failure mode is still a better idea, even if something goes
wrong elsewhere.

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [Qemu-devel] [PATCH v4 2/9] util: Introduce vfio helpers

[Fam Zheng]

On Wed, 01/10 13:09, Alex Williamson wrote:
> On Wed, 10 Jan 2018 17:18:39 +0800
> Fam Zheng  wrote:
> 
> > This is a library to manage the host vfio interface, which could be used
> > to implement userspace device driver code in QEMU such as NVMe or net
> > controllers.
> > 
> > Signed-off-by: Fam Zheng 
> > ---
> >  include/qemu/vfio-helpers.h |  30 ++
> >  util/Makefile.objs  |   1 +
> >  util/trace-events   |  11 +
> >  util/vfio-helpers.c | 723 
> > 
> >  4 files changed, 765 insertions(+)
> >  create mode 100644 include/qemu/vfio-helpers.h
> >  create mode 100644 util/vfio-helpers.c
> > 
> > diff --git a/include/qemu/vfio-helpers.h b/include/qemu/vfio-helpers.h
> > new file mode 100644
> > index 00..6bdba3b66e
> > --- /dev/null
> > +++ b/include/qemu/vfio-helpers.h
> ...
> > +/**
> > + * Map a PCI bar area.
> > + */
> > +void *qemu_vfio_pci_map_bar(QEMUVFIOState *s, int index, Error **errp)
> > +{
> > +void *p;
> > +assert_bar_index_valid(s, index);
> > +p = mmap(NULL, MIN(8192, s->bar_region_info[index].size),
> > + PROT_READ | PROT_WRITE, MAP_SHARED,
> > + s->device, s->bar_region_info[index].offset);
> > +if (p == MAP_FAILED) {
> > +error_setg_errno(errp, errno, "Failed to map BAR region");
> > +p = NULL;
> > +}
> > +return p;
> > +}
> > +
> > +/**
> > + * Unmap a PCI bar area.
> > + */
> > +void qemu_vfio_pci_unmap_bar(QEMUVFIOState *s, int index, void *bar)
> > +{
> > +if (bar) {
> > +munmap(bar, MIN(8192, s->bar_region_info[index].size));
> > +}
> > +}
> 
> What's up with this 8KB thing?  Is it perhaps a hack to avoid
> un-mmap'able MSI-X sections of the BAR, which would make this general
> purpose library very specific to devices which only operate in the lower
> 8KB of their MMIO space.  Maybe the interface should have an offset and
> size so that the NVMe driver could implement that dependency.  We could
> also be testing if the region supports mmap, but I suppose trying and
> failing is just as good.  Thanks,

Sounds good, will add the offset and size parameters.

Fam

Re: [Qemu-devel] [PATCH v5 0/4] vhost: two fixes and used_memslots refactoring

[Zhoujian (jay)]

> -Original Message-
> From: Michael S. Tsirkin [mailto:m...@redhat.com]
> Sent: Friday, January 12, 2018 3:56 AM
> To: Zhoujian (jay) 
> Cc: Igor Mammedov ; qemu-devel@nongnu.org; Huangweidong
> (C) ; wangxin (U) ;
> Gonglei (Arei) ; Liuzhe (Ahriy, Euler)
> 
> Subject: Re: [PATCH v5 0/4] vhost: two fixes and used_memslots refactoring
> 
> On Thu, Jan 11, 2018 at 01:55:38PM +, Zhoujian (jay) wrote:
> > Hi Igor,
> >
> > > -Original Message-
> > > From: Igor Mammedov [mailto:imamm...@redhat.com]
> > > Sent: Thursday, January 11, 2018 9:05 PM
> > > To: Zhoujian (jay) 
> > > Cc: qemu-devel@nongnu.org; m...@redhat.com; Huangweidong (C)
> > > ; wangxin (U)
> > > ; Gonglei
> > > (Arei) ; Liuzhe (Ahriy, Euler)
> > > 
> > > Subject: Re: [PATCH v5 0/4] vhost: two fixes and used_memslots
> > > refactoring
> > >
> > > On Wed, 10 Jan 2018 00:39:02 +0800
> > > Jay Zhou  wrote:
> > >
> > > Jay,
> > > considering how non trivial touched code is, would you mind first
> > > adding 'make check' testcases for success/failure paths that you are
> touching?
> > > It would help with merging and ensure that future changes to vhost
> > > won't break memslots handling.
> >
> > Will look into the testcases. Maybe it would take some time since I'm
> > not very familiar with them.
> >
> > Regards,
> > Jay
> 
> 
> ok can you repost whatever's ready please? And please use proper threading
> (e.g. generated with git format-patch --thread=shallow).
> 

Will send a v6 version soon.

PS: Considering vhost memslots series and the patch "tap: do not close fd
if only vhost failed to initialize" are two separate issues, will send
them separately.

Regards,
Jay

> > >
> > > > v4 -> v5:
> > > >   Making the followed up device_add to fall back to userspace
> > > >   virtio when netdev_add fails if vhost force flag does not set.
> > > >
> > > > Jay Zhou (4):
> > > >   vhost: remove assertion to prevent crash
> > > >   tap: do not close fd if only vhost failed to initialize
> > > >   vhost: fix memslot limit check
> > > >   vhost: used_memslots refactoring
> > > >
> > > >  hw/virtio/vhost-backend.c | 15 +++-
> > > >  hw/virtio/vhost-user.c| 74 +++
> 
> > > 
> > > >  hw/virtio/vhost.c | 30 +---
> > > >  include/hw/virtio/vhost-backend.h |  6 ++--
> > > >  net/tap.c | 25 +
> > > >  5 files changed, 104 insertions(+), 46 deletions(-)
> > > >
> > > > --
> > > > 1.8.3.1
> > > >
> > > >

Re: [Qemu-devel] [PATCH v4 2/9] util: Introduce vfio helpers

[Fam Zheng]

On Wed, 01/10 16:52, Stefan Hajnoczi wrote:
> On Wed, Jan 10, 2018 at 05:18:39PM +0800, Fam Zheng wrote:
> > +/* Map [host, host + size) area into a contiguous IOVA address space, and 
> > store
> > + * the result in @iova if not NULL. The caller need to make sure the area 
> > is
> > + * aligned to page size, and mustn't overlap with existing mapping areas 
> > (split
> > + * mapping status within this area is not allowed).
> > + */
> > +int qemu_vfio_dma_map(QEMUVFIOState *s, void *host, size_t size,
> > +  bool temporary, uint64_t *iova)
> > +{
> > +int ret = 0;
> > +int index;
> > +IOVAMapping *mapping;
> > +uint64_t iova0;
> > +
> > +assert(QEMU_PTR_IS_ALIGNED(host, getpagesize()));
> > +assert(QEMU_IS_ALIGNED(size, getpagesize()));
> > +trace_qemu_vfio_dma_map(s, host, size, temporary, iova);
> > +qemu_mutex_lock(>lock);
> > +mapping = qemu_vfio_find_mapping(s, host, );
> > +if (mapping) {
> > +iova0 = mapping->iova + ((uint8_t *)host - (uint8_t 
> > *)mapping->host);
> > +} else {
> > +if (s->high_water_mark - s->low_water_mark + 1 < size) {
> > +ret = -ENOMEM;
> > +goto out;
> > +}
> > +if (!temporary) {
> > +iova0 = s->low_water_mark;
> > +mapping = qemu_vfio_add_mapping(s, host, size, index + 1, 
> > iova0);
> > +if (!mapping) {
> > +ret = -ENOMEM;
> > +goto out;
> > +}
> > +assert(qemu_vfio_verify_mappings(s));
> > +ret = qemu_vfio_do_mapping(s, host, size, iova0);
> > +if (ret) {
> > +qemu_vfio_undo_mapping(s, mapping, NULL);
> > +goto out;
> > +}
> > +s->low_water_mark += size;
> > +qemu_vfio_dump_mappings(s);
> > +} else {
> > +iova0 = s->high_water_mark - size;
> > +ret = qemu_vfio_do_mapping(s, host, size, iova0);
> > +if (ret) {
> > +goto out;
> > +}
> > +s->high_water_mark -= size;
> > +}
> > +}
> > +if (iova) {
> > +*iova = iova0;
> > +}
> > +qemu_mutex_unlock(>lock);
> > +out:
> 
> Unlock needs to be here to avoid leaking the lock.

Yes, will fix.

Fam

[Qemu-devel] [PATCHv3] linux-user: Add getcpu() support

[Samuel Thibault]

Signed-off-by: Samuel Thibault 

---
Difference between v1 and v2: handle failure of put_user_u32 with goto efault;
Difference between v2 and v3: handle failure of sys_getcpu system call
---
 linux-user/syscall.c | 19 +++
 1 file changed, 19 insertions(+)

diff --git a/linux-user/syscall.c b/linux-user/syscall.c
index 11c9116c4a..f8cfaf043a 100644
--- a/linux-user/syscall.c
+++ b/linux-user/syscall.c
@@ -296,6 +296,8 @@ _syscall3(int, sys_sched_getaffinity, pid_t, pid, unsigned 
int, len,
 #define __NR_sys_sched_setaffinity __NR_sched_setaffinity
 _syscall3(int, sys_sched_setaffinity, pid_t, pid, unsigned int, len,
   unsigned long *, user_mask_ptr);
+#define __NR_sys_getcpu __NR_getcpu
+_syscall3(int, sys_getcpu, unsigned *, cpu, unsigned *, node, void *, tcache);
 _syscall4(int, reboot, int, magic1, int, magic2, unsigned int, cmd,
   void *, arg);
 _syscall2(int, capget, struct __user_cap_header_struct *, header,
@@ -10403,6 +10405,23 @@ abi_long do_syscall(void *cpu_env, int num, abi_long 
arg1,
 ret = get_errno(sys_sched_setaffinity(arg1, mask_size, mask));
 }
 break;
+case TARGET_NR_getcpu:
+{
+unsigned cpu, node;
+ret = get_errno(sys_getcpu(arg1 ?  : NULL,
+   arg2 ?  : NULL,
+   NULL));
+if (ret < 0) {
+goto fail;
+}
+if (arg1 && put_user_u32(cpu, arg1)) {
+goto efault;
+}
+if (arg2 && put_user_u32(node, arg2)) {
+goto efault;
+}
+}
+break;
 case TARGET_NR_sched_setparam:
 {
 struct sched_param *target_schp;
-- 
2.15.1

Re: [Qemu-devel] Call for GSoC & Outreachy 2018 mentors & project ideas

[Alistair Francis]

On Wed, Jan 10, 2018 at 4:52 AM, Stefan Hajnoczi  wrote:
> On Tue, Jan 9, 2018 at 9:45 PM, Alistair Francis  wrote:
>> Can anyone who has done this before chime in.
>>
>> What do you think about getting someone to cleanup and improve the GDB
>> support in QEMU? Would that be the right difficulty of task for a GSoC
>> project?
>
> There is not enough information to give feedback on whether this
> project idea is suitable.  What are the specific tasks you'd like the
> student to work on?
>
> In general, I'm sure there are well-defined 12-week project ideas
> around the GDB stub.  New features are easy to propose and are usually
> well-defined (e.g. implement these commands that are documented in the
> GDB protocol documentation).  Cleaning up code is less clear and it
> would depend on exactly what needs to be done.  Interns will not have
> a background in the QEMU codebase and may not be able to make
> judgements about how to structure things, so I would be more careful
> about refactoring/cleanup projects.
>
> Please see my talk about QEMU GSoC for guidelines on project ideas:
> https://www.youtube.com/watch?v=xNVCX7YMUL8=19m11s
> http://vmsplice.net/~stefan/stefanha-kvm-forum-2016.pdf

That helps a lot, thanks for that.

So for a more concrete solution, how would adding support for multi
CPU support to the GDB server sound?

This would allow GDB debugging for the A53 and the R5 on the Xilinx
ZynqMP for example. This is something we have in the Xilinx tree, but
it is in no state to go upstream and really needs to be re-write to be
upstreamable and more generic.

Alistair

>
> Hope this helps,
> Stefan

Re: [Qemu-devel] [RFC v6 11/27] qmp: introduce QMPCapability

[Eric Blake]

On 12/19/2017 02:45 AM, Peter Xu wrote:
> There was no QMP capabilities defined.  Define the first "oob" as

s/was/were/

> capability to allow out-of-band messages.
> 
> Also, touch up qmp-test.c to test the new bits.
> 
> Signed-off-by: Peter Xu 
> ---
>  monitor.c| 10 --
>  qapi-schema.json | 13 +
>  tests/qmp-test.c | 10 +-
>  3 files changed, 30 insertions(+), 3 deletions(-)

I'm assuming later patches will document this?

I'm somewhat a fan of documentation alongside or before implementation,
as getting the general overview right and then checking that the code
matches is a bit nicer than random coding then documenting what we ended
up with.  But I don't know if reordering patches in your series is
necessary, as long as the end product is properly documented.

> +++ b/qapi-schema.json
> @@ -118,6 +118,19 @@
>  ##
>  { 'command': 'qmp_capabilities' }

The client can't request a particular feature alongside the command?  Or
is that in later patches?  With just this patch, the enum QMPCapability
is not introspected, because it is not referenced by any command
(although introspection is a bit moot, since the client will learn what
the host advertises from the initial handshake before the client can
even request introspection).

>  
> +##
> +# @QMPCapability:
> +#
> +# QMP supported capabilities to be broadcasted to the clients.

'broadcast' is one of those weird verbs that doesn't change spelling
when constructing its past tense (there is no 'broadcasted').  However,
I think this description is a bit nicer (and avoids the problematic word
altogether):

Enumeration of capabilities to be advertised during initial client
connection, used for agreeing on particular QMP extension behaviors.

> +#
> +# @oob:   QMP ability to support Out-Of-Band requests.

Rather terse (it doesn't say what Out-Of-Band requests are); even a
pointer to the QMP spec (where OOB is more fully documented) might be
nice (of course, that means we need a patch to docs/interop/qmp-spec.txt
somewhere in the series, especially since this patch renders 2.2.1 in
that document obsolete...)

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

Re: [Qemu-devel] [PATCH v4 01/51] qlit: use QType instead of int

[Eric Blake]

On 01/11/2018 03:32 PM, Marc-André Lureau wrote:
> Suggested-by: Markus Armbruster 
> Signed-off-by: Marc-André Lureau 
> ---
>  include/qapi/qmp/qlit.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 

Reviewed-by: Eric Blake 

> diff --git a/include/qapi/qmp/qlit.h b/include/qapi/qmp/qlit.h
> index b18406bce9..0c6809ef3c 100644
> --- a/include/qapi/qmp/qlit.h
> +++ b/include/qapi/qmp/qlit.h
> @@ -21,7 +21,7 @@ typedef struct QLitDictEntry QLitDictEntry;
>  typedef struct QLitObject QLitObject;
>  
>  struct QLitObject {
> -int type;
> +QType type;
>  union {
>  bool qbool;
>  int64_t qnum;
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org



signature.asc
Description: OpenPGP digital signature

[Qemu-devel] [PATCH v4 47/51] target.json: add a note about query-cpu* not being s390x-specific

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
Acked-by: Cornelia Huck 
---
 qapi/target.json | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/qapi/target.json b/qapi/target.json
index 776f2da5e1..ca463a0854 100644
--- a/qapi/target.json
+++ b/qapi/target.json
@@ -79,6 +79,9 @@
 #  an unknown cpu definition name, unknown properties or properties
 #  with wrong types.
 #
+# Note: this command isn't specific to s390x, but is only implemented
+# on this architecture currently.
+#
 # Since: 2.8.0
 ##
 { 'command': 'query-cpu-model-comparison',
@@ -119,6 +122,9 @@
 #  an unknown cpu definition name, unknown properties or properties
 #  with wrong types.
 #
+# Note: this command isn't specific to s390x, but is only implemented
+# on this architecture currently.
+#
 # Since: 2.8.0
 ##
 { 'command': 'query-cpu-model-baseline',
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 43/51] build-sys: move qmp-introspect per target

[Marc-André Lureau]

The following patches are going to introduce per-target #ifdef in the
schemas.

The introspection data is statically generated once, and must thus be
built per-target to reflect target-specific configuration.

Drop "do_test_visitor_in_qmp_introspect(_schema_qlit)" since the
schema is no longer in a common object. It is covered by the per-target
query-qmp-schema test instead.

Signed-off-by: Marc-André Lureau 
---
 tests/test-qobject-input-visitor.c | 1 -
 Makefile.objs  | 3 +--
 Makefile.target| 2 ++
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/test-qobject-input-visitor.c 
b/tests/test-qobject-input-visitor.c
index 7c1d10e274..ad4c99a4c7 100644
--- a/tests/test-qobject-input-visitor.c
+++ b/tests/test-qobject-input-visitor.c
@@ -1267,7 +1267,6 @@ static void 
test_visitor_in_qmp_introspect(TestInputVisitorData *data,
const void *unused)
 {
 do_test_visitor_in_qmp_introspect(data, _qmp_schema_qlit);
-do_test_visitor_in_qmp_introspect(data, _schema_qlit);
 }
 
 int main(int argc, char **argv)
diff --git a/Makefile.objs b/Makefile.objs
index c8b1bba593..1b4d0ecfbc 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -2,7 +2,7 @@
 # Common libraries for tools and emulators
 stub-obj-y = stubs/ crypto/
 util-obj-y = util/ qobject/ qapi/
-util-obj-y += qmp-introspect.o qapi-types.o qapi-visit.o qapi-event.o
+util-obj-y += qapi-types.o qapi-visit.o qapi-event.o
 
 chardev-obj-y = chardev/
 
@@ -79,7 +79,6 @@ common-obj-$(CONFIG_FDT) += device_tree.o
 # qapi
 
 common-obj-y += qmp-marshal.o
-common-obj-y += qmp-introspect.o
 common-obj-y += qmp.o hmp.o
 endif
 
diff --git a/Makefile.target b/Makefile.target
index f9a9da7e7c..35108c6caf 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -150,6 +150,8 @@ endif
 
 GENERATED_FILES += hmp-commands.h hmp-commands-info.h
 
+obj-y += qmp-introspect.o
+
 endif # CONFIG_SOFTMMU
 
 # Workaround for http://gcc.gnu.org/PR55489, see configure.
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 42/51] qapi: add a 'unit' pragma

[Marc-André Lureau]

Add a pragma that allows to tag the following expressions in the
schema with a unit name. By default, an expression has no unit name.

See the docs/devel/qapi-code-gen.txt for more details.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py| 22 +++---
 docs/devel/qapi-code-gen.txt   |  4 
 tests/Makefile.include |  7 ++-
 tests/qapi-schema/pragma-unit-invalid.err  |  1 +
 tests/qapi-schema/pragma-unit-invalid.exit |  1 +
 tests/qapi-schema/pragma-unit-invalid.json |  3 +++
 tests/qapi-schema/pragma-unit-invalid.out  |  0
 tests/qapi-schema/qapi-schema-test.json|  6 +-
 tests/qapi-schema/qapi-schema-test.out |  2 ++
 9 files changed, 41 insertions(+), 5 deletions(-)
 create mode 100644 tests/qapi-schema/pragma-unit-invalid.err
 create mode 100644 tests/qapi-schema/pragma-unit-invalid.exit
 create mode 100644 tests/qapi-schema/pragma-unit-invalid.json
 create mode 100644 tests/qapi-schema/pragma-unit-invalid.out

diff --git a/scripts/qapi.py b/scripts/qapi.py
index f56460d028..07e738c3f1 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -47,6 +47,9 @@ returns_whitelist = []
 # Whitelist of entities allowed to violate case conventions
 name_case_whitelist = []
 
+# Unit names to include for the visit (default to all units)
+visit_units = []
+
 enum_types = {}
 struct_types = {}
 union_types = {}
@@ -269,11 +272,12 @@ class QAPISchemaParser(object):
 self.exprs = []
 self.docs = []
 self.accept()
+self.unit = None
 cur_doc = None
 
 while self.tok is not None:
 info = {'file': self.fname, 'line': self.line,
-'parent': self.incl_info}
+'parent': self.incl_info, 'unit': self.unit}
 if self.tok == '#':
 self.reject_expr_doc(cur_doc)
 cur_doc = self.get_doc(info)
@@ -362,6 +366,11 @@ class QAPISchemaParser(object):
"Pragma name-case-whitelist must be"
" a list of strings")
 name_case_whitelist = value
+elif name == 'unit':
+if not isinstance(value, str):
+raise QAPISemError(info,
+   "Pragma 'unit' must be string")
+self.unit = value
 else:
 raise QAPISemError(info, "Unknown pragma '%s'" % name)
 
@@ -1827,6 +1836,10 @@ class QAPISchema(object):
 def visit(self, visitor):
 visitor.visit_begin(self)
 for (name, entity) in sorted(self._entity_dict.items()):
+# FIXME: implicit array types should use element type unit
+unit = entity.info and entity.info.get('unit')
+if visit_units and unit not in visit_units:
+continue
 if visitor.visit_needed(entity):
 entity.visit(visitor)
 visitor.visit_end()
@@ -2126,13 +2139,14 @@ def parse_command_line(extra_options='', 
extra_long_options=[]):
 
 try:
 opts, args = getopt.gnu_getopt(
-sys.argv[1:], 'chp:o:i:' + extra_options,
+sys.argv[1:], 'chp:o:u:i:' + extra_options,
 ['source', 'header', 'prefix=', 'output-dir=',
- 'include='] + extra_long_options)
+ 'unit=', 'include='] + extra_long_options)
 except getopt.GetoptError as err:
 print >>sys.stderr, "%s: %s" % (sys.argv[0], str(err))
 sys.exit(1)
 
+global visit_units
 output_dir = ''
 prefix = ''
 do_c = False
@@ -2152,6 +2166,8 @@ def parse_command_line(extra_options='', 
extra_long_options=[]):
 prefix = a
 elif o in ('-o', '--output-dir'):
 output_dir = a + '/'
+elif o in ('-u', '--unit'):
+visit_units.append(a)
 elif o in ('-c', '--source'):
 do_c = True
 elif o in ('-h', '--header'):
diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index 479b755609..31bcbdac58 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -326,6 +326,10 @@ violate the rules on permitted return types.  Default is 
none.
 Pragma 'name-case-whitelist' takes a list of names that may violate
 rules on use of upper- vs. lower-case letters.  Default is none.
 
+Pragma 'unit' takes a string value.  It will set the unit name for the
+following expressions in the schema.  Most code generators can filter
+based on a unit name.  This allows to select a subset of the
+expressions.  Default is none.
 
 === Struct types ===
 
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 2b83f05954..2f39e2d5aa 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -510,6 +510,7 @@ qapi-schema += pragma-extra-junk.json
 qapi-schema += pragma-name-case-whitelist-crap.json
 qapi-schema += pragma-non-dict.json
 qapi-schema += 

[Qemu-devel] [PATCH v4 39/51] qapi: add conditions to REPLICATION type/commands on the schema

[Marc-André Lureau]

Add #if defined(CONFIG_REPLICATION) in generated code, and adjust the
code accordingly.

Made conditional:

* xen-set-replication, query-xen-replication-status,
  xen-colo-do-checkpoint

  Before the patch, we first register the commands unconditionally in
  generated code (requires a stub), then conditionally unregister in
  qmp_unregister_commands_hack().

  Afterwards, we register only when CONFIG_REPLICATION.  The command
  fails exactly the same, with CommandNotFound.

  Improvement, because now query-qmp-schema is accurate, and we're one
  step closer to killing qmp_unregister_commands_hack().

* enum BlockdevDriver value "replication" in command blockdev-add

* BlockdevOptions variant @replication

And related structures.

Signed-off-by: Marc-André Lureau 
---
 qapi/block-core.json | 15 ++-
 qapi/migration.json  | 12 
 migration/colo.c | 16 
 monitor.c|  5 -
 4 files changed, 22 insertions(+), 26 deletions(-)

diff --git a/qapi/block-core.json b/qapi/block-core.json
index e94a6881b2..cc1d495dda 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -2238,8 +2238,10 @@
 'dmg', 'file', 'ftp', 'ftps', 'gluster', 'host_cdrom',
 'host_device', 'http', 'https', 'iscsi', 'luks', 'nbd', 'nfs',
 'null-aio', 'null-co', 'parallels', 'qcow', 'qcow2', 'qed',
-'quorum', 'raw', 'rbd', 'replication', 'sheepdog', 'ssh',
-'throttle', 'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat', 'vxhs' ] }
+'quorum', 'raw', 'rbd',
+{ 'name': 'replication', 'if': 'defined(CONFIG_REPLICATION)' },
+'sheepdog', 'ssh', 'throttle',
+'vdi', 'vhdx', 'vmdk', 'vpc', 'vvfat', 'vxhs' ] }
 
 ##
 # @BlockdevOptionsFile:
@@ -2874,7 +2876,8 @@
 #
 # Since: 2.9
 ##
-{ 'enum' : 'ReplicationMode', 'data' : [ 'primary', 'secondary' ] }
+{ 'enum' : 'ReplicationMode', 'data' : [ 'primary', 'secondary' ],
+  'if': 'defined(CONFIG_REPLICATION)' }
 
 ##
 # @BlockdevOptionsReplication:
@@ -2892,7 +2895,8 @@
 { 'struct': 'BlockdevOptionsReplication',
   'base': 'BlockdevOptionsGenericFormat',
   'data': { 'mode': 'ReplicationMode',
-'*top-id': 'str' } }
+'*top-id': 'str' },
+  'if': 'defined(CONFIG_REPLICATION)' }
 
 ##
 # @NFSTransport:
@@ -3190,7 +3194,8 @@
   'quorum': 'BlockdevOptionsQuorum',
   'raw':'BlockdevOptionsRaw',
   'rbd':'BlockdevOptionsRbd',
-  'replication':'BlockdevOptionsReplication',
+  'replication': { 'type': 'BlockdevOptionsReplication',
+   'if': 'defined(CONFIG_REPLICATION)' },
   'sheepdog':   'BlockdevOptionsSheepdog',
   'ssh':'BlockdevOptionsSsh',
   'throttle':   'BlockdevOptionsThrottle',
diff --git a/qapi/migration.json b/qapi/migration.json
index 03f57c9616..fc80fc751c 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -1115,7 +1115,8 @@
 # Since: 2.9
 ##
 { 'command': 'xen-set-replication',
-  'data': { 'enable': 'bool', 'primary': 'bool', '*failover' : 'bool' } }
+  'data': { 'enable': 'bool', 'primary': 'bool', '*failover' : 'bool' },
+  'if': 'defined(CONFIG_REPLICATION)' }
 
 ##
 # @ReplicationStatus:
@@ -1130,7 +1131,8 @@
 # Since: 2.9
 ##
 { 'struct': 'ReplicationStatus',
-  'data': { 'error': 'bool', '*desc': 'str' } }
+  'data': { 'error': 'bool', '*desc': 'str' },
+  'if': 'defined(CONFIG_REPLICATION)' }
 
 ##
 # @query-xen-replication-status:
@@ -1147,7 +1149,8 @@
 # Since: 2.9
 ##
 { 'command': 'query-xen-replication-status',
-  'returns': 'ReplicationStatus' }
+  'returns': 'ReplicationStatus',
+  'if': 'defined(CONFIG_REPLICATION)' }
 
 ##
 # @xen-colo-do-checkpoint:
@@ -1163,4 +1166,5 @@
 #
 # Since: 2.9
 ##
-{ 'command': 'xen-colo-do-checkpoint' }
+{ 'command': 'xen-colo-do-checkpoint',
+  'if': 'defined(CONFIG_REPLICATION)' }
diff --git a/migration/colo.c b/migration/colo.c
index dee3aa8bf7..010c9e02e8 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -22,7 +22,9 @@
 #include "trace.h"
 #include "qemu/error-report.h"
 #include "migration/failover.h"
+#ifdef CONFIG_REPLICATION
 #include "replication.h"
+#endif
 #include "qmp-commands.h"
 
 static bool vmstate_loading;
@@ -147,11 +149,11 @@ void colo_do_failover(MigrationState *s)
 }
 }
 
+#ifdef CONFIG_REPLICATION
 void qmp_xen_set_replication(bool enable, bool primary,
  bool has_failover, bool failover,
  Error **errp)
 {
-#ifdef CONFIG_REPLICATION
 ReplicationMode mode = primary ?
REPLICATION_MODE_PRIMARY :
REPLICATION_MODE_SECONDARY;
@@ -170,14 +172,10 @@ void qmp_xen_set_replication(bool enable, bool primary,
 }
 replication_stop_all(failover, failover ? NULL : errp);
 }
-#else
-abort();
-#endif
 }
 
 ReplicationStatus *qmp_query_xen_replication_status(Error **errp)
 {
-#ifdef CONFIG_REPLICATION
 

[Qemu-devel] [PATCH v4 38/51] qapi: add conditions to SPICE type/commands/events on the schema

[Marc-André Lureau]

Add #if defined(CONFIG_SPICE) in generated code, and adjust the
qmp/hmp code accordingly.

query-qmp-schema no longer reports the command/events etc as
available when disabled at compile time.

Commands made conditional:

* query-spice

  Before the patch, the command for !CONFIG_SPICE is unregistered. It
  will fail with the same error.

Events made conditional:

* SPICE_CONNECTED, SPICE_INITIALIZED, SPICE_DISCONNECTED,
  SPICE_MIGRATE_COMPLETED

Chardev made conditional:

* spiceport, spicevmc

  Before and after the patch for !CONFIG_SPICE, the error is the
  same ('spiceport' is not a valid char driver name).

No HMP change, the code was already conditional.

Signed-off-by: Marc-André Lureau 
---
 qapi/char.json | 10 ++
 qapi/ui.json   | 30 --
 monitor.c  |  3 ---
 qmp.c  | 16 
 4 files changed, 26 insertions(+), 33 deletions(-)

diff --git a/qapi/char.json b/qapi/char.json
index ae19dcd1ed..7fa1762ae5 100644
--- a/qapi/char.json
+++ b/qapi/char.json
@@ -318,7 +318,8 @@
 # Since: 1.5
 ##
 { 'struct': 'ChardevSpiceChannel', 'data': { 'type'  : 'str' },
-  'base': 'ChardevCommon' }
+  'base': 'ChardevCommon',
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @ChardevSpicePort:
@@ -330,7 +331,8 @@
 # Since: 1.5
 ##
 { 'struct': 'ChardevSpicePort', 'data': { 'fqdn'  : 'str' },
-  'base': 'ChardevCommon' }
+  'base': 'ChardevCommon',
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @ChardevVC:
@@ -384,8 +386,8 @@
'testdev': 'ChardevCommon',
'stdio'  : 'ChardevStdio',
'console': 'ChardevCommon',
-   'spicevmc' : 'ChardevSpiceChannel',
-   'spiceport' : 'ChardevSpicePort',
+   'spicevmc' : { 'type': 
'ChardevSpiceChannel', 'if': 'defined(CONFIG_SPICE)' },
+   'spiceport' : { 'type': 
'ChardevSpicePort', 'if': 'defined(CONFIG_SPICE)' },
'vc' : 'ChardevVC',
'ringbuf': 'ChardevRingbuf',
# next one is just for compatibility
diff --git a/qapi/ui.json b/qapi/ui.json
index 05f1db27c8..17f673d7d1 100644
--- a/qapi/ui.json
+++ b/qapi/ui.json
@@ -110,7 +110,8 @@
 { 'struct': 'SpiceBasicInfo',
   'data': { 'host': 'str',
 'port': 'str',
-'family': 'NetworkAddressFamily' } }
+'family': 'NetworkAddressFamily' },
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @SpiceServerInfo:
@@ -123,7 +124,8 @@
 ##
 { 'struct': 'SpiceServerInfo',
   'base': 'SpiceBasicInfo',
-  'data': { '*auth': 'str' } }
+  'data': { '*auth': 'str' },
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @SpiceChannel:
@@ -148,7 +150,8 @@
 { 'struct': 'SpiceChannel',
   'base': 'SpiceBasicInfo',
   'data': {'connection-id': 'int', 'channel-type': 'int', 'channel-id': 'int',
-   'tls': 'bool'} }
+   'tls': 'bool'},
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @SpiceQueryMouseMode:
@@ -167,7 +170,8 @@
 # Since: 1.1
 ##
 { 'enum': 'SpiceQueryMouseMode',
-  'data': [ 'client', 'server', 'unknown' ] }
+  'data': [ 'client', 'server', 'unknown' ],
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @SpiceInfo:
@@ -204,7 +208,8 @@
 { 'struct': 'SpiceInfo',
   'data': {'enabled': 'bool', 'migrated': 'bool', '*host': 'str', '*port': 
'int',
'*tls-port': 'int', '*auth': 'str', '*compiled-version': 'str',
-   'mouse-mode': 'SpiceQueryMouseMode', '*channels': ['SpiceChannel']} 
}
+   'mouse-mode': 'SpiceQueryMouseMode', '*channels': ['SpiceChannel']},
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @query-spice:
@@ -249,7 +254,8 @@
 #}
 #
 ##
-{ 'command': 'query-spice', 'returns': 'SpiceInfo' }
+{ 'command': 'query-spice', 'returns': 'SpiceInfo',
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @SPICE_CONNECTED:
@@ -274,7 +280,8 @@
 ##
 { 'event': 'SPICE_CONNECTED',
   'data': { 'server': 'SpiceBasicInfo',
-'client': 'SpiceBasicInfo' } }
+'client': 'SpiceBasicInfo' },
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @SPICE_INITIALIZED:
@@ -302,7 +309,8 @@
 ##
 { 'event': 'SPICE_INITIALIZED',
   'data': { 'server': 'SpiceServerInfo',
-'client': 'SpiceChannel' } }
+'client': 'SpiceChannel' },
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @SPICE_DISCONNECTED:
@@ -327,7 +335,8 @@
 ##
 { 'event': 'SPICE_DISCONNECTED',
   'data': { 'server': 'SpiceBasicInfo',
-'client': 'SpiceBasicInfo' } }
+'client': 'SpiceBasicInfo' },
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # @SPICE_MIGRATE_COMPLETED:
@@ -342,7 +351,8 @@
 #  "event": "SPICE_MIGRATE_COMPLETED" }
 #
 ##
-{ 'event': 'SPICE_MIGRATE_COMPLETED' }
+{ 'event': 'SPICE_MIGRATE_COMPLETED',
+  'if': 'defined(CONFIG_SPICE)' }
 
 ##
 # == VNC

[Qemu-devel] [PATCH v4 35/51] qapi2texi: add 'If:' condition to struct members

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi2texi.py| 4 +++-
 tests/qapi-schema/doc-good.json | 3 ++-
 tests/qapi-schema/doc-good.out  | 1 +
 tests/qapi-schema/doc-good.texi | 2 ++
 4 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index d1e69358f6..d760596ce0 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -138,9 +138,11 @@ def texi_member(member, suffix=''):
 """Format a table of members item for an object type member"""
 typ = member.type.doc_type()
 membertype = ': ' + typ if typ else ''
-return '@item @code{%s%s}%s%s\n' % (
+return '@item @code{%s%s}%s%s%s\n' % (
 member.name, membertype,
 ' (optional)' if member.optional else '',
+'\n@b{If:} @code{%s}\n' %
+', '.join(member.ifcond) if member.ifcond else '',
 suffix)
 
 
diff --git a/tests/qapi-schema/doc-good.json b/tests/qapi-schema/doc-good.json
index c7fe08c530..158443b1a3 100644
--- a/tests/qapi-schema/doc-good.json
+++ b/tests/qapi-schema/doc-good.json
@@ -72,7 +72,8 @@
 #
 # Another paragraph (but no @var: line)
 ##
-{ 'struct': 'Variant1', 'data': { 'var1': 'str' } }
+{ 'struct': 'Variant1',
+  'data': { 'var1': { 'type': 'str', 'if': 'defined(IFSTR)' } } }
 
 ##
 # @Variant2:
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 74bf60bb83..972b58c815 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -29,6 +29,7 @@ enum SugaredUnionKind
 member two
 object Variant1
 member var1: str optional=False
+if ['defined(IFSTR)']
 object Variant2
 command cmd q_obj_cmd-arg -> Object
gen=True success_response=True boxed=False
diff --git a/tests/qapi-schema/doc-good.texi b/tests/qapi-schema/doc-good.texi
index c9988c1305..f2a902a06b 100644
--- a/tests/qapi-schema/doc-good.texi
+++ b/tests/qapi-schema/doc-good.texi
@@ -118,6 +118,8 @@ Another paragraph (but no @code{var}: line)
 @b{Members:}
 @table @asis
 @item @code{var1: string}
+@b{If:} @code{defined(IFSTR)}
+
 Not documented
 @end table
 
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 31/51] qapi: add #if conditions to generated code

[Marc-André Lureau]

Wrap generated enum/struct members and code with #if/#endif, using the
.ifcond members added in the previous patches.

Some types generate both enum and struct members for example, so a
step-by-step is unnecessarily complicated to deal with (it would
easily generate invalid intermediary code).

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py|  4 
 scripts/qapi-introspect.py | 13 +
 scripts/qapi-types.py  |  4 
 scripts/qapi-visit.py  |  8 +++-
 4 files changed, 24 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index f24cf8bda6..1668a6da6c 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -2043,11 +2043,13 @@ const QEnumLookup %(c_name)s_lookup = {
 ''',
 c_name=c_name(name))
 for m in members:
+ret += gen_if(m.ifcond)
 index = c_enum_const(name, m.name, prefix)
 ret += mcgen('''
 [%(index)s] = "%(name)s",
 ''',
  index=index, name=m.name)
+ret += gen_endif(m.ifcond)
 
 ret += mcgen('''
 },
@@ -2069,10 +2071,12 @@ typedef enum %(c_name)s {
 c_name=c_name(name))
 
 for m in enum_members:
+ret += gen_if(m.ifcond)
 ret += mcgen('''
 %(c_enum)s,
 ''',
  c_enum=c_enum_const(name, m.name, prefix))
+ret += gen_endif(m.ifcond)
 
 ret += mcgen('''
 } %(c_name)s;
diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 868d12f504..6a66047243 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -141,6 +141,8 @@ const QLitObject %(c_name)s = %(c_string)s;
 ret = {'name': member.name, 'type': self._use_type(member.type)}
 if member.optional:
 ret['default'] = None
+if member.ifcond:
+ret = (ret, member.ifcond)
 return ret
 
 def _gen_variants(self, tag_name, variants):
@@ -148,14 +150,16 @@ const QLitObject %(c_name)s = %(c_string)s;
 'variants': [self._gen_variant(v) for v in variants]}
 
 def _gen_variant(self, variant):
-return {'case': variant.name, 'type': self._use_type(variant.type)}
+return ({'case': variant.name, 'type': self._use_type(variant.type)},
+variant.ifcond)
 
 def visit_builtin_type(self, name, info, json_type):
 self._gen_qlit(name, 'builtin', {'json-type': json_type}, [])
 
 def visit_enum_type(self, name, info, ifcond, members, prefix):
 self._gen_qlit(name, 'enum',
-   {'values': [m.name for m in members]}, ifcond)
+   {'values': [(m.name, m.ifcond) for m in members]},
+   ifcond)
 
 def visit_array_type(self, name, info, ifcond, element_type):
 element = self._use_type(element_type)
@@ -171,8 +175,9 @@ const QLitObject %(c_name)s = %(c_string)s;
 
 def visit_alternate_type(self, name, info, ifcond, variants):
 self._gen_qlit(name, 'alternate',
-   {'members': [{'type': self._use_type(m.type)}
-for m in variants.variants]}, ifcond)
+   {'members': [
+   ({'type': self._use_type(m.type)}, m.ifcond)
+   for m in variants.variants]}, ifcond)
 
 def visit_command(self, name, info, ifcond, arg_type, ret_type,
   gen, success_response, boxed):
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index 75c1823e44..312685c295 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -41,6 +41,7 @@ struct %(c_name)s {
 def gen_struct_members(members):
 ret = ''
 for memb in members:
+ret += gen_if(memb.ifcond)
 if memb.optional:
 ret += mcgen('''
 bool has_%(c_name)s;
@@ -50,6 +51,7 @@ def gen_struct_members(members):
 %(c_type)s %(c_name)s;
 ''',
  c_type=memb.type.c_type(), c_name=c_name(memb.name))
+ret += gen_endif(memb.ifcond)
 return ret
 
 
@@ -129,11 +131,13 @@ def gen_variants(variants):
 c_name=c_name(variants.tag_member.name))
 
 for var in variants.variants:
+ret += gen_if(var.ifcond)
 ret += mcgen('''
 %(c_type)s %(c_name)s;
 ''',
  c_type=var.type.c_unboxed_type(),
  c_name=c_name(var.name))
+ret += gen_endif(var.ifcond)
 
 ret += mcgen('''
 } u;
diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index 7e816ae98e..48ac1c9a42 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -53,6 +53,7 @@ void visit_type_%(c_name)s_members(Visitor *v, %(c_name)s 
*obj, Error **errp)
  c_type=base.c_name())
 
 for memb in members:
+ret += gen_if(memb.ifcond)
 if memb.optional:
 ret += mcgen('''
 if (visit_optional(v, "%(name)s", >has_%(c_name)s)) {
@@ -72,6 +73,7 @@ void 

[Qemu-devel] [PATCH v4 29/51] qapi: add 'if' on union members

[Marc-André Lureau]

Add 'if' key to union members:

{ 'union': 'TestIfUnion', 'data':
'mem': { 'type': 'str', 'if': 'COND'} }

Generated code is not changed by this patch but with "qapi: add #if
conditions to generated code".

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 17 +
 tests/qapi-schema/qapi-schema-test.json |  7 ++-
 tests/qapi-schema/qapi-schema-test.out  | 10 ++
 tests/qapi-schema/test-qapi.py  |  1 +
 4 files changed, 26 insertions(+), 9 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 0f6d5dd179..24d6aef13f 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -803,7 +803,7 @@ def check_union(expr, info):
 for (key, value) in members.items():
 source = "Member of union '%s'" % name
 check_name(info, source, key)
-check_known_keys(info, source, value, ['type'], [])
+check_known_keys(info, source, value, ['type'], ['if'])
 typ = value['type']
 
 # Each value must name a known type
@@ -1460,8 +1460,8 @@ class QAPISchemaObjectTypeVariants(object):
 class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
 role = 'branch'
 
-def __init__(self, name, typ):
-QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
+def __init__(self, name, typ, ifcond=None):
+QAPISchemaObjectTypeMember.__init__(self, name, typ, False, ifcond)
 
 
 class QAPISchemaAlternateType(QAPISchemaType):
@@ -1722,14 +1722,14 @@ class QAPISchema(object):
 def _make_variant(self, case, typ):
 return QAPISchemaObjectTypeVariant(case, typ)
 
-def _make_simple_variant(self, case, typ, info):
+def _make_simple_variant(self, case, typ, ifcond, info):
 if isinstance(typ, list):
 assert len(typ) == 1
 typ = self._make_array_type(typ[0], info)
 typ = self._make_implicit_object_type(
 typ, info, None, self.lookup_type(typ),
 'wrapper', [self._make_member('data', typ, None, info)])
-return QAPISchemaObjectTypeVariant(case, typ)
+return QAPISchemaObjectTypeVariant(case, typ, ifcond)
 
 def _def_union_type(self, expr, info, doc):
 name = expr['union']
@@ -1747,10 +1747,11 @@ class QAPISchema(object):
 for (key, value) in data.iteritems()]
 members = []
 else:
-variants = [self._make_simple_variant(key, value['type'], info)
+variants = [self._make_simple_variant(key, value['type'],
+  value.get('if'), info)
 for (key, value) in data.iteritems()]
-typ = self._make_implicit_enum_type(name, info, ifcond,
-[v.name for v in variants])
+enum = [{'name': v.name, 'if': v.ifcond} for v in variants]
+typ = self._make_implicit_enum_type(name, info, ifcond, enum)
 tag_member = QAPISchemaObjectTypeMember('type', typ, False)
 members = [tag_member]
 self._def_entity(
diff --git a/tests/qapi-schema/qapi-schema-test.json 
b/tests/qapi-schema/qapi-schema-test.json
index e29c9f0769..b4b8a0a2cc 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -200,9 +200,14 @@
   [ 'foo', { 'name' : 'bar', 'if': 'defined(TEST_IF_ENUM_BAR)' } ],
   'if': 'defined(TEST_IF_ENUM)' }
 
-{ 'union': 'TestIfUnion', 'data': { 'foo': 'TestStruct' },
+{ 'union': 'TestIfUnion', 'data':
+  { 'foo': 'TestStruct',
+'union_bar': { 'type': 'str', 'if': 'defined(TEST_IF_UNION_BAR)'} },
   'if': 'defined(TEST_IF_UNION) && defined(TEST_IF_STRUCT)' }
 
+{ 'command': 'TestIfUnionCmd', 'data': { 'union_cmd_arg': 'TestIfUnion' },
+  'if': 'defined(TEST_IF_UNION)' }
+
 { 'alternate': 'TestIfAlternate', 'data': { 'foo': 'int', 'bar': 'TestStruct' 
},
   'if': 'defined(TEST_IF_ALT) && defined(TEST_IF_STRUCT)' }
 
diff --git a/tests/qapi-schema/qapi-schema-test.out 
b/tests/qapi-schema/qapi-schema-test.out
index 734d3f53aa..001fa143cd 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -89,9 +89,16 @@ object TestIfUnion
 member type: TestIfUnionKind optional=False
 tag type
 case foo: q_obj_TestStruct-wrapper
+case union_bar: q_obj_str-wrapper
+if ['defined(TEST_IF_UNION_BAR)']
 if ['defined(TEST_IF_UNION) && defined(TEST_IF_STRUCT)']
+command TestIfUnionCmd q_obj_TestIfUnionCmd-arg -> None
+   gen=True success_response=True boxed=False
+if ['defined(TEST_IF_UNION)']
 enum TestIfUnionKind
 member foo
+member union_bar
+if ['defined(TEST_IF_UNION_BAR)']
 if ['defined(TEST_IF_UNION) && defined(TEST_IF_STRUCT)']
 object TestStruct
 member integer: int optional=False
@@ -239,6 +246,9 @@ object q_obj_TestIfEvent-arg
 member bar: TestIfEnum optional=False
 if ['defined(TEST_IF_EVT_BAR)']

[Qemu-devel] [PATCH v4 51/51] qapi: remove qmp_unregister_command()

[Marc-André Lureau]

This command is no longer needed after the schema is made conditional.

Signed-off-by: Marc-André Lureau 
---
 include/qapi/qmp/dispatch.h | 1 -
 qapi/qmp-registry.c | 8 
 2 files changed, 9 deletions(-)

diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index 20578dcd48..b2c8d2041c 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -38,7 +38,6 @@ typedef QTAILQ_HEAD(QmpCommandList, QmpCommand) 
QmpCommandList;
 
 void qmp_register_command(QmpCommandList *cmds, const char *name,
   QmpCommandFunc *fn, QmpCommandOptions options);
-void qmp_unregister_command(QmpCommandList *cmds, const char *name);
 QmpCommand *qmp_find_command(QmpCommandList *cmds, const char *name);
 QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request);
 void qmp_disable_command(QmpCommandList *cmds, const char *name);
diff --git a/qapi/qmp-registry.c b/qapi/qmp-registry.c
index 5af484cd9a..ca00f74795 100644
--- a/qapi/qmp-registry.c
+++ b/qapi/qmp-registry.c
@@ -27,14 +27,6 @@ void qmp_register_command(QmpCommandList *cmds, const char 
*name,
 QTAILQ_INSERT_TAIL(cmds, cmd, node);
 }
 
-void qmp_unregister_command(QmpCommandList *cmds, const char *name)
-{
-QmpCommand *cmd = qmp_find_command(cmds, name);
-
-QTAILQ_REMOVE(cmds, cmd, node);
-g_free(cmd);
-}
-
 QmpCommand *qmp_find_command(QmpCommandList *cmds, const char *name)
 {
 QmpCommand *cmd;
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 40/51] qapi-commands: don't initialize command list in qmp_init_marshall()

[Marc-André Lureau]

This will let the caller add several list of commands coming from
different schemas (the following patches split the schemas for common
and arch-specific parts).

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi-commands.py  | 2 --
 monitor.c | 1 +
 qga/main.c| 1 +
 tests/test-qmp-commands.c | 1 +
 4 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 8af8d913b9..7455d2b8bb 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -211,8 +211,6 @@ def gen_registry(registry):
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
 {
-QTAILQ_INIT(cmds);
-
 ''',
 c_prefix=c_name(prefix, protect=False))
 ret += registry
diff --git a/monitor.c b/monitor.c
index c3a1a4ea6c..f5a786 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1001,6 +1001,7 @@ void monitor_init_qmp_commands(void)
  *   "qmp_capabilities", to enforce capability negotiation
  */
 
+QTAILQ_INIT(_commands);
 qmp_init_marshal(_commands);
 
 qmp_register_command(_commands, "query-qmp-schema",
diff --git a/qga/main.c b/qga/main.c
index 62a62755bd..b949b1ccb0 100644
--- a/qga/main.c
+++ b/qga/main.c
@@ -1360,6 +1360,7 @@ int main(int argc, char **argv)
 
 config->log_level = G_LOG_LEVEL_ERROR | G_LOG_LEVEL_CRITICAL;
 
+QTAILQ_INIT(_commands);
 qga_qmp_init_marshal(_commands);
 
 init_dfl_pathnames();
diff --git a/tests/test-qmp-commands.c b/tests/test-qmp-commands.c
index ad7b6e4e1d..28ce88e012 100644
--- a/tests/test-qmp-commands.c
+++ b/tests/test-qmp-commands.c
@@ -283,6 +283,7 @@ int main(int argc, char **argv)
 g_test_add_func("/0.15/dealloc_types", test_dealloc_types);
 g_test_add_func("/0.15/dealloc_partial", test_dealloc_partial);
 
+QTAILQ_INIT(_commands);
 test_qmp_init_marshal(_commands);
 g_test_run();
 
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 28/51] qapi: add an error in case a discriminator is conditionnal

[Marc-André Lureau]

Making a discriminator conditonal doesn't make much sense. Instead,
the union could be made conditional.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 11 +--
 tests/Makefile.include  |  1 +
 .../qapi-schema/flat-union-invalid-if-discriminator.err |  1 +
 .../flat-union-invalid-if-discriminator.exit|  1 +
 .../flat-union-invalid-if-discriminator.json| 17 +
 .../qapi-schema/flat-union-invalid-if-discriminator.out |  0
 6 files changed, 29 insertions(+), 2 deletions(-)
 create mode 100644 tests/qapi-schema/flat-union-invalid-if-discriminator.err
 create mode 100644 tests/qapi-schema/flat-union-invalid-if-discriminator.exit
 create mode 100644 tests/qapi-schema/flat-union-invalid-if-discriminator.json
 create mode 100644 tests/qapi-schema/flat-union-invalid-if-discriminator.out

diff --git a/scripts/qapi.py b/scripts/qapi.py
index c4a2623864..0f6d5dd179 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -568,7 +568,8 @@ def find_alternate_member_qtype(qapi_type):
 
 # Return the discriminator enum define if discriminator is specified as an
 # enum type, otherwise return None.
-def discriminator_find_enum_define(expr):
+def discriminator_find_enum_define(expr, info):
+name = expr['union']
 base = expr.get('base')
 discriminator = expr.get('discriminator')
 
@@ -583,6 +584,11 @@ def discriminator_find_enum_define(expr):
 if not discriminator_member:
 return None
 
+if discriminator_member.get('if'):
+raise QAPISemError(info, 'The discriminator %s.%s for union %s '
+   'must not be conditional' %
+   (base, discriminator, name))
+
 return enum_types.get(discriminator_member['type'])
 
 
@@ -1010,7 +1016,8 @@ def check_exprs(exprs):
 # Try again for hidden UnionKind enum
 for expr_elem in exprs:
 expr = expr_elem['expr']
-if 'union' in expr and not discriminator_find_enum_define(expr):
+info = expr_elem['info']
+if 'union' in expr and not discriminator_find_enum_define(expr, info):
 name = '%sKind' % expr['union']
 elif 'alternate' in expr:
 name = '%sKind' % expr['alternate']
diff --git a/tests/Makefile.include b/tests/Makefile.include
index afd2971147..3aec5b0874 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -479,6 +479,7 @@ qapi-schema += flat-union-inline.json
 qapi-schema += flat-union-int-branch.json
 qapi-schema += flat-union-invalid-branch-key.json
 qapi-schema += flat-union-invalid-discriminator.json
+qapi-schema += flat-union-invalid-if-discriminator.json
 qapi-schema += flat-union-no-base.json
 qapi-schema += flat-union-optional-discriminator.json
 qapi-schema += flat-union-string-discriminator.json
diff --git a/tests/qapi-schema/flat-union-invalid-if-discriminator.err 
b/tests/qapi-schema/flat-union-invalid-if-discriminator.err
new file mode 100644
index 00..0c94c9860d
--- /dev/null
+++ b/tests/qapi-schema/flat-union-invalid-if-discriminator.err
@@ -0,0 +1 @@
+tests/qapi-schema/flat-union-invalid-if-discriminator.json:13: The 
discriminator TestBase.enum1 for union TestUnion must not be conditional
diff --git a/tests/qapi-schema/flat-union-invalid-if-discriminator.exit 
b/tests/qapi-schema/flat-union-invalid-if-discriminator.exit
new file mode 100644
index 00..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/flat-union-invalid-if-discriminator.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/flat-union-invalid-if-discriminator.json 
b/tests/qapi-schema/flat-union-invalid-if-discriminator.json
new file mode 100644
index 00..618ec36396
--- /dev/null
+++ b/tests/qapi-schema/flat-union-invalid-if-discriminator.json
@@ -0,0 +1,17 @@
+{ 'enum': 'TestEnum',
+  'data': [ 'value1', 'value2' ] }
+
+{ 'struct': 'TestBase',
+  'data': { 'enum1': { 'type': 'TestEnum', 'if': 'FOO' } } }
+
+{ 'struct': 'TestTypeA',
+  'data': { 'string': 'str' } }
+
+{ 'struct': 'TestTypeB',
+  'data': { 'integer': 'int' } }
+
+{ 'union': 'TestUnion',
+  'base': 'TestBase',
+  'discriminator': 'enum1',
+  'data': { 'value1': 'TestTypeA',
+'value2': 'TestTypeB' } }
diff --git a/tests/qapi-schema/flat-union-invalid-if-discriminator.out 
b/tests/qapi-schema/flat-union-invalid-if-discriminator.out
new file mode 100644
index 00..e69de29bb2
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 44/51] build-sys: add a target schema

[Marc-André Lureau]

This schema is going to contain target-specific commands/events &
types, that can be conditionnally guarded with poisoned defines.

And new rules to compile the schema generated files per-target, using
the 'target' unit name filter.

This new schema is now the top-level schema to generate the
documentation (to include also target-specific commands/types).

Signed-off-by: Marc-André Lureau 
---
 qapi/target.json |  5 +
 monitor.c|  2 ++
 Makefile | 45 +
 Makefile.target  |  2 ++
 4 files changed, 46 insertions(+), 8 deletions(-)
 create mode 100644 qapi/target.json

diff --git a/qapi/target.json b/qapi/target.json
new file mode 100644
index 00..6cac484f68
--- /dev/null
+++ b/qapi/target.json
@@ -0,0 +1,5 @@
+# -*- Mode: Python -*-
+
+{ 'include': '../qapi-schema.json' }
+
+{ 'pragma': { 'unit': 'target' } }
diff --git a/monitor.c b/monitor.c
index f5a786..8a4fbdd09b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -67,6 +67,7 @@
 #include "exec/exec-all.h"
 #include "qemu/log.h"
 #include "qmp-commands.h"
+#include "target-qmp-commands.h"
 #include "hmp.h"
 #include "qemu/thread.h"
 #include "block/qapi.h"
@@ -1003,6 +1004,7 @@ void monitor_init_qmp_commands(void)
 
 QTAILQ_INIT(_commands);
 qmp_init_marshal(_commands);
+target_qmp_init_marshal(_commands);
 
 qmp_register_command(_commands, "query-qmp-schema",
  qmp_query_qmp_schema,
diff --git a/Makefile b/Makefile
index d86ecd2dd4..8a8ae4ce3c 100644
--- a/Makefile
+++ b/Makefile
@@ -87,6 +87,8 @@ include $(SRC_PATH)/rules.mak
 GENERATED_FILES = qemu-version.h config-host.h qemu-options.def
 GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h qapi-event.h
 GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
+GENERATED_FILES += target-qmp-commands.h target-qapi-types.h 
target-qapi-visit.h target-qapi-event.h
+GENERATED_FILES += target-qmp-marshal.c target-qapi-types.c 
target-qapi-visit.c target-qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
 
@@ -485,7 +487,7 @@ $(SRC_PATH)/qga/qapi-schema.json 
$(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
$(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
"GEN","$@")
 
-qapi-modules = $(SRC_PATH)/qapi-schema.json $(SRC_PATH)/qapi/common.json \
+common-qapi-modules = $(SRC_PATH)/qapi-schema.json 
$(SRC_PATH)/qapi/common.json \
$(SRC_PATH)/qapi/block.json $(SRC_PATH)/qapi/block-core.json \
$(SRC_PATH)/qapi/char.json \
$(SRC_PATH)/qapi/crypto.json \
@@ -500,30 +502,57 @@ qapi-modules = $(SRC_PATH)/qapi-schema.json 
$(SRC_PATH)/qapi/common.json \
$(SRC_PATH)/qapi/transaction.json \
$(SRC_PATH)/qapi/ui.json
 
+target-qapi-modules = $(SRC_PATH)/qapi/target.json \
+   $(common-qapi-modules)
+
 qapi-types.c qapi-types.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
+$(common-qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
$(gen-out-type) -o "." -b $<, \
"GEN","$@")
 qapi-visit.c qapi-visit.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
+$(common-qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
$(gen-out-type) -o "." -b $<, \
"GEN","$@")
 qapi-event.c qapi-event.h :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
+$(common-qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-event.py \
$(gen-out-type) -o "." $<, \
"GEN","$@")
 qmp-commands.h qmp-marshal.c :\
-$(qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
+$(common-qapi-modules) $(SRC_PATH)/scripts/qapi-commands.py $(qapi-py)
$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-commands.py \
$(gen-out-type) -o "." $<, \
"GEN","$@")
+
+target-qapi-types.c target-qapi-types.h :\
+$(target-qapi-modules) $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
+   $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-types.py \
+   -i qapi-types.h \
+   $(gen-out-type) -p target- -u target $<, \
+   "GEN","$@")
+target-qapi-visit.c target-qapi-visit.h :\
+$(target-qapi-modules) $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
+   $(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-visit.py \
+   -i qapi-visit.h \
+   $(gen-out-type) -p target- -u target $<, \
+   "GEN","$@")
+target-qapi-event.c target-qapi-event.h :\
+$(target-qapi-modules) $(SRC_PATH)/scripts/qapi-event.py $(qapi-py)
+   $(call quiet-command,$(PYTHON) 

[Qemu-devel] [PATCH v4 48/51] qapi: make query-gic-capabilities depend on TARGET_ARM

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
---
 qapi-schema.json | 43 ---
 qapi/target.json | 45 +
 monitor.c| 11 ---
 target/arm/monitor.c |  2 +-
 4 files changed, 46 insertions(+), 55 deletions(-)

diff --git a/qapi-schema.json b/qapi-schema.json
index ee9b2f53ff..f444e684c0 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -2899,49 +2899,6 @@
 ##
 { 'command': 'xen-load-devices-state', 'data': {'filename': 'str'} }
 
-##
-# @GICCapability:
-#
-# The struct describes capability for a specific GIC (Generic
-# Interrupt Controller) version. These bits are not only decided by
-# QEMU/KVM software version, but also decided by the hardware that
-# the program is running upon.
-#
-# @version:  version of GIC to be described. Currently, only 2 and 3
-#are supported.
-#
-# @emulated: whether current QEMU/hardware supports emulated GIC
-#device in user space.
-#
-# @kernel:   whether current QEMU/hardware supports hardware
-#accelerated GIC device in kernel.
-#
-# Since: 2.6
-##
-{ 'struct': 'GICCapability',
-  'data': { 'version': 'int',
-'emulated': 'bool',
-'kernel': 'bool' } }
-
-##
-# @query-gic-capabilities:
-#
-# This command is ARM-only. It will return a list of GICCapability
-# objects that describe its capability bits.
-#
-# Returns: a list of GICCapability objects.
-#
-# Since: 2.6
-#
-# Example:
-#
-# -> { "execute": "query-gic-capabilities" }
-# <- { "return": [{ "version": 2, "emulated": true, "kernel": false },
-# { "version": 3, "emulated": false, "kernel": true } ] }
-#
-##
-{ 'command': 'query-gic-capabilities', 'returns': ['GICCapability'] }
-
 ##
 # @CpuInstanceProperties:
 #
diff --git a/qapi/target.json b/qapi/target.json
index ca463a0854..7494a7e58f 100644
--- a/qapi/target.json
+++ b/qapi/target.json
@@ -132,3 +132,48 @@
 'modelb': 'CpuModelInfo' },
   'returns': 'CpuModelBaselineInfo',
   'if': 'defined(TARGET_S390X)' }
+
+##
+# @GICCapability:
+#
+# The struct describes capability for a specific GIC (Generic
+# Interrupt Controller) version. These bits are not only decided by
+# QEMU/KVM software version, but also decided by the hardware that
+# the program is running upon.
+#
+# @version:  version of GIC to be described. Currently, only 2 and 3
+#are supported.
+#
+# @emulated: whether current QEMU/hardware supports emulated GIC
+#device in user space.
+#
+# @kernel:   whether current QEMU/hardware supports hardware
+#accelerated GIC device in kernel.
+#
+# Since: 2.6
+##
+{ 'struct': 'GICCapability',
+  'data': { 'version': 'int',
+'emulated': 'bool',
+'kernel': 'bool' },
+  'if': 'defined(TARGET_ARM)' }
+
+##
+# @query-gic-capabilities:
+#
+# This command is ARM-only. It will return a list of GICCapability
+# objects that describe its capability bits.
+#
+# Returns: a list of GICCapability objects.
+#
+# Since: 2.6
+#
+# Example:
+#
+# -> { "execute": "query-gic-capabilities" }
+# <- { "return": [{ "version": 2, "emulated": true, "kernel": false },
+# { "version": 3, "emulated": false, "kernel": true } ] }
+#
+##
+{ 'command': 'query-gic-capabilities', 'returns': ['GICCapability'],
+  'if': 'defined(TARGET_ARM)' }
diff --git a/monitor.c b/monitor.c
index 5d1ae89421..ed719651d9 100644
--- a/monitor.c
+++ b/monitor.c
@@ -971,9 +971,6 @@ static void qmp_query_qmp_schema(QDict *qdict, QObject 
**ret_data,
  */
 static void qmp_unregister_commands_hack(void)
 {
-#ifndef TARGET_ARM
-qmp_unregister_command(_commands, "query-gic-capabilities");
-#endif
 #if !defined(TARGET_S390X) && !defined(TARGET_I386)
 qmp_unregister_command(_commands, "query-cpu-model-expansion");
 #endif
@@ -4135,14 +4132,6 @@ QemuOptsList qemu_mon_opts = {
 },
 };
 
-#ifndef TARGET_ARM
-GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
-{
-error_setg(errp, QERR_FEATURE_DISABLED, "query-gic-capabilities");
-return NULL;
-}
-#endif
-
 HotpluggableCPUList *qmp_query_hotpluggable_cpus(Error **errp)
 {
 MachineState *ms = MACHINE(qdev_get_machine());
diff --git a/target/arm/monitor.c b/target/arm/monitor.c
index 299cb80ae7..a46b0f98b4 100644
--- a/target/arm/monitor.c
+++ b/target/arm/monitor.c
@@ -20,7 +20,7 @@
  * THE SOFTWARE.
  */
 #include "qemu/osdep.h"
-#include "qmp-commands.h"
+#include "target-qmp-commands.h"
 #include "hw/boards.h"
 #include "kvm_arm.h"
 
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 19/51] qapi: change enum visitor to take QAPISchemaMember

[Marc-André Lureau]

This will allow to add and access more properties associated with enum
values/members, like the associated 'if' condition. We may want to
have a specialized type QAPISchemaEnumMember, for now this will do.

Suggested-by: Markus Armbruster 
Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py| 12 ++--
 scripts/qapi-event.py  |  2 +-
 scripts/qapi-introspect.py |  3 ++-
 tests/qapi-schema/test-qapi.py |  2 +-
 4 files changed, 10 insertions(+), 9 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index c7cd98a890..f8c711b81b 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1166,7 +1166,7 @@ class QAPISchemaEnumType(QAPISchemaType):
 
 def visit(self, visitor):
 visitor.visit_enum_type(self.name, self.info, self.ifcond,
-self.member_names(), self.prefix)
+self.members, self.prefix)
 
 
 class QAPISchemaArrayType(QAPISchemaType):
@@ -1966,11 +1966,11 @@ const QEnumLookup %(c_name)s_lookup = {
 ''',
 c_name=c_name(name))
 for m in members:
-index = c_enum_const(name, m, prefix)
+index = c_enum_const(name, m.name, prefix)
 ret += mcgen('''
-[%(index)s] = "%(value)s",
+[%(index)s] = "%(name)s",
 ''',
- index=index, value=m)
+ index=index, name=m.name)
 
 ret += mcgen('''
 },
@@ -1983,7 +1983,7 @@ const QEnumLookup %(c_name)s_lookup = {
 
 def gen_enum(name, members, prefix=None):
 # append automatically generated _MAX value
-enum_members = members + ['_MAX']
+enum_members = members + [QAPISchemaMember('_MAX')]
 
 ret = mcgen('''
 
@@ -1995,7 +1995,7 @@ typedef enum %(c_name)s {
 ret += mcgen('''
 %(c_enum)s,
 ''',
- c_enum=c_enum_const(name, m, prefix))
+ c_enum=c_enum_const(name, m.name, prefix))
 
 ret += mcgen('''
 } %(c_name)s;
diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index bef301dfe9..38f4264817 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -168,7 +168,7 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 def visit_event(self, name, info, ifcond, arg_type, boxed):
 self.decl += gen_event_send_decl(name, arg_type, boxed)
 self.defn += gen_event_send(name, arg_type, boxed)
-self._event_names.append(name)
+self._event_names.append(QAPISchemaMember(name))
 
 
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 9fcb342c91..868d12f504 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -154,7 +154,8 @@ const QLitObject %(c_name)s = %(c_string)s;
 self._gen_qlit(name, 'builtin', {'json-type': json_type}, [])
 
 def visit_enum_type(self, name, info, ifcond, members, prefix):
-self._gen_qlit(name, 'enum', {'values': members}, ifcond)
+self._gen_qlit(name, 'enum',
+   {'values': [m.name for m in members]}, ifcond)
 
 def visit_array_type(self, name, info, ifcond, element_type):
 element = self._use_type(element_type)
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index 1c4f9e0dac..2ab79230c2 100644
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -18,7 +18,7 @@ import sys
 
 class QAPISchemaTestVisitor(QAPISchemaVisitor):
 def visit_enum_type(self, name, info, ifcond, members, prefix):
-print 'enum %s %s' % (name, members)
+print 'enum %s %s' % (name, [m.name for m in members])
 if prefix:
 print 'prefix %s' % prefix
 self._print_if(ifcond)
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 18/51] qapi: rename QAPISchemaEnumType.values to .members

[Marc-André Lureau]

Rename QAPISchemaEnumType.values and related variables to members.
Makes sense ever since commit 93bda4dd4 changed .values from list of
string to list of QAPISchemaMember. Obvious no-op.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py| 36 ++--
 scripts/qapi-introspect.py |  4 ++--
 scripts/qapi-types.py  | 10 +-
 scripts/qapi-visit.py  |  2 +-
 scripts/qapi2texi.py   |  2 +-
 tests/qapi-schema/test-qapi.py |  4 ++--
 6 files changed, 29 insertions(+), 29 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index ee759489cb..c7cd98a890 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1046,7 +1046,7 @@ class QAPISchemaVisitor(object):
 def visit_builtin_type(self, name, info, json_type):
 pass
 
-def visit_enum_type(self, name, info, ifcond, values, prefix):
+def visit_enum_type(self, name, info, ifcond, members, prefix):
 pass
 
 def visit_array_type(self, name, info, ifcond, element_type):
@@ -1134,22 +1134,22 @@ class QAPISchemaBuiltinType(QAPISchemaType):
 
 
 class QAPISchemaEnumType(QAPISchemaType):
-def __init__(self, name, info, doc, ifcond, values, prefix):
+def __init__(self, name, info, doc, ifcond, members, prefix):
 QAPISchemaType.__init__(self, name, info, doc, ifcond)
-for v in values:
-assert isinstance(v, QAPISchemaMember)
-v.set_owner(name)
+for m in members:
+assert isinstance(m, QAPISchemaMember)
+m.set_owner(name)
 assert prefix is None or isinstance(prefix, str)
-self.values = values
+self.members = members
 self.prefix = prefix
 
 def check(self, schema):
 QAPISchemaType.check(self, schema)
 seen = {}
-for v in self.values:
-v.check_clash(self.info, seen)
+for m in self.members:
+m.check_clash(self.info, seen)
 if self.doc:
-self.doc.connect_member(v)
+self.doc.connect_member(m)
 
 def is_implicit(self):
 # See QAPISchema._make_implicit_enum_type() and ._def_predefineds()
@@ -1159,7 +1159,7 @@ class QAPISchemaEnumType(QAPISchemaType):
 return c_name(self.name)
 
 def member_names(self):
-return [v.name for v in self.values]
+return [m.name for m in self.members]
 
 def json_type(self):
 return 'string'
@@ -1958,19 +1958,19 @@ def ifcond_decorator(func):
 return func_wrapper
 
 
-def gen_enum_lookup(name, values, prefix=None):
+def gen_enum_lookup(name, members, prefix=None):
 ret = mcgen('''
 
 const QEnumLookup %(c_name)s_lookup = {
 .array = (const char *const[]) {
 ''',
 c_name=c_name(name))
-for value in values:
-index = c_enum_const(name, value, prefix)
+for m in members:
+index = c_enum_const(name, m, prefix)
 ret += mcgen('''
 [%(index)s] = "%(value)s",
 ''',
- index=index, value=value)
+ index=index, value=m)
 
 ret += mcgen('''
 },
@@ -1981,9 +1981,9 @@ const QEnumLookup %(c_name)s_lookup = {
 return ret
 
 
-def gen_enum(name, values, prefix=None):
+def gen_enum(name, members, prefix=None):
 # append automatically generated _MAX value
-enum_values = values + ['_MAX']
+enum_members = members + ['_MAX']
 
 ret = mcgen('''
 
@@ -1991,11 +1991,11 @@ typedef enum %(c_name)s {
 ''',
 c_name=c_name(name))
 
-for value in enum_values:
+for m in enum_members:
 ret += mcgen('''
 %(c_enum)s,
 ''',
- c_enum=c_enum_const(name, value, prefix))
+ c_enum=c_enum_const(name, m, prefix))
 
 ret += mcgen('''
 } %(c_name)s;
diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 7d3a5c37fd..9fcb342c91 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -153,8 +153,8 @@ const QLitObject %(c_name)s = %(c_string)s;
 def visit_builtin_type(self, name, info, json_type):
 self._gen_qlit(name, 'builtin', {'json-type': json_type}, [])
 
-def visit_enum_type(self, name, info, ifcond, values, prefix):
-self._gen_qlit(name, 'enum', {'values': values}, ifcond)
+def visit_enum_type(self, name, info, ifcond, members, prefix):
+self._gen_qlit(name, 'enum', {'values': members}, ifcond)
 
 def visit_array_type(self, name, info, ifcond, element_type):
 element = self._use_type(element_type)
diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index 789e89ff59..75c1823e44 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -203,16 +203,16 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 self.defn += gen_type_cleanup(name)
 
 @ifcond_decorator
-def visit_enum_type(self, name, info, ifcond, values, prefix):
+def visit_enum_type(self, name, 

[Qemu-devel] [PATCH v4 37/51] qapi: add conditions to VNC type/commands/events on the schema

[Marc-André Lureau]

Add #if defined(CONFIG_VNC) in generated code, and adjust the
qmp/hmp code accordingly.

query-qmp-schema no longer reports the command/events etc as
available when disabled at compile.

Commands made conditional:

* query-vnc, query-vnc-servers, change-vnc-password

  Before the patch, the commands for !CONFIG_VNC are stubs that fail
  like this:

{"error": {"class": "GenericError",
   "desc": "The feature 'vnc' is not enabled"}}

  Afterwards, they fail like this:

{"error": {"class": "CommandNotFound",
   "desc": "The command FOO has not been found"}}

  I call that an improvement, because it lets clients distinguish
  between command unavailable (class CommandNotFound) and command failed
  (class GenericError).

Events made conditional:

* VNC_CONNECTED, VNC_INITIALIZED, VNC_DISCONNECTED

HMP change:

* info vnc

  Will return "unknown command: 'info vnc'" when VNC is compiled
  out (same as error for spice when --disable-spice)

Occurrences of VNC (case insensitive) in the schema that aren't
covered by this change:

* add_client

  Command has other uses, including "socket bases character devices".
  These are unconditional as far as I can tell.

* set_password, expire_password

  In theory, these commands could be used for managing any service's
  password.  In practice, they're used for VNC and SPICE services.
  They're documented for "remote display session" / "remote display
  server".

  The service is selected by argument @protocol.  The code special-cases
  protocol-specific argument checking, then calls a protocol-specific
  function to do the work.  If it fails, the command fails with "Could
  not set password".  It does when the service isn't compiled in (it's a
  stub then).

  We could make these commands conditional on the conjunction of all
  services [currently: defined(CONFIG_VNC) || defined(CONFIG_SPICE)],
  but I doubt it's worthwhile.

* change

  Command has other uses, namely changing media.
  This patch inlines a stub; no functional change.

Signed-off-by: Marc-André Lureau 
---
 qapi/ui.json | 45 -
 ui/vnc.h |  2 ++
 hmp.c|  9 -
 qmp.c| 30 --
 hmp-commands-info.hx |  2 ++
 5 files changed, 44 insertions(+), 44 deletions(-)

diff --git a/qapi/ui.json b/qapi/ui.json
index 07b468f625..05f1db27c8 100644
--- a/qapi/ui.json
+++ b/qapi/ui.json
@@ -369,7 +369,8 @@
   'data': { 'host': 'str',
 'service': 'str',
 'family': 'NetworkAddressFamily',
-'websocket': 'bool' } }
+'websocket': 'bool' },
+  'if': 'defined(CONFIG_VNC)' }
 
 ##
 # @VncServerInfo:
@@ -383,7 +384,8 @@
 ##
 { 'struct': 'VncServerInfo',
   'base': 'VncBasicInfo',
-  'data': { '*auth': 'str' } }
+  'data': { '*auth': 'str' },
+  'if': 'defined(CONFIG_VNC)' }
 
 ##
 # @VncClientInfo:
@@ -400,7 +402,8 @@
 ##
 { 'struct': 'VncClientInfo',
   'base': 'VncBasicInfo',
-  'data': { '*x509_dname': 'str', '*sasl_username': 'str' } }
+  'data': { '*x509_dname': 'str', '*sasl_username': 'str' },
+  'if': 'defined(CONFIG_VNC)' }
 
 ##
 # @VncInfo:
@@ -441,7 +444,8 @@
 { 'struct': 'VncInfo',
   'data': {'enabled': 'bool', '*host': 'str',
'*family': 'NetworkAddressFamily',
-   '*service': 'str', '*auth': 'str', '*clients': ['VncClientInfo']} }
+   '*service': 'str', '*auth': 'str', '*clients': ['VncClientInfo']},
+  'if': 'defined(CONFIG_VNC)' }
 
 ##
 # @VncPrimaryAuth:
@@ -452,7 +456,8 @@
 ##
 { 'enum': 'VncPrimaryAuth',
   'data': [ 'none', 'vnc', 'ra2', 'ra2ne', 'tight', 'ultra',
-'tls', 'vencrypt', 'sasl' ] }
+'tls', 'vencrypt', 'sasl' ],
+  'if': 'defined(CONFIG_VNC)' }
 
 ##
 # @VncVencryptSubAuth:
@@ -466,8 +471,8 @@
 'tls-none',  'x509-none',
 'tls-vnc',   'x509-vnc',
 'tls-plain', 'x509-plain',
-'tls-sasl',  'x509-sasl' ] }
-
+'tls-sasl',  'x509-sasl' ],
+  'if': 'defined(CONFIG_VNC)' }
 
 ##
 # @VncServerInfo2:
@@ -484,8 +489,8 @@
 { 'struct': 'VncServerInfo2',
   'base': 'VncBasicInfo',
   'data': { 'auth'  : 'VncPrimaryAuth',
-'*vencrypt' : 'VncVencryptSubAuth' } }
-
+'*vencrypt' : 'VncVencryptSubAuth' },
+  'if': 'defined(CONFIG_VNC)' }
 
 ##
 # @VncInfo2:
@@ -517,7 +522,8 @@
 'clients'   : ['VncClientInfo'],
 'auth'  : 'VncPrimaryAuth',
 '*vencrypt' : 'VncVencryptSubAuth',
-'*display'  : 'str' } }
+'*display'  : 'str' },
+  'if': 'defined(CONFIG_VNC)' }
 
 ##
 # @query-vnc:
@@ -548,8 +554,8 @@
 #}
 #
 ##
-{ 'command': 'query-vnc', 'returns': 'VncInfo' }
-
+{ 'command': 'query-vnc', 'returns': 'VncInfo',
+  'if': 'defined(CONFIG_VNC)' }
 ##
 # @query-vnc-servers:
 #
@@ -559,7 +565,8 @@
 #
 # Since: 2.3
 ##
-{ 'command': 'query-vnc-servers', 'returns': ['VncInfo2'] }
+{ 'command': 

[Qemu-devel] [PATCH v4 16/51] qapi-types: add #if conditions to types & visitors

[Marc-André Lureau]

Types & visitors are coupled and must be handled together to avoid
temporary build regression.

Wrap generated types/visitor code with #if/#endif using the
ifcond_decorator & helpers.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi-types.py | 18 ++
 scripts/qapi-visit.py |  5 +
 2 files changed, 19 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index 2b3588267b..789e89ff59 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -59,8 +59,10 @@ def gen_variants_objects(variants):
 for v in variants.variants:
 if isinstance(v.type, QAPISchemaObjectType):
 ret += gen_variants_objects(v.type.variants)
+ret += gen_if(v.type.ifcond)
 ret += gen_object(v.type.name, v.type.base,
   v.type.local_members, v.type.variants)
+ret += gen_endif(v.type.ifcond)
 return ret
 
 
@@ -175,6 +177,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 self.defn = None
 self._fwdecl = None
 self._btin = None
+self.if_members = ['decl', 'defn', '_fwdecl', '_btin']
 
 def visit_begin(self, schema):
 # gen_object() is recursive, ensure it doesn't visit the empty type
@@ -199,6 +202,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 self.decl += gen_type_cleanup_decl(name)
 self.defn += gen_type_cleanup(name)
 
+@ifcond_decorator
 def visit_enum_type(self, name, info, ifcond, values, prefix):
 # Special case for our lone builtin enum type
 # TODO use something cleaner than existence of info
@@ -210,6 +214,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 self._fwdecl += gen_enum(name, values, prefix)
 self.defn += gen_enum_lookup(name, values, prefix)
 
+@ifcond_decorator
 def visit_array_type(self, name, info, ifcond, element_type):
 if isinstance(element_type, QAPISchemaBuiltinType):
 self._btin += gen_fwd_object_or_array(name)
@@ -222,6 +227,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 self.decl += gen_array(name, element_type)
 self._gen_type_cleanup(name)
 
+@ifcond_decorator
 def _gen_object(self, name, info, ifcond, base, members, variants):
 self.decl += gen_object(name, base, members, variants)
 if base and not base.is_implicit():
@@ -232,18 +238,22 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 # implicit types won't be directly allocated/freed
 self._gen_type_cleanup(name)
 
+@ifcond_decorator
+def _gen_fwd_object_or_array(self, name, ifcond):
+self._fwdecl += gen_fwd_object_or_array(name)
+
 def visit_object_type(self, name, info, ifcond, base, members, variants):
 # Nothing to do for the special empty builtin
 if name == 'q_empty':
 return
-self._fwdecl += gen_fwd_object_or_array(name)
+self._gen_fwd_object_or_array(name, ifcond)
 self.decl += gen_variants_objects(variants)
-self._gen_object(name, info, None, base, members, variants)
+self._gen_object(name, info, ifcond, base, members, variants)
 
 def visit_alternate_type(self, name, info, ifcond, variants):
-self._fwdecl += gen_fwd_object_or_array(name)
+self._gen_fwd_object_or_array(name, ifcond)
 self.decl += gen_variants_objects(variants)
-self._gen_object(name, info, None, None,
+self._gen_object(name, info, ifcond, None,
  [variants.tag_member], variants)
 
 # If you link code generated from multiple schemata, you want only one
diff --git a/scripts/qapi-visit.py b/scripts/qapi-visit.py
index aceea2a9f9..4b0e005437 100644
--- a/scripts/qapi-visit.py
+++ b/scripts/qapi-visit.py
@@ -267,6 +267,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
 self.decl = None
 self.defn = None
 self._btin = None
+self.if_members = ['decl', 'defn', '_btin']
 
 def visit_begin(self, schema):
 self.decl = ''
@@ -282,6 +283,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
 self.decl = self._btin + self.decl
 self._btin = None
 
+@ifcond_decorator
 def visit_enum_type(self, name, info, ifcond, values, prefix):
 # Special case for our lone builtin enum type
 # TODO use something cleaner than existence of info
@@ -293,6 +295,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
 self.decl += gen_visit_decl(name, scalar=True)
 self.defn += gen_visit_enum(name)
 
+@ifcond_decorator
 def visit_array_type(self, name, info, ifcond, element_type):
 decl = gen_visit_decl(name)
 defn = gen_visit_list(name, element_type)
@@ -304,6 +307,7 @@ class QAPISchemaGenVisitVisitor(QAPISchemaVisitor):
 self.decl += decl
 

[Qemu-devel] [PATCH v4 50/51] qapi: make query-cpu-definitions depend on specific targets

[Marc-André Lureau]

It depends on TARGET_PPC || TARGET_ARM || TARGET_I386 || TARGET_S390X.

Signed-off-by: Marc-André Lureau 
Reviewed-by: Eduardo Habkost 
Acked-by: Cornelia Huck 
---
 qapi-schema.json| 10 --
 qapi/target.json| 12 
 include/sysemu/arch_init.h  |  1 -
 monitor.c   | 22 --
 qmp.c   |  5 -
 stubs/arch-query-cpu-def.c  | 10 --
 target/arm/helper.c |  3 ++-
 target/i386/cpu.c   |  2 +-
 target/ppc/translate_init.c |  3 ++-
 target/s390x/cpu_models.c   |  2 +-
 stubs/Makefile.objs |  1 -
 11 files changed, 18 insertions(+), 53 deletions(-)
 delete mode 100644 stubs/arch-query-cpu-def.c

diff --git a/qapi-schema.json b/qapi-schema.json
index b5b9103180..b0b8e2806d 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1965,16 +1965,6 @@
 ##
 { 'command': 'query-memory-size-summary', 'returns': 'MemoryInfo' }
 
-##
-# @query-cpu-definitions:
-#
-# Return a list of supported virtual CPU definitions
-#
-# Returns: a list of CpuDefInfo
-#
-# Since: 1.2.0
-##
-{ 'command': 'query-cpu-definitions', 'returns': ['CpuDefinitionInfo'] }
 
 ##
 # @CpuModelInfo:
diff --git a/qapi/target.json b/qapi/target.json
index abf1423fba..973da4a849 100644
--- a/qapi/target.json
+++ b/qapi/target.json
@@ -216,3 +216,15 @@
 'model': 'CpuModelInfo' },
   'returns': 'CpuModelExpansionInfo',
   'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
+
+##
+# @query-cpu-definitions:
+#
+# Return a list of supported virtual CPU definitions
+#
+# Returns: a list of CpuDefInfo
+#
+# Since: 1.2.0
+##
+{ 'command': 'query-cpu-definitions', 'returns': ['CpuDefinitionInfo'],
+  'if': 'defined(TARGET_PPC) || defined(TARGET_ARM) || defined(TARGET_I386) || 
defined(TARGET_S390X)' }
diff --git a/include/sysemu/arch_init.h b/include/sysemu/arch_init.h
index 08a607b89b..e9721b9ce8 100644
--- a/include/sysemu/arch_init.h
+++ b/include/sysemu/arch_init.h
@@ -31,5 +31,4 @@ extern const uint32_t arch_type;
 int kvm_available(void);
 int xen_available(void);
 
-CpuDefinitionInfoList *arch_query_cpu_definitions(Error **errp);
 #endif
diff --git a/monitor.c b/monitor.c
index 707cc49385..a8d5af70ad 100644
--- a/monitor.c
+++ b/monitor.c
@@ -957,26 +957,6 @@ static void qmp_query_qmp_schema(QDict *qdict, QObject 
**ret_data,
 *ret_data = qobject_from_qlit(_schema_qlit);
 }
 
-/*
- * We used to define commands in qmp-commands.hx in addition to the
- * QAPI schema.  This permitted defining some of them only in certain
- * configurations.  query-commands has always reflected that (good,
- * because it lets QMP clients figure out what's actually available),
- * while query-qmp-schema never did (not so good).  This function is a
- * hack to keep the configuration-specific commands defined exactly as
- * before, even though qmp-commands.hx is gone.
- *
- * FIXME Educate the QAPI schema on configuration-specific commands,
- * and drop this hack.
- */
-static void qmp_unregister_commands_hack(void)
-{
-#if !defined(TARGET_PPC) && !defined(TARGET_ARM) && !defined(TARGET_I386) \
-&& !defined(TARGET_S390X)
-qmp_unregister_command(_commands, "query-cpu-definitions");
-#endif
-}
-
 void monitor_init_qmp_commands(void)
 {
 /*
@@ -998,8 +978,6 @@ void monitor_init_qmp_commands(void)
 qmp_register_command(_commands, "netdev_add", qmp_netdev_add,
  QCO_NO_OPTIONS);
 
-qmp_unregister_commands_hack();
-
 QTAILQ_INIT(_cap_negotiation_commands);
 qmp_register_command(_cap_negotiation_commands, "qmp_capabilities",
  qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
diff --git a/qmp.c b/qmp.c
index a1f42bcac8..4acf693edf 100644
--- a/qmp.c
+++ b/qmp.c
@@ -536,11 +536,6 @@ DevicePropertyInfoList *qmp_device_list_properties(const 
char *typename,
 return prop_list;
 }
 
-CpuDefinitionInfoList *qmp_query_cpu_definitions(Error **errp)
-{
-return arch_query_cpu_definitions(errp);
-}
-
 void qmp_add_client(const char *protocol, const char *fdname,
 bool has_skipauth, bool skipauth, bool has_tls, bool tls,
 Error **errp)
diff --git a/stubs/arch-query-cpu-def.c b/stubs/arch-query-cpu-def.c
deleted file mode 100644
index cefe4beb82..00
--- a/stubs/arch-query-cpu-def.c
+++ /dev/null
@@ -1,10 +0,0 @@
-#include "qemu/osdep.h"
-#include "qemu-common.h"
-#include "sysemu/arch_init.h"
-#include "qapi/qmp/qerror.h"
-
-CpuDefinitionInfoList *arch_query_cpu_definitions(Error **errp)
-{
-error_setg(errp, QERR_UNSUPPORTED);
-return NULL;
-}
diff --git a/target/arm/helper.c b/target/arm/helper.c
index d1395f9b73..eafa640687 100644
--- a/target/arm/helper.c
+++ b/target/arm/helper.c
@@ -15,6 +15,7 @@
 #include  /* For crc32 */
 #include "exec/semihost.h"
 #include "sysemu/kvm.h"
+#include "target-qmp-commands.h"
 
 #define ARM_CPU_FREQ 10 /* 

[Qemu-devel] [PATCH v4 33/51] qapi2texi: add 'If:' section to generated documentation

[Marc-André Lureau]

The documentation is generated only once, and doesn't know C
pre-conditions. Add 'If:' sections for top-level entities.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi2texi.py| 23 +--
 tests/qapi-schema/doc-good.json |  2 +-
 tests/qapi-schema/doc-good.out  |  1 +
 tests/qapi-schema/doc-good.texi |  2 ++
 4 files changed, 17 insertions(+), 11 deletions(-)

diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index 755bd2f15d..76c336af34 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -174,7 +174,7 @@ def texi_members(doc, what, base, variants, member_func):
 return '\n@b{%s:}\n@table @asis\n%s@end table\n' % (what, items)
 
 
-def texi_sections(doc):
+def texi_sections(doc, ifcond):
 """Format additional sections following arguments"""
 body = ''
 for section in doc.sections:
@@ -185,14 +185,16 @@ def texi_sections(doc):
 body += texi_example(section.text)
 else:
 body += texi_format(section.text)
+if ifcond:
+body += '\n\n@b{If:} @code{%s}' % ", ".join(ifcond)
 return body
 
 
-def texi_entity(doc, what, base=None, variants=None,
+def texi_entity(doc, what, ifcond, base=None, variants=None,
 member_func=texi_member):
 return (texi_body(doc)
 + texi_members(doc, what, base, variants, member_func)
-+ texi_sections(doc))
++ texi_sections(doc, ifcond))
 
 
 class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
@@ -207,7 +209,7 @@ class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
 doc = self.cur_doc
 self.out += TYPE_FMT(type='Enum',
  name=doc.symbol,
- body=texi_entity(doc, 'Values',
+ body=texi_entity(doc, 'Values', ifcond,
   member_func=texi_enum_value))
 
 def visit_object_type(self, name, info, ifcond, base, members, variants):
@@ -216,13 +218,14 @@ class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
 base = None
 self.out += TYPE_FMT(type='Object',
  name=doc.symbol,
- body=texi_entity(doc, 'Members', base, variants))
+ body=texi_entity(doc, 'Members', ifcond,
+  base, variants))
 
 def visit_alternate_type(self, name, info, ifcond, variants):
 doc = self.cur_doc
 self.out += TYPE_FMT(type='Alternate',
  name=doc.symbol,
- body=texi_entity(doc, 'Members'))
+ body=texi_entity(doc, 'Members', ifcond))
 
 def visit_command(self, name, info, ifcond, arg_type, ret_type,
   gen, success_response, boxed):
@@ -231,9 +234,9 @@ class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
 body = texi_body(doc)
 body += ('\n@b{Arguments:} the members of @code{%s}\n'
  % arg_type.name)
-body += texi_sections(doc)
+body += texi_sections(doc, ifcond)
 else:
-body = texi_entity(doc, 'Arguments')
+body = texi_entity(doc, 'Arguments', ifcond)
 self.out += MSG_FMT(type='Command',
 name=doc.symbol,
 body=body)
@@ -242,7 +245,7 @@ class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
 doc = self.cur_doc
 self.out += MSG_FMT(type='Event',
 name=doc.symbol,
-body=texi_entity(doc, 'Arguments'))
+body=texi_entity(doc, 'Arguments', ifcond))
 
 def symbol(self, doc, entity):
 if self.out:
@@ -255,7 +258,7 @@ class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
 assert not doc.args
 if self.out:
 self.out += '\n'
-self.out += texi_body(doc) + texi_sections(doc)
+self.out += texi_body(doc) + texi_sections(doc, None)
 
 
 def texi_schema(schema):
diff --git a/tests/qapi-schema/doc-good.json b/tests/qapi-schema/doc-good.json
index 97ab4625ff..984cd8ed06 100644
--- a/tests/qapi-schema/doc-good.json
+++ b/tests/qapi-schema/doc-good.json
@@ -55,7 +55,7 @@
 #
 # @two is undocumented
 ##
-{ 'enum': 'Enum', 'data': [ 'one', 'two' ] }
+{ 'enum': 'Enum', 'data': [ 'one', 'two' ], 'if': 'defined(IFCOND)' }
 
 ##
 # @Base:
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index e615b04281..8c75da180c 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -3,6 +3,7 @@ object Base
 enum Enum
 member one
 member two
+if ['defined(IFCOND)']
 object Object
 base Base
 tag base1
diff --git a/tests/qapi-schema/doc-good.texi b/tests/qapi-schema/doc-good.texi
index 1778312581..cd422f8a8f 100644
--- a/tests/qapi-schema/doc-good.texi
+++ 

[Qemu-devel] [PATCH v4 49/51] qapi: make query-cpu-model-expansion depend on s390 or x86

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
Reviewed-by: Eduardo Habkost 
Acked-by: Cornelia Huck 
---
 qapi-schema.json   | 38 -
 qapi/target.json   | 39 ++
 include/sysemu/arch_init.h |  3 ---
 monitor.c  |  3 ---
 qmp.c  |  7 --
 stubs/arch-query-cpu-model-expansion.c | 12 ---
 target/i386/cpu.c  |  3 ++-
 target/s390x/cpu_models.c  |  2 +-
 stubs/Makefile.objs|  1 -
 9 files changed, 42 insertions(+), 66 deletions(-)
 delete mode 100644 stubs/arch-query-cpu-model-expansion.c

diff --git a/qapi-schema.json b/qapi-schema.json
index f444e684c0..b5b9103180 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -2040,44 +2040,6 @@
   'data': { 'model': 'CpuModelInfo' } }
 
 
-##
-# @query-cpu-model-expansion:
-#
-# Expands a given CPU model (or a combination of CPU model + additional 
options)
-# to different granularities, allowing tooling to get an understanding what a
-# specific CPU model looks like in QEMU under a certain configuration.
-#
-# This interface can be used to query the "host" CPU model.
-#
-# The data returned by this command may be affected by:
-#
-# * QEMU version: CPU models may look different depending on the QEMU version.
-#   (Except for CPU models reported as "static" in query-cpu-definitions.)
-# * machine-type: CPU model  may look different depending on the machine-type.
-#   (Except for CPU models reported as "static" in query-cpu-definitions.)
-# * machine options (including accelerator): in some architectures, CPU models
-#   may look different depending on machine and accelerator options. (Except 
for
-#   CPU models reported as "static" in query-cpu-definitions.)
-# * "-cpu" arguments and global properties: arguments to the -cpu option and
-#   global properties may affect expansion of CPU models. Using
-#   query-cpu-model-expansion while using these is not advised.
-#
-# Some architectures may not support all expansion types. s390x supports
-# "full" and "static".
-#
-# Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
-#  not supported, if the model cannot be expanded, if the model 
contains
-#  an unknown CPU definition name, unknown properties or properties
-#  with a wrong type. Also returns an error if an expansion type is
-#  not supported.
-#
-# Since: 2.8.0
-##
-{ 'command': 'query-cpu-model-expansion',
-  'data': { 'type': 'CpuModelExpansionType',
-'model': 'CpuModelInfo' },
-  'returns': 'CpuModelExpansionInfo' }
-
 ##
 # @CpuModelCompareResult:
 #
diff --git a/qapi/target.json b/qapi/target.json
index 7494a7e58f..abf1423fba 100644
--- a/qapi/target.json
+++ b/qapi/target.json
@@ -177,3 +177,42 @@
 ##
 { 'command': 'query-gic-capabilities', 'returns': ['GICCapability'],
   'if': 'defined(TARGET_ARM)' }
+
+##
+# @query-cpu-model-expansion:
+#
+# Expands a given CPU model (or a combination of CPU model + additional 
options)
+# to different granularities, allowing tooling to get an understanding what a
+# specific CPU model looks like in QEMU under a certain configuration.
+#
+# This interface can be used to query the "host" CPU model.
+#
+# The data returned by this command may be affected by:
+#
+# * QEMU version: CPU models may look different depending on the QEMU version.
+#   (Except for CPU models reported as "static" in query-cpu-definitions.)
+# * machine-type: CPU model  may look different depending on the machine-type.
+#   (Except for CPU models reported as "static" in query-cpu-definitions.)
+# * machine options (including accelerator): in some architectures, CPU models
+#   may look different depending on machine and accelerator options. (Except 
for
+#   CPU models reported as "static" in query-cpu-definitions.)
+# * "-cpu" arguments and global properties: arguments to the -cpu option and
+#   global properties may affect expansion of CPU models. Using
+#   query-cpu-model-expansion while using these is not advised.
+#
+# Some architectures may not support all expansion types. s390x supports
+# "full" and "static".
+#
+# Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
+#  not supported, if the model cannot be expanded, if the model 
contains
+#  an unknown CPU definition name, unknown properties or properties
+#  with a wrong type. Also returns an error if an expansion type is
+#  not supported.
+#
+# Since: 2.8.0
+##
+{ 'command': 'query-cpu-model-expansion',
+  'data': { 'type': 'CpuModelExpansionType',
+'model': 'CpuModelInfo' },
+  'returns': 'CpuModelExpansionInfo',
+  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
diff --git a/include/sysemu/arch_init.h b/include/sysemu/arch_init.h
index 

[Qemu-devel] [PATCH v4 15/51] qapi-types: refactor variants handling

[Marc-André Lureau]

Generate variants objects outside gen_object(). This will allow to
easily wrap gen_object() with ifcond_decorator in the following patch.

gen_variants_objects() calls gen_object() for each variants, so it
remains guarded for each generated variant object.

self._gen_type_cleanup(name) is factored out in _gen_object(), helping
generated code to be wrapped by the same condition in the following
patch.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi-types.py | 37 +++--
 1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/scripts/qapi-types.py b/scripts/qapi-types.py
index 915786c463..2b3588267b 100644
--- a/scripts/qapi-types.py
+++ b/scripts/qapi-types.py
@@ -53,23 +53,27 @@ def gen_struct_members(members):
 return ret
 
 
-def gen_object(name, base, members, variants):
-if name in objects_seen:
-return ''
-objects_seen.add(name)
-
+def gen_variants_objects(variants):
 ret = ''
 if variants:
 for v in variants.variants:
 if isinstance(v.type, QAPISchemaObjectType):
+ret += gen_variants_objects(v.type.variants)
 ret += gen_object(v.type.name, v.type.base,
   v.type.local_members, v.type.variants)
+return ret
 
-ret += mcgen('''
+
+def gen_object(name, base, members, variants):
+if name in objects_seen:
+return ''
+objects_seen.add(name)
+
+ret = mcgen('''
 
 struct %(c_name)s {
 ''',
- c_name=c_name(name))
+c_name=c_name(name))
 
 if base:
 if not base.is_implicit():
@@ -218,11 +222,7 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 self.decl += gen_array(name, element_type)
 self._gen_type_cleanup(name)
 
-def visit_object_type(self, name, info, ifcond, base, members, variants):
-# Nothing to do for the special empty builtin
-if name == 'q_empty':
-return
-self._fwdecl += gen_fwd_object_or_array(name)
+def _gen_object(self, name, info, ifcond, base, members, variants):
 self.decl += gen_object(name, base, members, variants)
 if base and not base.is_implicit():
 self.decl += gen_upcast(name, base)
@@ -232,10 +232,19 @@ class QAPISchemaGenTypeVisitor(QAPISchemaVisitor):
 # implicit types won't be directly allocated/freed
 self._gen_type_cleanup(name)
 
+def visit_object_type(self, name, info, ifcond, base, members, variants):
+# Nothing to do for the special empty builtin
+if name == 'q_empty':
+return
+self._fwdecl += gen_fwd_object_or_array(name)
+self.decl += gen_variants_objects(variants)
+self._gen_object(name, info, None, base, members, variants)
+
 def visit_alternate_type(self, name, info, ifcond, variants):
 self._fwdecl += gen_fwd_object_or_array(name)
-self.decl += gen_object(name, None, [variants.tag_member], variants)
-self._gen_type_cleanup(name)
+self.decl += gen_variants_objects(variants)
+self._gen_object(name, info, None, None,
+ [variants.tag_member], variants)
 
 # If you link code generated from multiple schemata, you want only one
 # instance of the code for built-in types.  Generate it only when
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 36/51] qapi2texi: add condition to variants

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi2texi.py| 5 +++--
 tests/qapi-schema/doc-good.json | 4 ++--
 tests/qapi-schema/doc-good.out  | 3 +++
 tests/qapi-schema/doc-good.texi | 4 ++--
 4 files changed, 10 insertions(+), 6 deletions(-)

diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index d760596ce0..d8a0379c52 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -165,8 +165,9 @@ def texi_members(doc, what, base, variants, member_func):
 items += '@item The members of @code{%s}\n' % base.doc_type()
 if variants:
 for v in variants.variants:
-when = ' when @code{%s} is @t{"%s"}' % (
-variants.tag_member.name, v.name)
+when = ' when @code{%s} is @t{"%s"}%s' % (
+variants.tag_member.name, v.name,
+' (@b{If:} @code{%s})' % v.ifcond if v.ifcond else '')
 if v.type.is_implicit():
 assert not v.type.base and not v.type.variants
 for m in v.type.local_members:
diff --git a/tests/qapi-schema/doc-good.json b/tests/qapi-schema/doc-good.json
index 158443b1a3..afe46d93f0 100644
--- a/tests/qapi-schema/doc-good.json
+++ b/tests/qapi-schema/doc-good.json
@@ -86,13 +86,13 @@
 { 'union': 'Object',
   'base': 'Base',
   'discriminator': 'base1',
-  'data': { 'one': 'Variant1', 'two': 'Variant2' } }
+  'data': { 'one': 'Variant1', 'two': { 'type': 'Variant2', 'if': 'IFTWO' } } }
 
 ##
 # @SugaredUnion:
 ##
 { 'union': 'SugaredUnion',
-  'data': { 'one': 'Variant1', 'two': 'Variant2' } }
+  'data': { 'one': 'Variant1', 'two': { 'type': 'Variant2', 'if': 'IFTWO' } } }
 
 ##
 # == Another subsection
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 972b58c815..9a9d745f8e 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -10,6 +10,7 @@ object Object
 tag base1
 case one: Variant1
 case two: Variant2
+if ['IFTWO']
 enum QType
 prefix QTYPE
 member none
@@ -24,9 +25,11 @@ object SugaredUnion
 tag type
 case one: q_obj_Variant1-wrapper
 case two: q_obj_Variant2-wrapper
+if ['IFTWO']
 enum SugaredUnionKind
 member one
 member two
+if ['IFTWO']
 object Variant1
 member var1: str optional=False
 if ['defined(IFSTR)']
diff --git a/tests/qapi-schema/doc-good.texi b/tests/qapi-schema/doc-good.texi
index f2a902a06b..3d513a89b7 100644
--- a/tests/qapi-schema/doc-good.texi
+++ b/tests/qapi-schema/doc-good.texi
@@ -143,7 +143,7 @@ Not documented
 @table @asis
 @item The members of @code{Base}
 @item The members of @code{Variant1} when @code{base1} is @t{"one"}
-@item The members of @code{Variant2} when @code{base1} is @t{"two"}
+@item The members of @code{Variant2} when @code{base1} is @t{"two"} (@b{If:} 
@code{['IFTWO']})
 @end table
 
 @end deftp
@@ -159,7 +159,7 @@ Not documented
 @item @code{type}
 One of @t{"one"}, @t{"two"}
 @item @code{data: Variant1} when @code{type} is @t{"one"}
-@item @code{data: Variant2} when @code{type} is @t{"two"}
+@item @code{data: Variant2} when @code{type} is @t{"two"} (@b{If:} 
@code{['IFTWO']})
 @end table
 
 @end deftp
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 14/51] qapi-event: add #if conditions to events

[Marc-André Lureau]

Wrap generated code with #if/#endif using the ifcond_decorator.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi-event.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index dda496e824..bef301dfe9 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -152,6 +152,7 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 self.decl = None
 self.defn = None
 self._event_names = None
+self.if_members = ['decl', 'defn']
 
 def visit_begin(self, schema):
 self.decl = ''
@@ -163,6 +164,7 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 self.defn += gen_enum_lookup(event_enum_name, self._event_names)
 self._event_names = None
 
+@ifcond_decorator
 def visit_event(self, name, info, ifcond, arg_type, boxed):
 self.decl += gen_event_send_decl(name, arg_type, boxed)
 self.defn += gen_event_send(name, arg_type, boxed)
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 46/51] qapi: make s390 commands depend on TARGET_S390X

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
Acked-by: Cornelia Huck 
---
 qapi-schema.json| 101 ---
 qapi/target.json| 104 
 include/sysemu/arch_init.h  |   7 ---
 hw/s390x/s390-skeys.c   |   2 +-
 monitor.c   |  14 -
 qmp.c   |  14 -
 stubs/arch-query-cpu-model-baseline.c   |  12 
 stubs/arch-query-cpu-model-comparison.c |  12 
 target/s390x/cpu_models.c   |   5 +-
 stubs/Makefile.objs |   2 -
 10 files changed, 108 insertions(+), 165 deletions(-)
 delete mode 100644 stubs/arch-query-cpu-model-baseline.c
 delete mode 100644 stubs/arch-query-cpu-model-comparison.c

diff --git a/qapi-schema.json b/qapi-schema.json
index 2cd13a07c4..ee9b2f53ff 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -1759,27 +1759,6 @@
 { 'command': 'query-dump-guest-memory-capability',
   'returns': 'DumpGuestMemoryCapability' }
 
-##
-# @dump-skeys:
-#
-# Dump guest's storage keys
-#
-# @filename: the path to the file to dump to
-#
-# This command is only supported on s390 architecture.
-#
-# Since: 2.5
-#
-# Example:
-#
-# -> { "execute": "dump-skeys",
-#  "arguments": { "filename": "/tmp/skeys" } }
-# <- { "return": {} }
-#
-##
-{ 'command': 'dump-skeys',
-  'data': { 'filename': 'str' } }
-
 ##
 # @object-add:
 #
@@ -2146,46 +2125,6 @@
   }
 }
 
-##
-# @query-cpu-model-comparison:
-#
-# Compares two CPU models, returning how they compare in a specific
-# configuration. The results indicates how both models compare regarding
-# runnability. This result can be used by tooling to make decisions if a
-# certain CPU model will run in a certain configuration or if a compatible
-# CPU model has to be created by baselining.
-#
-# Usually, a CPU model is compared against the maximum possible CPU model
-# of a certain configuration (e.g. the "host" model for KVM). If that CPU
-# model is identical or a subset, it will run in that configuration.
-#
-# The result returned by this command may be affected by:
-#
-# * QEMU version: CPU models may look different depending on the QEMU version.
-#   (Except for CPU models reported as "static" in query-cpu-definitions.)
-# * machine-type: CPU model may look different depending on the machine-type.
-#   (Except for CPU models reported as "static" in query-cpu-definitions.)
-# * machine options (including accelerator): in some architectures, CPU models
-#   may look different depending on machine and accelerator options. (Except 
for
-#   CPU models reported as "static" in query-cpu-definitions.)
-# * "-cpu" arguments and global properties: arguments to the -cpu option and
-#   global properties may affect expansion of CPU models. Using
-#   query-cpu-model-expansion while using these is not advised.
-#
-# Some architectures may not support comparing CPU models. s390x supports
-# comparing CPU models.
-#
-# Returns: a CpuModelBaselineInfo. Returns an error if comparing CPU models is
-#  not supported, if a model cannot be used, if a model contains
-#  an unknown cpu definition name, unknown properties or properties
-#  with wrong types.
-#
-# Since: 2.8.0
-##
-{ 'command': 'query-cpu-model-comparison',
-  'data': { 'modela': 'CpuModelInfo', 'modelb': 'CpuModelInfo' },
-  'returns': 'CpuModelCompareInfo' }
-
 ##
 # @CpuModelBaselineInfo:
 #
@@ -2198,46 +2137,6 @@
 { 'struct': 'CpuModelBaselineInfo',
   'data': { 'model': 'CpuModelInfo' } }
 
-##
-# @query-cpu-model-baseline:
-#
-# Baseline two CPU models, creating a compatible third model. The created
-# model will always be a static, migration-safe CPU model (see "static"
-# CPU model expansion for details).
-#
-# This interface can be used by tooling to create a compatible CPU model out
-# two CPU models. The created CPU model will be identical to or a subset of
-# both CPU models when comparing them. Therefore, the created CPU model is
-# guaranteed to run where the given CPU models run.
-#
-# The result returned by this command may be affected by:
-#
-# * QEMU version: CPU models may look different depending on the QEMU version.
-#   (Except for CPU models reported as "static" in query-cpu-definitions.)
-# * machine-type: CPU model may look different depending on the machine-type.
-#   (Except for CPU models reported as "static" in query-cpu-definitions.)
-# * machine options (including accelerator): in some architectures, CPU models
-#   may look different depending on machine and accelerator options. (Except 
for
-#   CPU models reported as "static" in query-cpu-definitions.)
-# * "-cpu" arguments and global properties: arguments to the -cpu option and
-#   global properties may affect expansion of CPU models. Using
-#   query-cpu-model-expansion while using these is not advised.
-#
-# Some architectures may not support 

[Qemu-devel] [PATCH v4 34/51] qapi2texi: add 'If:' condition to enum values

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi2texi.py| 5 -
 tests/qapi-schema/doc-good.json | 4 +++-
 tests/qapi-schema/doc-good.out  | 1 +
 tests/qapi-schema/doc-good.texi | 2 ++
 4 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index 76c336af34..d1e69358f6 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -128,7 +128,10 @@ def texi_body(doc):
 
 def texi_enum_value(value):
 """Format a table of members item for an enumeration value"""
-return '@item @code{%s}\n' % value.name
+return '@item @code{%s}%s\n' % (
+value.name,
+'\n@b{If:} @code{%s}\n' %
+', '.join(value.ifcond) if value.ifcond else '')
 
 
 def texi_member(member, suffix=''):
diff --git a/tests/qapi-schema/doc-good.json b/tests/qapi-schema/doc-good.json
index 984cd8ed06..c7fe08c530 100644
--- a/tests/qapi-schema/doc-good.json
+++ b/tests/qapi-schema/doc-good.json
@@ -55,7 +55,9 @@
 #
 # @two is undocumented
 ##
-{ 'enum': 'Enum', 'data': [ 'one', 'two' ], 'if': 'defined(IFCOND)' }
+{ 'enum': 'Enum', 'data':
+  [ { 'name': 'one', 'if': 'defined(IFENUM)' }, 'two' ],
+  'if': 'defined(IFCOND)' }
 
 ##
 # @Base:
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 8c75da180c..74bf60bb83 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -2,6 +2,7 @@ object Base
 member base1: Enum optional=False
 enum Enum
 member one
+if ['defined(IFENUM)']
 member two
 if ['defined(IFCOND)']
 object Object
diff --git a/tests/qapi-schema/doc-good.texi b/tests/qapi-schema/doc-good.texi
index cd422f8a8f..c9988c1305 100644
--- a/tests/qapi-schema/doc-good.texi
+++ b/tests/qapi-schema/doc-good.texi
@@ -81,6 +81,8 @@ Examples:
 @b{Values:}
 @table @asis
 @item @code{one}
+@b{If:} @code{defined(IFENUM)}
+
 The @emph{one} @{and only@}
 @item @code{two}
 Not documented
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 09/51] qapi: mcgen() shouldn't indent # lines

[Marc-André Lureau]

Skip preprocessor lines when adding indentation, since that would
likely result in invalid code.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index ffb4fb58e0..3d33ed7d76 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1876,8 +1876,8 @@ def cgen(code, **kwds):
 if indent_level:
 indent = genindent(indent_level)
 # re.subn() lacks flags support before Python 2.7, use re.compile()
-raw = re.subn(re.compile(r'^.', re.MULTILINE),
-  indent + r'\g<0>', raw)
+raw = re.subn(re.compile(r'^(?!(#|$))', re.MULTILINE),
+  indent, raw)
 raw = raw[0]
 return re.sub(re.escape(eatspace) + r' *', '', raw)
 
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 11/51] qapi-introspect: modify to_qlit() to append ', ' on level > 0

[Marc-André Lureau]

The following patch is going to break list entries with #if/#endif, so
they should have the trailing ',' as suffix.

Signed-off-by: Marc-André Lureau 
Reviewed-by: Markus Armbruster 
---
 scripts/qapi-introspect.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 56c1f9d548..b1d08ec97b 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -29,7 +29,7 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
 for elt in obj]
 elts.append(indent(level + 1) + "{}")
 ret += 'QLIT_QLIST(((QLitObject[]) {\n'
-ret += ',\n'.join(elts) + '\n'
+ret += '\n'.join(elts) + '\n'
 ret += indent(level) + '}))'
 elif isinstance(obj, dict):
 elts = []
@@ -42,6 +42,8 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
 ret += indent(level) + '}))'
 else:
 assert False# not implemented
+if level > 0:
+ret += ','
 return ret
 
 
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 45/51] qapi: make rtc-reset-reinjection depend on TARGET_I386

[Marc-André Lureau]

Move rtc-reset-reinjection in target.json and make it conditional on
TARGET_I386.

Signed-off-by: Marc-André Lureau 
---
 qapi-schema.json   | 18 --
 qapi/target.json   | 19 +++
 hw/timer/mc146818rtc.c |  2 +-
 monitor.c  | 10 --
 4 files changed, 20 insertions(+), 29 deletions(-)

diff --git a/qapi-schema.json b/qapi-schema.json
index 5c06745c79..2cd13a07c4 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -2939,24 +2939,6 @@
 { 'event': 'ACPI_DEVICE_OST',
  'data': { 'info': 'ACPIOSTInfo' } }
 
-##
-# @rtc-reset-reinjection:
-#
-# This command will reset the RTC interrupt reinjection backlog.
-# Can be used if another mechanism to synchronize guest time
-# is in effect, for example QEMU guest agent's guest-set-time
-# command.
-#
-# Since: 2.1
-#
-# Example:
-#
-# -> { "execute": "rtc-reset-reinjection" }
-# <- { "return": {} }
-#
-##
-{ 'command': 'rtc-reset-reinjection' }
-
 ##
 # @RTC_CHANGE:
 #
diff --git a/qapi/target.json b/qapi/target.json
index 6cac484f68..9bfbb54276 100644
--- a/qapi/target.json
+++ b/qapi/target.json
@@ -3,3 +3,22 @@
 { 'include': '../qapi-schema.json' }
 
 { 'pragma': { 'unit': 'target' } }
+
+##
+# @rtc-reset-reinjection:
+#
+# This command will reset the RTC interrupt reinjection backlog.
+# Can be used if another mechanism to synchronize guest time
+# is in effect, for example QEMU guest agent's guest-set-time
+# command.
+#
+# Since: 2.1
+#
+# Example:
+#
+# -> { "execute": "rtc-reset-reinjection" }
+# <- { "return": {} }
+#
+##
+{ 'command': 'rtc-reset-reinjection',
+  'if': 'defined(TARGET_I386)' }
diff --git a/hw/timer/mc146818rtc.c b/hw/timer/mc146818rtc.c
index 35a05a64cc..49bcf21099 100644
--- a/hw/timer/mc146818rtc.c
+++ b/hw/timer/mc146818rtc.c
@@ -31,7 +31,7 @@
 #include "hw/timer/mc146818rtc.h"
 #include "qapi/visitor.h"
 #include "qapi-event.h"
-#include "qmp-commands.h"
+#include "target-qmp-commands.h"
 
 #ifdef TARGET_I386
 #include "hw/i386/apic.h"
diff --git a/monitor.c b/monitor.c
index 8a4fbdd09b..f78a3aee9e 100644
--- a/monitor.c
+++ b/monitor.c
@@ -971,9 +971,6 @@ static void qmp_query_qmp_schema(QDict *qdict, QObject 
**ret_data,
  */
 static void qmp_unregister_commands_hack(void)
 {
-#ifndef TARGET_I386
-qmp_unregister_command(_commands, "rtc-reset-reinjection");
-#endif
 #ifndef TARGET_S390X
 qmp_unregister_command(_commands, "dump-skeys");
 #endif
@@ -4145,13 +4142,6 @@ QemuOptsList qemu_mon_opts = {
 },
 };
 
-#ifndef TARGET_I386
-void qmp_rtc_reset_reinjection(Error **errp)
-{
-error_setg(errp, QERR_FEATURE_DISABLED, "rtc-reset-reinjection");
-}
-#endif
-
 #ifndef TARGET_S390X
 void qmp_dump_skeys(const char *filename, Error **errp)
 {
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 30/51] qapi: add 'if' to alternate members

[Marc-André Lureau]

Add 'if' key to alternate members:

{ 'alternate': 'TestIfAlternate', 'data':
  { 'alt': { 'type': 'TestStruct', 'if': 'COND' } } }

Generated code is not changed by this patch but with "qapi: add #if
conditions to generated code".

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 10 +-
 tests/qapi-schema/qapi-schema-test.json |  6 +-
 tests/qapi-schema/qapi-schema-test.out  |  9 -
 3 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 24d6aef13f..f24cf8bda6 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -840,7 +840,7 @@ def check_alternate(expr, info):
 for (key, value) in members.items():
 check_name(info, "Member of alternate '%s'" % name, key)
 source = "Member '%s' of alternate '%s'" % (key, name)
-check_known_keys(info, source, value, ['type'], [])
+check_known_keys(info, source, value, ['type'], ['if'])
 typ = value['type']
 
 # Ensure alternates have no type conflicts.
@@ -1719,8 +1719,8 @@ class QAPISchema(object):
   self._make_members(data, info),
   None))
 
-def _make_variant(self, case, typ):
-return QAPISchemaObjectTypeVariant(case, typ)
+def _make_variant(self, case, typ, ifcond):
+return QAPISchemaObjectTypeVariant(case, typ, ifcond)
 
 def _make_simple_variant(self, case, typ, ifcond, info):
 if isinstance(typ, list):
@@ -1743,7 +1743,7 @@ class QAPISchema(object):
 name, info, doc, ifcond,
 'base', self._make_members(base, info))
 if tag_name:
-variants = [self._make_variant(key, value['type'])
+variants = [self._make_variant(key, value['type'], value.get('if'))
 for (key, value) in data.iteritems()]
 members = []
 else:
@@ -1764,7 +1764,7 @@ class QAPISchema(object):
 name = expr['alternate']
 data = expr['data']
 ifcond = expr.get('if')
-variants = [self._make_variant(key, value['type'])
+variants = [self._make_variant(key, value['type'], value.get('if'))
 for (key, value) in data.iteritems()]
 tag_member = QAPISchemaObjectTypeMember('type', 'QType', False)
 self._def_entity(
diff --git a/tests/qapi-schema/qapi-schema-test.json 
b/tests/qapi-schema/qapi-schema-test.json
index b4b8a0a2cc..cd33c084cb 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -208,9 +208,13 @@
 { 'command': 'TestIfUnionCmd', 'data': { 'union_cmd_arg': 'TestIfUnion' },
   'if': 'defined(TEST_IF_UNION)' }
 
-{ 'alternate': 'TestIfAlternate', 'data': { 'foo': 'int', 'bar': 'TestStruct' 
},
+{ 'alternate': 'TestIfAlternate', 'data':
+  { 'foo': 'int', 'alt_bar': { 'type': 'TestStruct', 'if': 
'defined(TEST_IF_ALT_BAR)'} },
   'if': 'defined(TEST_IF_ALT) && defined(TEST_IF_STRUCT)' }
 
+{ 'command': 'TestIfAlternateCmd', 'data': { 'alt_cmd_arg': 'TestIfAlternate' 
},
+  'if': 'defined(TEST_IF_ALT)' }
+
 { 'command': 'TestIfCmd', 'data':
   { 'foo': 'TestIfStruct',
 'bar': { 'type': 'TestIfEnum', 'if': 'defined(TEST_IF_CMD_BAR)' } },
diff --git a/tests/qapi-schema/qapi-schema-test.out 
b/tests/qapi-schema/qapi-schema-test.out
index 001fa143cd..7215aeb4a6 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -67,8 +67,12 @@ enum QType
 alternate TestIfAlternate
 tag type
 case foo: int
-case bar: TestStruct
+case alt_bar: TestStruct
+if ['defined(TEST_IF_ALT_BAR)']
 if ['defined(TEST_IF_ALT) && defined(TEST_IF_STRUCT)']
+command TestIfAlternateCmd q_obj_TestIfAlternateCmd-arg -> None
+   gen=True success_response=True boxed=False
+if ['defined(TEST_IF_ALT)']
 command TestIfCmd q_obj_TestIfCmd-arg -> None
gen=True success_response=True boxed=False
 if ['defined(TEST_IF_CMD)', 'defined(TEST_IF_STRUCT)']
@@ -236,6 +240,9 @@ object q_obj_EVENT_D-arg
 member b: str optional=False
 member c: str optional=True
 member enum3: EnumOne optional=True
+object q_obj_TestIfAlternateCmd-arg
+member alt_cmd_arg: TestIfAlternate optional=False
+if ['defined(TEST_IF_ALT)']
 object q_obj_TestIfCmd-arg
 member foo: TestIfStruct optional=False
 member bar: TestIfEnum optional=False
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 27/51] qapi: add 'if' to implicit struct members

[Marc-André Lureau]

Generated code is not changed by this patch, but with "qapi: add #if
conditions to generated code" patch.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 14 +++---
 tests/qapi-schema/qapi-schema-test.json | 12 +---
 tests/qapi-schema/qapi-schema-test.out  |  5 +
 3 files changed, 21 insertions(+), 10 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 49fdf6bacf..c4a2623864 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -705,7 +705,7 @@ def check_type(info, source, value, allow_array=False,
 # Todo: allow dictionaries to represent default values of
 # an optional argument.
 member_source = "Member '%s' of %s" % (key, source)
-check_known_keys(info, member_source, arg, ['type'], [])
+check_known_keys(info, member_source, arg, ['type'], ['if'])
 check_type(info, member_source, arg['type'], allow_array=True,
allow_metas=['built-in', 'union', 'alternate', 'struct',
 'enum'])
@@ -1394,8 +1394,8 @@ class QAPISchemaMember(object):
 
 
 class QAPISchemaObjectTypeMember(QAPISchemaMember):
-def __init__(self, name, typ, optional):
-QAPISchemaMember.__init__(self, name)
+def __init__(self, name, typ, optional, ifcond=None):
+QAPISchemaMember.__init__(self, name, ifcond)
 assert isinstance(typ, str)
 assert isinstance(optional, bool)
 self._type_name = typ
@@ -1689,7 +1689,7 @@ class QAPISchema(object):
 name, info, doc, ifcond,
 self._make_enum_members(data), prefix))
 
-def _make_member(self, name, typ, info):
+def _make_member(self, name, typ, ifcond, info):
 optional = False
 if name.startswith('*'):
 name = name[1:]
@@ -1697,10 +1697,10 @@ class QAPISchema(object):
 if isinstance(typ, list):
 assert len(typ) == 1
 typ = self._make_array_type(typ[0], info)
-return QAPISchemaObjectTypeMember(name, typ, optional)
+return QAPISchemaObjectTypeMember(name, typ, optional, ifcond)
 
 def _make_members(self, data, info):
-return [self._make_member(key, value['type'], info)
+return [self._make_member(key, value['type'], value.get('if'), info)
 for (key, value) in data.iteritems()]
 
 def _def_struct_type(self, expr, info, doc):
@@ -1721,7 +1721,7 @@ class QAPISchema(object):
 typ = self._make_array_type(typ[0], info)
 typ = self._make_implicit_object_type(
 typ, info, None, self.lookup_type(typ),
-'wrapper', [self._make_member('data', typ, info)])
+'wrapper', [self._make_member('data', typ, None, info)])
 return QAPISchemaObjectTypeVariant(case, typ)
 
 def _def_union_type(self, expr, info, doc):
diff --git a/tests/qapi-schema/qapi-schema-test.json 
b/tests/qapi-schema/qapi-schema-test.json
index 71023fb277..e29c9f0769 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -191,7 +191,9 @@
 
 # test 'if' condition handling
 
-{ 'struct': 'TestIfStruct', 'data': { 'foo': 'int' },
+{ 'struct': 'TestIfStruct', 'data':
+  { 'foo': 'int',
+'bar': { 'type': 'int', 'if': 'defined(TEST_IF_STRUCT_BAR)'} },
   'if': 'defined(TEST_IF_STRUCT)' }
 
 { 'enum': 'TestIfEnum', 'data':
@@ -204,8 +206,12 @@
 { 'alternate': 'TestIfAlternate', 'data': { 'foo': 'int', 'bar': 'TestStruct' 
},
   'if': 'defined(TEST_IF_ALT) && defined(TEST_IF_STRUCT)' }
 
-{ 'command': 'TestIfCmd', 'data': { 'foo': 'TestIfStruct', 'bar': 'TestIfEnum' 
},
+{ 'command': 'TestIfCmd', 'data':
+  { 'foo': 'TestIfStruct',
+'bar': { 'type': 'TestIfEnum', 'if': 'defined(TEST_IF_CMD_BAR)' } },
   'if': ['defined(TEST_IF_CMD)', 'defined(TEST_IF_STRUCT)'] }
 
-{ 'event': 'TestIfEvent', 'data': { 'foo': 'TestIfStruct' },
+{ 'event': 'TestIfEvent', 'data':
+  { 'foo': 'TestIfStruct',
+'bar': { 'type': 'TestIfEnum', 'if': 'defined(TEST_IF_EVT_BAR)' } },
   'if': 'defined(TEST_IF_EVT) && defined(TEST_IF_STRUCT)' }
diff --git a/tests/qapi-schema/qapi-schema-test.out 
b/tests/qapi-schema/qapi-schema-test.out
index b5fff8d538..734d3f53aa 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -82,6 +82,8 @@ event TestIfEvent q_obj_TestIfEvent-arg
 if ['defined(TEST_IF_EVT) && defined(TEST_IF_STRUCT)']
 object TestIfStruct
 member foo: int optional=False
+member bar: int optional=False
+if ['defined(TEST_IF_STRUCT_BAR)']
 if ['defined(TEST_IF_STRUCT)']
 object TestIfUnion
 member type: TestIfUnionKind optional=False
@@ -230,9 +232,12 @@ object q_obj_EVENT_D-arg
 object q_obj_TestIfCmd-arg
 member foo: TestIfStruct optional=False
 member bar: TestIfEnum optional=False
+if ['defined(TEST_IF_CMD_BAR)']
 if ['defined(TEST_IF_CMD)', 'defined(TEST_IF_STRUCT)']
 object 

[Qemu-devel] [PATCH v4 08/51] qapi: add 'ifcond' to visitor methods

[Marc-André Lureau]

Modify the test visitor to check correct passing of values.

Signed-off-by: Marc-André Lureau 
Reviewed-by: Markus Armbruster 
---
 scripts/qapi.py| 31 +--
 scripts/qapi-commands.py   |  2 +-
 scripts/qapi-event.py  |  2 +-
 scripts/qapi-introspect.py | 12 ++--
 scripts/qapi-types.py  |  8 
 scripts/qapi-visit.py  |  8 
 scripts/qapi2texi.py   | 10 +-
 tests/qapi-schema/qapi-schema-test.out |  9 +
 tests/qapi-schema/test-qapi.py | 21 -
 9 files changed, 63 insertions(+), 40 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 4d2c214f19..ffb4fb58e0 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1046,26 +1046,26 @@ class QAPISchemaVisitor(object):
 def visit_builtin_type(self, name, info, json_type):
 pass
 
-def visit_enum_type(self, name, info, values, prefix):
+def visit_enum_type(self, name, info, ifcond, values, prefix):
 pass
 
-def visit_array_type(self, name, info, element_type):
+def visit_array_type(self, name, info, ifcond, element_type):
 pass
 
-def visit_object_type(self, name, info, base, members, variants):
+def visit_object_type(self, name, info, ifcond, base, members, variants):
 pass
 
-def visit_object_type_flat(self, name, info, members, variants):
+def visit_object_type_flat(self, name, info, ifcond, members, variants):
 pass
 
-def visit_alternate_type(self, name, info, variants):
+def visit_alternate_type(self, name, info, ifcond, variants):
 pass
 
-def visit_command(self, name, info, arg_type, ret_type,
+def visit_command(self, name, info, ifcond, arg_type, ret_type,
   gen, success_response, boxed):
 pass
 
-def visit_event(self, name, info, arg_type, boxed):
+def visit_event(self, name, info, ifcond, arg_type, boxed):
 pass
 
 
@@ -1165,7 +1165,7 @@ class QAPISchemaEnumType(QAPISchemaType):
 return 'string'
 
 def visit(self, visitor):
-visitor.visit_enum_type(self.name, self.info,
+visitor.visit_enum_type(self.name, self.info, self.ifcond,
 self.member_names(), self.prefix)
 
 
@@ -1199,7 +1199,8 @@ class QAPISchemaArrayType(QAPISchemaType):
 return 'array of ' + elt_doc_type
 
 def visit(self, visitor):
-visitor.visit_array_type(self.name, self.info, self.element_type)
+visitor.visit_array_type(self.name, self.info, self.ifcond,
+ self.element_type)
 
 
 class QAPISchemaObjectType(QAPISchemaType):
@@ -1281,9 +1282,9 @@ class QAPISchemaObjectType(QAPISchemaType):
 return 'object'
 
 def visit(self, visitor):
-visitor.visit_object_type(self.name, self.info,
+visitor.visit_object_type(self.name, self.info, self.ifcond,
   self.base, self.local_members, self.variants)
-visitor.visit_object_type_flat(self.name, self.info,
+visitor.visit_object_type_flat(self.name, self.info, self.ifcond,
self.members, self.variants)
 
 
@@ -1428,7 +1429,8 @@ class QAPISchemaAlternateType(QAPISchemaType):
 return 'value'
 
 def visit(self, visitor):
-visitor.visit_alternate_type(self.name, self.info, self.variants)
+visitor.visit_alternate_type(self.name, self.info, self.ifcond,
+ self.variants)
 
 def is_empty(self):
 return False
@@ -1469,7 +1471,7 @@ class QAPISchemaCommand(QAPISchemaEntity):
 assert isinstance(self.ret_type, QAPISchemaType)
 
 def visit(self, visitor):
-visitor.visit_command(self.name, self.info,
+visitor.visit_command(self.name, self.info, self.ifcond,
   self.arg_type, self.ret_type,
   self.gen, self.success_response, self.boxed)
 
@@ -1500,7 +1502,8 @@ class QAPISchemaEvent(QAPISchemaEntity):
 raise QAPISemError(self.info, "Use of 'boxed' requires 'data'")
 
 def visit(self, visitor):
-visitor.visit_event(self.name, self.info, self.arg_type, self.boxed)
+visitor.visit_event(self.name, self.info, self.ifcond,
+self.arg_type, self.boxed)
 
 
 class QAPISchema(object):
diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 974d0a4a80..669aef1eb7 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -240,7 +240,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 self._regy = None
 self._visited_ret_types = None
 
-def visit_command(self, name, info, arg_type, ret_type,
+def visit_command(self, name, info, ifcond, arg_type, ret_type,
 

[Qemu-devel] [PATCH v4 25/51] qapi: rename allow_dict to allow_implicit

[Marc-André Lureau]

This makes it a bit clearer what is the intent of the dictionnary for
the check_type() function, since there was some confusion on a
previous iteration of this series.

Suggested-by: Markus Armbruster 
Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 12 ++--
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index e203734254..5b41114949 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -652,7 +652,7 @@ def check_if(expr, info):
 
 
 def check_type(info, source, value, allow_array=False,
-   allow_dict=False, allow_optional=False,
+   allow_implicit=False, allow_optional=False,
allow_metas=[]):
 global all_names
 
@@ -679,7 +679,7 @@ def check_type(info, source, value, allow_array=False,
(source, all_names[value], value))
 return
 
-if not allow_dict:
+if not allow_implicit:
 raise QAPISemError(info, "%s should be a type name" % source)
 
 if not isinstance(value, OrderedDict):
@@ -709,7 +709,7 @@ def check_command(expr, info):
 if boxed:
 args_meta += ['union', 'alternate']
 check_type(info, "'data' for command '%s'" % name,
-   expr.get('data'), allow_dict=not boxed, allow_optional=True,
+   expr.get('data'), allow_implicit=not boxed, allow_optional=True,
allow_metas=args_meta)
 returns_meta = ['union', 'struct']
 if name in returns_whitelist:
@@ -727,7 +727,7 @@ def check_event(expr, info):
 if boxed:
 meta += ['union', 'alternate']
 check_type(info, "'data' for event '%s'" % name,
-   expr.get('data'), allow_dict=not boxed, allow_optional=True,
+   expr.get('data'), allow_implicit=not boxed, allow_optional=True,
allow_metas=meta)
 
 
@@ -755,7 +755,7 @@ def check_union(expr, info):
 else:
 # The object must have a string or dictionary 'base'.
 check_type(info, "'base' for union '%s'" % name,
-   base, allow_dict=True, allow_optional=True,
+   base, allow_implicit=True, allow_optional=True,
allow_metas=['struct'])
 if not base:
 raise QAPISemError(info, "Flat union '%s' must have a base"
@@ -886,7 +886,7 @@ def check_struct(expr, info):
 members = expr['data']
 
 check_type(info, "'data' for struct '%s'" % name, members,
-   allow_dict=True, allow_optional=True)
+   allow_implicit=True, allow_optional=True)
 check_type(info, "'base' for struct '%s'" % name, expr.get('base'),
allow_metas=['struct'])
 
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 26/51] qapi: add a dictionary form with 'type' key for members

[Marc-André Lureau]

Wherever a struct/union/alternate/command/event member with NAME: TYPE
form is accepted, desugar it to a NAME: { 'type': TYPE } form.

This will allow to add new member details, such as 'if' in the
following patch to introduce conditionals, or 'default' for default
values etc.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py   | 57 +++
 tests/Makefile.include|  3 ++
 tests/qapi-schema/alternate-invalid-dict.err  |  1 +
 tests/qapi-schema/alternate-invalid-dict.exit |  1 +
 tests/qapi-schema/alternate-invalid-dict.json |  4 ++
 tests/qapi-schema/alternate-invalid-dict.out  |  0
 tests/qapi-schema/event-nest-struct.err   |  2 +-
 tests/qapi-schema/flat-union-inline.err   |  2 +-
 tests/qapi-schema/nested-struct-data.err  |  2 +-
 tests/qapi-schema/qapi-schema-test.json   | 10 ++--
 tests/qapi-schema/struct-member-invalid-dict.err  |  1 +
 tests/qapi-schema/struct-member-invalid-dict.exit |  1 +
 tests/qapi-schema/struct-member-invalid-dict.json |  3 ++
 tests/qapi-schema/struct-member-invalid-dict.out  |  0
 tests/qapi-schema/union-branch-invalid-dict.err   |  1 +
 tests/qapi-schema/union-branch-invalid-dict.exit  |  1 +
 tests/qapi-schema/union-branch-invalid-dict.json  |  4 ++
 tests/qapi-schema/union-branch-invalid-dict.out   |  0
 18 files changed, 66 insertions(+), 27 deletions(-)
 create mode 100644 tests/qapi-schema/alternate-invalid-dict.err
 create mode 100644 tests/qapi-schema/alternate-invalid-dict.exit
 create mode 100644 tests/qapi-schema/alternate-invalid-dict.json
 create mode 100644 tests/qapi-schema/alternate-invalid-dict.out
 create mode 100644 tests/qapi-schema/struct-member-invalid-dict.err
 create mode 100644 tests/qapi-schema/struct-member-invalid-dict.exit
 create mode 100644 tests/qapi-schema/struct-member-invalid-dict.json
 create mode 100644 tests/qapi-schema/struct-member-invalid-dict.out
 create mode 100644 tests/qapi-schema/union-branch-invalid-dict.err
 create mode 100644 tests/qapi-schema/union-branch-invalid-dict.exit
 create mode 100644 tests/qapi-schema/union-branch-invalid-dict.json
 create mode 100644 tests/qapi-schema/union-branch-invalid-dict.out

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 5b41114949..49fdf6bacf 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -579,11 +579,11 @@ def discriminator_find_enum_define(expr):
 if not base_members:
 return None
 
-discriminator_type = base_members.get(discriminator)
-if not discriminator_type:
+discriminator_member = base_members.get(discriminator)
+if not discriminator_member:
 return None
 
-return enum_types.get(discriminator_type)
+return enum_types.get(discriminator_member['type'])
 
 
 # Names must be letters, numbers, -, and _.  They must start with letter,
@@ -651,6 +651,15 @@ def check_if(expr, info):
 check_if_str(ifcond, info)
 
 
+def normalize_members(expr, field):
+members = expr.get(field)
+if isinstance(members, OrderedDict):
+for key, arg in members.items():
+if isinstance(arg, dict):
+continue
+members[key] = {'type': arg}
+
+
 def check_type(info, source, value, allow_array=False,
allow_implicit=False, allow_optional=False,
allow_metas=[]):
@@ -695,8 +704,9 @@ def check_type(info, source, value, allow_array=False,
% (source, key))
 # Todo: allow dictionaries to represent default values of
 # an optional argument.
-check_type(info, "Member '%s' of %s" % (key, source), arg,
-   allow_array=True,
+member_source = "Member '%s' of %s" % (key, source)
+check_known_keys(info, member_source, arg, ['type'], [])
+check_type(info, member_source, arg['type'], allow_array=True,
allow_metas=['built-in', 'union', 'alternate', 'struct',
 'enum'])
 
@@ -767,13 +777,13 @@ def check_union(expr, info):
 # member of the base struct.
 check_name(info, "Discriminator of flat union '%s'" % name,
discriminator)
-discriminator_type = base_members.get(discriminator)
-if not discriminator_type:
+discriminator_member = base_members.get(discriminator)
+if not discriminator_member:
 raise QAPISemError(info,
"Discriminator '%s' is not a member of base "
"struct '%s'"
% (discriminator, base))
-enum_define = enum_types.get(discriminator_type)
+enum_define = enum_types.get(discriminator_member['type'])
 allow_metas = ['struct']
 # Do not allow string discriminator
 if not enum_define:
@@ -785,11 +795,14 @@ def check_union(expr, info):
 if len(members) == 0:
 

[Qemu-devel] [PATCH v4 41/51] qapi: add -i/--include filename.h

[Marc-André Lureau]

Add a new option to add user-specified #include lines in the generated
headers. This will help to split a schema, where one generated header
will depend on another.

Fix some pycodestyle on the way.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py| 15 ++-
 scripts/qapi-commands.py   |  7 +--
 scripts/qapi-event.py  |  6 --
 scripts/qapi-introspect.py |  6 --
 scripts/qapi-types.py  |  7 +--
 scripts/qapi-visit.py  |  6 --
 tests/Makefile.include |  4 +++-
 7 files changed, 35 insertions(+), 16 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 1668a6da6c..f56460d028 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -2125,10 +2125,10 @@ def build_params(arg_type, boxed, extra):
 def parse_command_line(extra_options='', extra_long_options=[]):
 
 try:
-opts, args = getopt.gnu_getopt(sys.argv[1:],
-   'chp:o:' + extra_options,
-   ['source', 'header', 'prefix=',
-'output-dir='] + extra_long_options)
+opts, args = getopt.gnu_getopt(
+sys.argv[1:], 'chp:o:i:' + extra_options,
+['source', 'header', 'prefix=', 'output-dir=',
+ 'include='] + extra_long_options)
 except getopt.GetoptError as err:
 print >>sys.stderr, "%s: %s" % (sys.argv[0], str(err))
 sys.exit(1)
@@ -2137,6 +2137,7 @@ def parse_command_line(extra_options='', 
extra_long_options=[]):
 prefix = ''
 do_c = False
 do_h = False
+includes = []
 extra_opts = []
 
 for oa in opts:
@@ -2155,6 +2156,8 @@ def parse_command_line(extra_options='', 
extra_long_options=[]):
 do_c = True
 elif o in ('-h', '--header'):
 do_h = True
+elif o in ('-i', '--include'):
+includes.append(a)
 else:
 extra_opts.append(oa)
 
@@ -2167,7 +2170,9 @@ def parse_command_line(extra_options='', 
extra_long_options=[]):
 sys.exit(1)
 fname = args[0]
 
-return (fname, output_dir, do_c, do_h, prefix, extra_opts)
+includes = "\n".join(['#include "%s"' % inc for inc in includes])
+
+return (fname, output_dir, do_c, do_h, prefix, includes, extra_opts)
 
 #
 # Generate output files with boilerplate
diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 7455d2b8bb..4841f4d9a1 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -253,7 +253,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 self._regy += gen_register_command(name, success_response)
 
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
+(input_file, output_dir, do_c, do_h, prefix, includes, opts) = \
+parse_command_line()
 
 c_comment = '''
 /*
@@ -305,6 +306,7 @@ fdef.write(mcgen('''
  prefix=prefix))
 
 fdecl.write(mcgen('''
+%(includes)s
 #include "%(prefix)sqapi-types.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/dispatch.h"
@@ -312,7 +314,8 @@ fdecl.write(mcgen('''
 
 void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
 ''',
-  prefix=prefix, c_prefix=c_name(prefix, protect=False)))
+  includes=includes, prefix=prefix,
+  c_prefix=c_name(prefix, protect=False)))
 
 schema = QAPISchema(input_file)
 gen = QAPISchemaGenCommandVisitor()
diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 60c6f7030d..0aba866dc8 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -171,7 +171,8 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 self._event_names.append(QAPISchemaMember(name, ifcond))
 
 
-(input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
+(input_file, output_dir, do_c, do_h, prefix, includes, dummy) = \
+parse_command_line()
 
 c_comment = '''
 /*
@@ -218,13 +219,14 @@ fdef.write(mcgen('''
  prefix=prefix))
 
 fdecl.write(mcgen('''
+%(includes)s
 #include "qapi/error.h"
 #include "qapi/util.h"
 #include "qapi/qmp/qdict.h"
 #include "%(prefix)sqapi-types.h"
 
 ''',
-  prefix=prefix))
+  includes=includes, prefix=prefix))
 
 event_enum_name = c_name(prefix + 'QAPIEvent', protect=False)
 
diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 6a66047243..65277e98fd 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -192,11 +192,12 @@ const QLitObject %(c_name)s = %(c_string)s;
 self._gen_qlit(name, 'event', {'arg-type': self._use_type(arg_type)},
ifcond)
 
+
 # Debugging aid: unmask QAPI schema's type names
 # We normally mask them, because they're not QMP wire ABI
 opt_unmask = False
 
-(input_file, output_dir, do_c, do_h, prefix, opts) = \
+(input_file, output_dir, do_c, do_h, prefix, includes, opts) = \
 parse_command_line('u', ['unmask-non-abi-names'])
 
 

[Qemu-devel] [PATCH v4 06/51] qapi: pass 'if' condition into QAPISchemaEntity objects

[Marc-André Lureau]

Built-in objects remain unconditional.  Explicitly defined objects
use the condition specified in the schema.  Implicitly defined
objects inherit their condition from their users.  For most of them,
there is exactly one user, so the condition to use is obvious.  The
exception is the wrapped type's generated for simple union variants,
which can be shared by any number of simple unions.  The tight
condition would be the disjunction of the conditions of these simple
unions.  For now, use wrapped type's condition instead.  Much
simpler and good enough for now.

Signed-off-by: Marc-André Lureau 
Reviewed-by: Markus Armbruster 
---
 scripts/qapi.py | 98 ++---
 1 file changed, 66 insertions(+), 32 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 27df0fcf48..8f54dead8d 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -991,8 +991,17 @@ def check_exprs(exprs):
 # Schema compiler frontend
 #
 
+def listify_cond(ifcond):
+if not ifcond:
+return []
+elif not isinstance(ifcond, list):
+return [ifcond]
+else:
+return ifcond
+
+
 class QAPISchemaEntity(object):
-def __init__(self, name, info, doc):
+def __init__(self, name, info, doc, ifcond=None):
 assert isinstance(name, str)
 self.name = name
 # For explicitly defined entities, info points to the (explicit)
@@ -1002,6 +1011,7 @@ class QAPISchemaEntity(object):
 # such place).
 self.info = info
 self.doc = doc
+self.ifcond = listify_cond(ifcond)
 
 def c_name(self):
 return c_name(self.name)
@@ -1118,8 +1128,8 @@ class QAPISchemaBuiltinType(QAPISchemaType):
 
 
 class QAPISchemaEnumType(QAPISchemaType):
-def __init__(self, name, info, doc, values, prefix):
-QAPISchemaType.__init__(self, name, info, doc)
+def __init__(self, name, info, doc, ifcond, values, prefix):
+QAPISchemaType.__init__(self, name, info, doc, ifcond)
 for v in values:
 assert isinstance(v, QAPISchemaMember)
 v.set_owner(name)
@@ -1154,7 +1164,7 @@ class QAPISchemaEnumType(QAPISchemaType):
 
 class QAPISchemaArrayType(QAPISchemaType):
 def __init__(self, name, info, element_type):
-QAPISchemaType.__init__(self, name, info, None)
+QAPISchemaType.__init__(self, name, info, None, None)
 assert isinstance(element_type, str)
 self._element_type_name = element_type
 self.element_type = None
@@ -1162,6 +1172,7 @@ class QAPISchemaArrayType(QAPISchemaType):
 def check(self, schema):
 self.element_type = schema.lookup_type(self._element_type_name)
 assert self.element_type
+self.ifcond = self.element_type.ifcond
 
 def is_implicit(self):
 return True
@@ -1183,11 +1194,12 @@ class QAPISchemaArrayType(QAPISchemaType):
 
 
 class QAPISchemaObjectType(QAPISchemaType):
-def __init__(self, name, info, doc, base, local_members, variants):
+def __init__(self, name, info, doc, ifcond,
+ base, local_members, variants):
 # struct has local_members, optional base, and no variants
 # flat union has base, variants, and no local_members
 # simple union has local_members, variants, and no base
-QAPISchemaType.__init__(self, name, info, doc)
+QAPISchemaType.__init__(self, name, info, doc, ifcond)
 assert base is None or isinstance(base, str)
 for m in local_members:
 assert isinstance(m, QAPISchemaObjectTypeMember)
@@ -1375,8 +1387,8 @@ class 
QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
 
 
 class QAPISchemaAlternateType(QAPISchemaType):
-def __init__(self, name, info, doc, variants):
-QAPISchemaType.__init__(self, name, info, doc)
+def __init__(self, name, info, doc, ifcond, variants):
+QAPISchemaType.__init__(self, name, info, doc, ifcond)
 assert isinstance(variants, QAPISchemaObjectTypeVariants)
 assert variants.tag_member
 variants.set_owner(name)
@@ -1412,9 +1424,9 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
 
 class QAPISchemaCommand(QAPISchemaEntity):
-def __init__(self, name, info, doc, arg_type, ret_type,
+def __init__(self, name, info, doc, ifcond, arg_type, ret_type,
  gen, success_response, boxed):
-QAPISchemaEntity.__init__(self, name, info, doc)
+QAPISchemaEntity.__init__(self, name, info, doc, ifcond)
 assert not arg_type or isinstance(arg_type, str)
 assert not ret_type or isinstance(ret_type, str)
 self._arg_type_name = arg_type
@@ -1451,8 +1463,8 @@ class QAPISchemaCommand(QAPISchemaEntity):
 
 
 class QAPISchemaEvent(QAPISchemaEntity):
-def __init__(self, name, info, doc, arg_type, boxed):
-QAPISchemaEntity.__init__(self, name, info, doc)
+def __init__(self, name, info, doc, ifcond, arg_type, boxed):
+   

[Qemu-devel] [PATCH v4 21/51] qapi: factor out check_known_keys()

[Marc-André Lureau]

The following patches are going to need similar checks from various
code path. This refactoring will report all conflicting keys (instead
of the first one encountered).

Modify unknown-expr-key to check plural form.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 27 ---
 tests/qapi-schema/alternate-base.err|  2 +-
 tests/qapi-schema/double-type.err   |  2 +-
 tests/qapi-schema/enum-missing-data.err |  2 +-
 tests/qapi-schema/unknown-expr-key.err  |  2 +-
 tests/qapi-schema/unknown-expr-key.json |  2 +-
 6 files changed, 25 insertions(+), 12 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index f8c711b81b..a9dcfc064c 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -871,6 +871,24 @@ def check_struct(expr, info):
allow_metas=['struct'])
 
 
+def check_known_keys(info, source, keys, required, optional):
+
+def pprint(elems):
+return ', '.join("'" + e + "'" for e in elems)
+
+missing = set(required) - set(keys)
+if missing:
+raise QAPISemError(info, "%s must have %s key%s"
+   % (source, pprint(missing),
+  's' if len(missing) > 1 else ''))
+allowed = set(required + optional)
+unknown = set(keys) - allowed
+if unknown:
+raise QAPISemError(info, "%s has unknown key%s %s (allowed: %s)"
+   % (source, 's' if len(unknown) > 1 else '',
+  pprint(unknown), pprint(allowed)))
+
+
 def check_keys(expr_elem, meta, required, optional=[]):
 expr = expr_elem['expr']
 info = expr_elem['info']
@@ -878,10 +896,9 @@ def check_keys(expr_elem, meta, required, optional=[]):
 if not isinstance(name, str):
 raise QAPISemError(info, "'%s' key must have a string value" % meta)
 required = required + [meta]
+source = "%s '%s'" % (meta, name)
+check_known_keys(info, source, expr, required, optional)
 for (key, value) in expr.items():
-if key not in required and key not in optional:
-raise QAPISemError(info, "Unknown key '%s' in %s '%s'"
-   % (key, meta, name))
 if (key == 'gen' or key == 'success-response') and value is not False:
 raise QAPISemError(info,
"'%s' of %s '%s' should only use false value"
@@ -892,10 +909,6 @@ def check_keys(expr_elem, meta, required, optional=[]):
% (key, meta, name))
 if key == 'if':
 check_if(expr, info)
-for key in required:
-if key not in expr:
-raise QAPISemError(info, "Key '%s' is missing from %s '%s'"
-   % (key, meta, name))
 
 
 def check_exprs(exprs):
diff --git a/tests/qapi-schema/alternate-base.err 
b/tests/qapi-schema/alternate-base.err
index 30d8a34373..2b09c4c7a3 100644
--- a/tests/qapi-schema/alternate-base.err
+++ b/tests/qapi-schema/alternate-base.err
@@ -1 +1 @@
-tests/qapi-schema/alternate-base.json:4: Unknown key 'base' in alternate 'Alt'
+tests/qapi-schema/alternate-base.json:4: alternate 'Alt' has unknown key 
'base' (allowed: 'alternate', 'data', 'if')
diff --git a/tests/qapi-schema/double-type.err 
b/tests/qapi-schema/double-type.err
index f9613c6d6b..968fac6b66 100644
--- a/tests/qapi-schema/double-type.err
+++ b/tests/qapi-schema/double-type.err
@@ -1 +1 @@
-tests/qapi-schema/double-type.json:2: Unknown key 'command' in struct 'bar'
+tests/qapi-schema/double-type.json:2: struct 'bar' has unknown key 'command' 
(allowed: 'base', 'data', 'struct', 'if')
diff --git a/tests/qapi-schema/enum-missing-data.err 
b/tests/qapi-schema/enum-missing-data.err
index ba4873ae69..68e286badc 100644
--- a/tests/qapi-schema/enum-missing-data.err
+++ b/tests/qapi-schema/enum-missing-data.err
@@ -1 +1 @@
-tests/qapi-schema/enum-missing-data.json:2: Key 'data' is missing from enum 
'MyEnum'
+tests/qapi-schema/enum-missing-data.json:2: enum 'MyEnum' must have 'data' key
diff --git a/tests/qapi-schema/unknown-expr-key.err 
b/tests/qapi-schema/unknown-expr-key.err
index 12f5ed5b43..6a28af8ffc 100644
--- a/tests/qapi-schema/unknown-expr-key.err
+++ b/tests/qapi-schema/unknown-expr-key.err
@@ -1 +1 @@
-tests/qapi-schema/unknown-expr-key.json:2: Unknown key 'bogus' in struct 'bar'
+tests/qapi-schema/unknown-expr-key.json:2: struct 'bar' has unknown keys 
'foo', 'bogus' (allowed: 'base', 'data', 'struct', 'if')
diff --git a/tests/qapi-schema/unknown-expr-key.json 
b/tests/qapi-schema/unknown-expr-key.json
index 3b2be00cc4..5bcb8efd1d 100644
--- a/tests/qapi-schema/unknown-expr-key.json
+++ b/tests/qapi-schema/unknown-expr-key.json
@@ -1,2 +1,2 @@
 # we reject an expression with unknown top-level keys
-{ 'struct': 'bar', 'data': { 'string': 'str'}, 'bogus': { } }
+{ 'struct': 'bar', 'data': { 'string': 'str'}, 'bogus': { }, 'foo': { } }
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 24/51] qapi-event: add 'if' condition to implicit event enum

[Marc-André Lureau]

Add condition to QAPIEvent enum members based on the event 'if'.

Generated code is not changed by this patch, but with "qapi: add #if
conditions to generated code" patch.

There is no coverage of this change in qapi-schema-test.out since the
event_names enum is an implicit type created by qapi-event.py.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi-event.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi-event.py b/scripts/qapi-event.py
index 38f4264817..60c6f7030d 100644
--- a/scripts/qapi-event.py
+++ b/scripts/qapi-event.py
@@ -168,7 +168,7 @@ class QAPISchemaGenEventVisitor(QAPISchemaVisitor):
 def visit_event(self, name, info, ifcond, arg_type, boxed):
 self.decl += gen_event_send_decl(name, arg_type, boxed)
 self.defn += gen_event_send(name, arg_type, boxed)
-self._event_names.append(QAPISchemaMember(name))
+self._event_names.append(QAPISchemaMember(name, ifcond))
 
 
 (input_file, output_dir, do_c, do_h, prefix, dummy) = parse_command_line()
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 04/51] qapi2texi: minor python code simplification

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
Reviewed-by: Markus Armbruster 
---
 scripts/qapi2texi.py | 7 +++
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index 92e2af2cd6..0a08fbaf7b 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -134,10 +134,9 @@ def texi_enum_value(value):
 def texi_member(member, suffix=''):
 """Format a table of members item for an object type member"""
 typ = member.type.doc_type()
-return '@item @code{%s%s%s}%s%s\n' % (
-member.name,
-': ' if typ else '',
-typ if typ else '',
+membertype = ': ' + typ if typ else ''
+return '@item @code{%s%s}%s%s\n' % (
+member.name, membertype,
 ' (optional)' if member.optional else '',
 suffix)
 
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 32/51] docs: document schema configuration

[Marc-André Lureau]

Signed-off-by: Marc-André Lureau 
---
 docs/devel/qapi-code-gen.txt | 38 ++
 1 file changed, 38 insertions(+)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index f58d62686a..479b755609 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -682,6 +682,44 @@ Example: Red Hat, Inc. controls redhat.com, and may 
therefore add a
 downstream command __com.redhat_drive-mirror.
 
 
+=== Configuring the schema ===
+
+Top-level QAPI expressions and various type expressions listed below
+can take an 'if' key.  The value must be a string or a list of
+string.  The corresponding generated code will then guard the inclusion
+of that member in the larger struct or function with #if IFCOND
+(or several #if lines for a list), where IFCOND is the value of the
+'if' key.
+
+'struct', 'enum', 'union', 'alternate', 'command' and 'event'
+top-level QAPI expressions can take an 'if' keyword like:
+
+{ 'struct': 'IfStruct', 'data': { 'foo': 'int' },
+  'if': 'defined(IFCOND)' }
+
+Where a member can normally be defined with a single string value as its
+type, it is also possible to supply a dictionary with both 'type' and
+'if' keys.
+
+{ 'struct': 'IfStruct', 'data':
+  { 'foo': 'int',
+'bar': { 'type': 'int', 'if': 'defined(IFCOND)'} } }
+
+An enum value can be replaced by a dictionary with a 'name' and a 'if'
+key:
+
+{ 'enum': 'IfEnum', 'data':
+  [ 'foo',
+{ 'name' : 'bar', 'if': 'defined(IFCOND)' } ] }
+
+Please note that you are responsible to ensure that the C code will
+compile with an arbitrary combination of conditions, since the
+generators are unable to check it at this point.
+
+The presence of 'if' keys in the schema is reflected through to the
+introspection output depending on the build configuration.
+
+
 == Client JSON Protocol introspection ==
 
 Clients of a Client JSON Protocol commonly need to figure out what
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 17/51] qapi: do not define enumeration value explicitely

[Marc-André Lureau]

The C standard has the initial value at 0 and the subsequent values
incremented by 1. No need to set this explicitely.

This will prevent from artificial "gaps" when compiling out some enum
values and having unnecessarily large MAX values & enums arrays.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 7 ++-
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 71f28fc6d8..ee759489cb 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1991,14 +1991,11 @@ typedef enum %(c_name)s {
 ''',
 c_name=c_name(name))
 
-i = 0
 for value in enum_values:
 ret += mcgen('''
-%(c_enum)s = %(i)d,
+%(c_enum)s,
 ''',
- c_enum=c_enum_const(name, value, prefix),
- i=i)
-i += 1
+ c_enum=c_enum_const(name, value, prefix))
 
 ret += mcgen('''
 } %(c_name)s;
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 22/51] qapi: add a dictionnary form with 'name' key for enum members

[Marc-André Lureau]

Desugar the enum NAME form to { 'name': NAME }. This will allow to add
new enum members, such as 'if' in the following patch.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py| 49 ++
 tests/Makefile.include |  3 +-
 tests/qapi-schema/enum-bad-member.err  |  1 +
 ...{enum-dict-member.exit => enum-bad-member.exit} |  0
 tests/qapi-schema/enum-bad-member.json |  2 +
 .../{enum-dict-member.out => enum-bad-member.out}  |  0
 tests/qapi-schema/enum-dict-member-unknown.err |  1 +
 tests/qapi-schema/enum-dict-member-unknown.exit|  1 +
 tests/qapi-schema/enum-dict-member-unknown.json|  2 +
 tests/qapi-schema/enum-dict-member-unknown.out |  0
 tests/qapi-schema/enum-dict-member.err |  1 -
 tests/qapi-schema/enum-dict-member.json|  2 -
 tests/qapi-schema/enum-missing-data.err|  2 +-
 13 files changed, 51 insertions(+), 13 deletions(-)
 create mode 100644 tests/qapi-schema/enum-bad-member.err
 rename tests/qapi-schema/{enum-dict-member.exit => enum-bad-member.exit} (100%)
 create mode 100644 tests/qapi-schema/enum-bad-member.json
 rename tests/qapi-schema/{enum-dict-member.out => enum-bad-member.out} (100%)
 create mode 100644 tests/qapi-schema/enum-dict-member-unknown.err
 create mode 100644 tests/qapi-schema/enum-dict-member-unknown.exit
 create mode 100644 tests/qapi-schema/enum-dict-member-unknown.json
 create mode 100644 tests/qapi-schema/enum-dict-member-unknown.out
 delete mode 100644 tests/qapi-schema/enum-dict-member.err
 delete mode 100644 tests/qapi-schema/enum-dict-member.json

diff --git a/scripts/qapi.py b/scripts/qapi.py
index a9dcfc064c..1338be4efe 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -731,6 +731,10 @@ def check_event(expr, info):
allow_metas=meta)
 
 
+def enum_get_names(expr):
+return [e['name'] for e in expr['data']]
+
+
 def check_union(expr, info):
 name = expr['union']
 base = expr.get('base')
@@ -790,7 +794,7 @@ def check_union(expr, info):
 # If the discriminator names an enum type, then all members
 # of 'data' must also be members of the enum type.
 if enum_define:
-if key not in enum_define['data']:
+if key not in enum_get_names(enum_define):
 raise QAPISemError(info,
"Discriminator value '%s' is not found in "
"enum '%s'"
@@ -798,7 +802,7 @@ def check_union(expr, info):
 
 # If discriminator is user-defined, ensure all values are covered
 if enum_define:
-for value in enum_define['data']:
+for value in enum_get_names(enum_define):
 if value not in members.keys():
 raise QAPISemError(info, "Union '%s' data missing '%s' branch"
% (name, value))
@@ -829,10 +833,10 @@ def check_alternate(expr, info):
 if qtype == 'QTYPE_QSTRING':
 enum_expr = enum_types.get(value)
 if enum_expr:
-for v in enum_expr['data']:
+for v in enum_get_names(enum_expr):
 if v in ['on', 'off']:
 conflicting.add('QTYPE_QBOOL')
-if re.match(r'[-+0-9.]', v): # lazy, could be tightened
+if re.match(r'[-+0-9.]', v):  # lazy, could be tightened
 conflicting.add('QTYPE_QNUM')
 else:
 conflicting.add('QTYPE_QNUM')
@@ -845,19 +849,34 @@ def check_alternate(expr, info):
 types_seen[qt] = key
 
 
-def check_enum(expr, info):
+def normalize_enum(expr, info):
 name = expr['enum']
 members = expr.get('data')
-prefix = expr.get('prefix')
 
 if not isinstance(members, list):
 raise QAPISemError(info,
"Enum '%s' requires an array for 'data'" % name)
+
+# translate short member form to dict form
+for i, member in enumerate(members):
+if not isinstance(member, dict):
+member = {'name': member}
+members[i] = member
+
+
+def check_enum(expr, info):
+name = expr['enum']
+members = expr.get('data')
+prefix = expr.get('prefix')
+
 if prefix is not None and not isinstance(prefix, str):
 raise QAPISemError(info,
"Enum '%s' requires a string for 'prefix'" % name)
+
 for member in members:
-check_name(info, "Member of enum '%s'" % name, member,
+source = "Dictionary member of enum '%s'" % name
+check_known_keys(info, source, member, ['name'], [])
+check_name(info, "Member of enum '%s'" % name, member['name'],
enum_member=True)
 
 
@@ -918,6 +937,13 @@ def check_exprs(exprs):
 for builtin in builtin_types.keys():
 all_names[builtin] = 'built-in'
 
+# Normalize exprs
+for 

[Qemu-devel] [PATCH v4 01/51] qlit: use QType instead of int

[Marc-André Lureau]

Suggested-by: Markus Armbruster 
Signed-off-by: Marc-André Lureau 
---
 include/qapi/qmp/qlit.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/qapi/qmp/qlit.h b/include/qapi/qmp/qlit.h
index b18406bce9..0c6809ef3c 100644
--- a/include/qapi/qmp/qlit.h
+++ b/include/qapi/qmp/qlit.h
@@ -21,7 +21,7 @@ typedef struct QLitDictEntry QLitDictEntry;
 typedef struct QLitObject QLitObject;
 
 struct QLitObject {
-int type;
+QType type;
 union {
 bool qbool;
 int64_t qnum;
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 23/51] qapi: add 'if' to enum members

[Marc-André Lureau]

QAPISchemaMember gains .ifcond for enum members: inherited classes,
such as QAPISchemaObjectTypeMember, will thus have an ifcond member
after this (those different types will also use the .ifcond to store
the condition and generate conditional code in the following patches).

Generated code is not changed by this patch, but with "qapi: add #if
conditions to generated code" patch.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py| 10 +++---
 tests/Makefile.include |  1 +
 tests/qapi-schema/enum-dict-member-unknown.err |  2 +-
 tests/qapi-schema/enum-if-invalid.err  |  1 +
 tests/qapi-schema/enum-if-invalid.exit |  1 +
 tests/qapi-schema/enum-if-invalid.json |  3 +++
 tests/qapi-schema/enum-if-invalid.out  |  0
 tests/qapi-schema/qapi-schema-test.json|  5 +++--
 tests/qapi-schema/qapi-schema-test.out |  2 ++
 tests/qapi-schema/test-qapi.py |  1 +
 10 files changed, 20 insertions(+), 6 deletions(-)
 create mode 100644 tests/qapi-schema/enum-if-invalid.err
 create mode 100644 tests/qapi-schema/enum-if-invalid.exit
 create mode 100644 tests/qapi-schema/enum-if-invalid.json
 create mode 100644 tests/qapi-schema/enum-if-invalid.out

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 1338be4efe..e203734254 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -875,7 +875,8 @@ def check_enum(expr, info):
 
 for member in members:
 source = "Dictionary member of enum '%s'" % name
-check_known_keys(info, source, member, ['name'], [])
+check_known_keys(info, source, member, ['name'], ['if'])
+check_if(member, info)
 check_name(info, "Member of enum '%s'" % name, member['name'],
enum_member=True)
 
@@ -1330,9 +1331,10 @@ class QAPISchemaObjectType(QAPISchemaType):
 class QAPISchemaMember(object):
 role = 'member'
 
-def __init__(self, name):
+def __init__(self, name, ifcond=None):
 assert isinstance(name, str)
 self.name = name
+self.ifcond = listify_cond(ifcond)
 self.owner = None
 
 def set_owner(self, name):
@@ -1616,9 +1618,11 @@ class QAPISchema(object):
 for v in values:
 if isinstance(v, dict):
 name = v['name']
+ifcond = v.get('if')
 else:
 name = v
-enum.append(QAPISchemaMember(name))
+ifcond = None
+enum.append(QAPISchemaMember(name, ifcond))
 return enum
 
 def _make_implicit_enum_type(self, name, info, ifcond, values):
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 3634e8f67c..aa5a572403 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -455,6 +455,7 @@ qapi-schema += enum-bad-name.json
 qapi-schema += enum-bad-prefix.json
 qapi-schema += enum-clash-member.json
 qapi-schema += enum-dict-member-unknown.json
+qapi-schema += enum-if-invalid.json
 qapi-schema += enum-int-member.json
 qapi-schema += enum-member-case.json
 qapi-schema += enum-missing-data.json
diff --git a/tests/qapi-schema/enum-dict-member-unknown.err 
b/tests/qapi-schema/enum-dict-member-unknown.err
index 7ed48beb2e..94e02ecd25 100644
--- a/tests/qapi-schema/enum-dict-member-unknown.err
+++ b/tests/qapi-schema/enum-dict-member-unknown.err
@@ -1 +1 @@
-tests/qapi-schema/enum-dict-member-unknown.json:2: Dictionary member of enum 
'MyEnum' has unknown key 'bad-key' (allowed: 'name')
+tests/qapi-schema/enum-dict-member-unknown.json:2: Dictionary member of enum 
'MyEnum' has unknown key 'bad-key' (allowed: 'name', 'if')
diff --git a/tests/qapi-schema/enum-if-invalid.err 
b/tests/qapi-schema/enum-if-invalid.err
new file mode 100644
index 00..54c3cf887b
--- /dev/null
+++ b/tests/qapi-schema/enum-if-invalid.err
@@ -0,0 +1 @@
+tests/qapi-schema/enum-if-invalid.json:2: 'if' condition must be a string or a 
list of strings
diff --git a/tests/qapi-schema/enum-if-invalid.exit 
b/tests/qapi-schema/enum-if-invalid.exit
new file mode 100644
index 00..d00491fd7e
--- /dev/null
+++ b/tests/qapi-schema/enum-if-invalid.exit
@@ -0,0 +1 @@
+1
diff --git a/tests/qapi-schema/enum-if-invalid.json 
b/tests/qapi-schema/enum-if-invalid.json
new file mode 100644
index 00..60bd0ef1d7
--- /dev/null
+++ b/tests/qapi-schema/enum-if-invalid.json
@@ -0,0 +1,3 @@
+# check invalid 'if' type
+{ 'enum': 'TestIfEnum', 'data':
+  [ 'foo', { 'name' : 'bar', 'if': { 'val': 'foo' } } ] }
diff --git a/tests/qapi-schema/enum-if-invalid.out 
b/tests/qapi-schema/enum-if-invalid.out
new file mode 100644
index 00..e69de29bb2
diff --git a/tests/qapi-schema/qapi-schema-test.json 
b/tests/qapi-schema/qapi-schema-test.json
index b997b2d43d..6a1d3b6337 100644
--- a/tests/qapi-schema/qapi-schema-test.json
+++ b/tests/qapi-schema/qapi-schema-test.json
@@ -194,7 +194,8 @@
 { 'struct': 'TestIfStruct', 'data': { 'foo': 'int' },
   'if': 

[Qemu-devel] [PATCH v4 13/51] qapi-commands: add #if conditions to commands

[Marc-André Lureau]

Wrap generated code with #if/#endif using the ifcond_decorator.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi-commands.py  | 2 ++
 tests/test-qmp-commands.c | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 669aef1eb7..8af8d913b9 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -228,6 +228,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 self.defn = None
 self._regy = None
 self._visited_ret_types = None
+self.if_members = ['decl', 'defn', '_regy']
 
 def visit_begin(self, schema):
 self.decl = ''
@@ -240,6 +241,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
 self._regy = None
 self._visited_ret_types = None
 
+@ifcond_decorator
 def visit_command(self, name, info, ifcond, arg_type, ret_type,
   gen, success_response, boxed):
 if not gen:
diff --git a/tests/test-qmp-commands.c b/tests/test-qmp-commands.c
index 9b9a7ffee7..ad7b6e4e1d 100644
--- a/tests/test-qmp-commands.c
+++ b/tests/test-qmp-commands.c
@@ -10,11 +10,11 @@
 
 static QmpCommandList qmp_commands;
 
-/* #if defined(TEST_IF_CMD) */
+#if defined(TEST_IF_CMD)
 void qmp_TestIfCmd(TestIfStruct *foo, Error **errp)
 {
 }
-/* #endif */
+#endif
 
 void qmp_user_def_cmd(Error **errp)
 {
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 07/51] qapi: leave the ifcond attribute undefined until check()

[Marc-André Lureau]

We commonly initialize attributes to None in .init(), then set their
real value in .check().  Accessing the attribute before .check()
yields None.  If we're lucky, the code that accesses the attribute
prematurely chokes on None.

It won't for .ifcond, because None is a legitimate value.

Leave the ifcond attribute undefined until check().

Suggested-by: Markus Armbruster 
Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 21 +
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 8f54dead8d..4d2c214f19 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1011,13 +1011,19 @@ class QAPISchemaEntity(object):
 # such place).
 self.info = info
 self.doc = doc
-self.ifcond = listify_cond(ifcond)
+self._ifcond = ifcond  # self.ifcond is set after check()
 
 def c_name(self):
 return c_name(self.name)
 
 def check(self, schema):
-pass
+if isinstance(self._ifcond, QAPISchemaType):
+# inherit the condition from a type
+typ = self._ifcond
+typ.check(schema)
+self.ifcond = typ.ifcond
+else:
+self.ifcond = listify_cond(self._ifcond)
 
 def is_implicit(self):
 return not self.info
@@ -1138,6 +1144,7 @@ class QAPISchemaEnumType(QAPISchemaType):
 self.prefix = prefix
 
 def check(self, schema):
+QAPISchemaType.check(self, schema)
 seen = {}
 for v in self.values:
 v.check_clash(self.info, seen)
@@ -1170,8 +1177,10 @@ class QAPISchemaArrayType(QAPISchemaType):
 self.element_type = None
 
 def check(self, schema):
+QAPISchemaType.check(self, schema)
 self.element_type = schema.lookup_type(self._element_type_name)
 assert self.element_type
+self.element_type.check(schema)
 self.ifcond = self.element_type.ifcond
 
 def is_implicit(self):
@@ -1214,6 +1223,7 @@ class QAPISchemaObjectType(QAPISchemaType):
 self.members = None
 
 def check(self, schema):
+QAPISchemaType.check(self, schema)
 if self.members is False:   # check for cycles
 raise QAPISemError(self.info,
"Object %s contains itself" % self.name)
@@ -1396,6 +1406,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 self.variants = variants
 
 def check(self, schema):
+QAPISchemaType.check(self, schema)
 self.variants.tag_member.check(schema)
 # Not calling self.variants.check_clash(), because there's nothing
 # to clash with
@@ -1438,6 +1449,7 @@ class QAPISchemaCommand(QAPISchemaEntity):
 self.boxed = boxed
 
 def check(self, schema):
+QAPISchemaEntity.check(self, schema)
 if self._arg_type_name:
 self.arg_type = schema.lookup_type(self._arg_type_name)
 assert (isinstance(self.arg_type, QAPISchemaObjectType) or
@@ -1471,6 +1483,7 @@ class QAPISchemaEvent(QAPISchemaEntity):
 self.boxed = boxed
 
 def check(self, schema):
+QAPISchemaEntity.check(self, schema)
 if self._arg_type_name:
 self.arg_type = schema.lookup_type(self._arg_type_name)
 assert (isinstance(self.arg_type, QAPISchemaObjectType) or
@@ -1589,7 +1602,7 @@ class QAPISchema(object):
 # But it's not tight: the disjunction need not imply it.  We
 # may end up compiling useless wrapper types.
 # TODO kill simple unions or implement the disjunction
-assert ifcond == typ.ifcond
+assert ifcond == typ._ifcond
 else:
 self._def_entity(QAPISchemaObjectType(name, info, doc, ifcond,
   None, members, None))
@@ -1635,7 +1648,7 @@ class QAPISchema(object):
 assert len(typ) == 1
 typ = self._make_array_type(typ[0], info)
 typ = self._make_implicit_object_type(
-typ, info, None, self.lookup_type(typ).ifcond,
+typ, info, None, self.lookup_type(typ),
 'wrapper', [self._make_member('data', typ, info)])
 return QAPISchemaObjectTypeVariant(case, typ)
 
-- 
2.16.0.rc1.1.gef27df75a1

[Qemu-devel] [PATCH v4 12/51] qapi-introspect: add preprocessor conditions to generated QLit

[Marc-André Lureau]

The generator will take (obj, condition) tuples to wrap generated QLit
objects for 'obj' with #if/#endif conditions.

This commit adds 'ifcond' condition to top-level QLit objects.

See generated tests/test-qmp-introspect.c. Example diff after this patch:

--- before  2018-01-08 11:55:24.757083654 +0100
+++ tests/test-qmp-introspect.c 2018-01-08 13:08:44.477641629 +0100
@@ -51,6 +51,8 @@
 { "name", QLIT_QSTR("EVENT_F"), },
 {}
 })),
+#if defined(TEST_IF_CMD)
+#if defined(TEST_IF_STRUCT)
 QLIT_QDICT(((QLitDictEntry[]) {
 { "arg-type", QLIT_QSTR("5"), },
 { "meta-type", QLIT_QSTR("command"), },
@@ -58,12 +60,16 @@
 { "ret-type", QLIT_QSTR("0"), },
 {}
 })),
+#endif /* defined(TEST_IF_STRUCT) */
+#endif /* defined(TEST_IF_CMD) */

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi-introspect.py | 31 +--
 1 file changed, 21 insertions(+), 10 deletions(-)

diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index b1d08ec97b..7d3a5c37fd 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -17,6 +17,15 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
 def indent(level):
 return level * 4 * ' '
 
+if isinstance(obj, tuple):
+ifobj, ifcond = obj
+ret = gen_if(ifcond)
+ret += to_qlit(ifobj, level)
+endif = gen_endif(ifcond)
+if endif:
+ret += '\n' + endif
+return ret
+
 ret = ''
 if not suppress_first_indent:
 ret += indent(level)
@@ -25,7 +34,7 @@ def to_qlit(obj, level=0, suppress_first_indent=False):
 elif isinstance(obj, str):
 ret += 'QLIT_QSTR(' + to_c_string(obj) + ')'
 elif isinstance(obj, list):
-elts = [to_qlit(elt, level + 1)
+elts = [to_qlit(elt, level + 1).strip('\n')
 for elt in obj]
 elts.append(indent(level + 1) + "{}")
 ret += 'QLIT_QLIST(((QLitObject[]) {\n'
@@ -121,12 +130,12 @@ const QLitObject %(c_name)s = %(c_string)s;
 return '[' + self._use_type(typ.element_type) + ']'
 return self._name(typ.name)
 
-def _gen_qlit(self, name, mtype, obj):
+def _gen_qlit(self, name, mtype, obj, ifcond):
 if mtype not in ('command', 'event', 'builtin', 'array'):
 name = self._name(name)
 obj['name'] = name
 obj['meta-type'] = mtype
-self._qlits.append(obj)
+self._qlits.append((obj, ifcond))
 
 def _gen_member(self, member):
 ret = {'name': member.name, 'type': self._use_type(member.type)}
@@ -142,26 +151,27 @@ const QLitObject %(c_name)s = %(c_string)s;
 return {'case': variant.name, 'type': self._use_type(variant.type)}
 
 def visit_builtin_type(self, name, info, json_type):
-self._gen_qlit(name, 'builtin', {'json-type': json_type})
+self._gen_qlit(name, 'builtin', {'json-type': json_type}, [])
 
 def visit_enum_type(self, name, info, ifcond, values, prefix):
-self._gen_qlit(name, 'enum', {'values': values})
+self._gen_qlit(name, 'enum', {'values': values}, ifcond)
 
 def visit_array_type(self, name, info, ifcond, element_type):
 element = self._use_type(element_type)
-self._gen_qlit('[' + element + ']', 'array', {'element-type': element})
+self._gen_qlit('[' + element + ']', 'array', {'element-type': element},
+   ifcond)
 
 def visit_object_type_flat(self, name, info, ifcond, members, variants):
 obj = {'members': [self._gen_member(m) for m in members]}
 if variants:
 obj.update(self._gen_variants(variants.tag_member.name,
   variants.variants))
-self._gen_qlit(name, 'object', obj)
+self._gen_qlit(name, 'object', obj, ifcond)
 
 def visit_alternate_type(self, name, info, ifcond, variants):
 self._gen_qlit(name, 'alternate',
{'members': [{'type': self._use_type(m.type)}
-for m in variants.variants]})
+for m in variants.variants]}, ifcond)
 
 def visit_command(self, name, info, ifcond, arg_type, ret_type,
   gen, success_response, boxed):
@@ -169,11 +179,12 @@ const QLitObject %(c_name)s = %(c_string)s;
 ret_type = ret_type or self._schema.the_empty_object_type
 self._gen_qlit(name, 'command',
{'arg-type': self._use_type(arg_type),
-'ret-type': self._use_type(ret_type)})
+'ret-type': self._use_type(ret_type)}, ifcond)
 
 def visit_event(self, name, info, ifcond, arg_type, boxed):
 arg_type = arg_type or self._schema.the_empty_object_type
-self._gen_qlit(name, 'event', {'arg-type': self._use_type(arg_type)})
+self._gen_qlit(name, 'event', {'arg-type': 

Re: [Qemu-devel] [PATCH v3 26/50] qapi: add 'if' on union variants

[Marc-André Lureau]

Hi

On Mon, Dec 11, 2017 at 11:36 AM, Markus Armbruster  wrote:
> Marc-André Lureau  writes:
>
>> Signed-off-by: Marc-André Lureau 
>> ---
>>  scripts/qapi.py | 15 ++-
>>  tests/qapi-schema/qapi-schema-test.json |  7 ++-
>>  tests/qapi-schema/qapi-schema-test.out  |  8 
>>  tests/qapi-schema/test-qapi.py  |  5 -
>>  4 files changed, 28 insertions(+), 7 deletions(-)
>>
>> diff --git a/scripts/qapi.py b/scripts/qapi.py
>> index 15711f96fa..2f14edfa1f 100644
>> --- a/scripts/qapi.py
>> +++ b/scripts/qapi.py
>> @@ -1412,8 +1412,8 @@ class QAPISchemaObjectTypeVariants(object):
>>  class QAPISchemaObjectTypeVariant(QAPISchemaObjectTypeMember):
>>  role = 'branch'
>>
>> -def __init__(self, name, typ):
>> -QAPISchemaObjectTypeMember.__init__(self, name, typ, False)
>> +def __init__(self, name, typ, ifcond=None):
>> +QAPISchemaObjectTypeMember.__init__(self, name, typ, False, ifcond)
>>
>>
>>  class QAPISchemaAlternateType(QAPISchemaType):
>> @@ -1674,13 +1674,18 @@ class QAPISchema(object):
>>  return QAPISchemaObjectTypeVariant(case, typ)
>>
>>  def _make_simple_variant(self, case, typ, info):
>> +ifcond = None
>> +if isinstance(typ, dict):
>> +check_unknown_keys(info, typ, {'type', 'if'})
>> +ifcond = typ.get('if')
>> +typ = typ['type']
>>  if isinstance(typ, list):
>>  assert len(typ) == 1
>>  typ = self._make_array_type(typ[0], info)
>>  typ = self._make_implicit_object_type(
>>  typ, info, None, self.lookup_type(typ).ifcond,
>>  'wrapper', [self._make_member('data', typ, info)])
>> -return QAPISchemaObjectTypeVariant(case, typ)
>> +return QAPISchemaObjectTypeVariant(case, typ, ifcond)
>>
>>  def _def_union_type(self, expr, info, doc):
>>  name = expr['union']
>> @@ -1700,8 +1705,8 @@ class QAPISchema(object):
>>  else:
>>  variants = [self._make_simple_variant(key, value, info)
>>  for (key, value) in data.iteritems()]
>> -typ = self._make_implicit_enum_type(name, info, ifcond,
>> -[v.name for v in variants])
>> +values = [{'name': v.name, 'if': v.ifcond} for v in variants]
>> +typ = self._make_implicit_enum_type(name, info, ifcond, values)
>>  tag_member = QAPISchemaObjectTypeMember('type', typ, False)
>>  members = [tag_member]
>>  self._def_entity(
>> diff --git a/tests/qapi-schema/qapi-schema-test.json 
>> b/tests/qapi-schema/qapi-schema-test.json
>> index 5cfccabb3d..895e80a978 100644
>> --- a/tests/qapi-schema/qapi-schema-test.json
>> +++ b/tests/qapi-schema/qapi-schema-test.json
>> @@ -200,9 +200,14 @@
>>[ 'foo', { 'name' : 'bar', 'if': 'defined(TEST_IF_ENUM_BAR)' } ],
>>'if': 'defined(TEST_IF_ENUM)' }
>>
>> -{ 'union': 'TestIfUnion', 'data': { 'foo': 'TestStruct' },
>> +{ 'union': 'TestIfUnion', 'data':
>> +  { 'foo': 'TestStruct',
>> +'union_bar': { 'type': 'str', 'if': 'defined(TEST_IF_UNION_BAR)'} },
>>'if': 'defined(TEST_IF_UNION) && defined(TEST_IF_STRUCT)' }
>>
>> +{ 'command': 'TestIfUnionCmd', 'data': { 'union_cmd_arg': 'TestIfUnion' },
>> +  'if': 'defined(TEST_IF_UNION)' }
>> +
>>  { 'alternate': 'TestIfAlternate', 'data': { 'foo': 'int', 'bar': 
>> 'TestStruct' },
>>'if': 'defined(TEST_IF_ALT) && defined(TEST_IF_STRUCT)' }
>>
>> diff --git a/tests/qapi-schema/qapi-schema-test.out 
>> b/tests/qapi-schema/qapi-schema-test.out
>> index 6df4e49c69..ee009c5626 100644
>> --- a/tests/qapi-schema/qapi-schema-test.out
>> +++ b/tests/qapi-schema/qapi-schema-test.out
>> @@ -87,9 +87,14 @@ object TestIfUnion
>>  member type: TestIfUnionKind optional=False
>>  tag type
>>  case foo: q_obj_TestStruct-wrapper
>> +case union_bar: q_obj_str-wrapper if=defined(TEST_IF_UNION_BAR)
>
> PATCH 22, but I only spotted it here.  We say "if=COND" in some places,
> ...
>
>>  if defined(TEST_IF_UNION) && defined(TEST_IF_STRUCT)
>> +command TestIfUnionCmd q_obj_TestIfUnionCmd-arg -> None
>> +   gen=True success_response=True boxed=False
>> +if defined(TEST_IF_UNION)
>
> ... and "if COND" in other places.  I guess the '=' is there to match
> existing flag printing like optional=BOOL.
>
> I'd prefer less decorated output, i.e. instead of
>
> enum TestIfEnum
> member foo:
> member bar: if=defined(TEST_IF_ENUM_BAR)
> if defined(TEST_IF_ENUM)
>
> something like
>
> enum TestIfEnum
> member foo
> member bar
> if defined(TEST_IF_ENUM_BAR)
> if defined(TEST_IF_ENUM)
>
> Could touch that up on commit.
>

ok, changed

>>  enum TestIfUnionKind
>>  member foo:
>> +member union_bar: if=defined(TEST_IF_UNION_BAR)
>>  if defined(TEST_IF_UNION) && 

[Qemu-devel] [PATCH v4 20/51] tests: modify visit_enum_type() in test-qapi to print members

[Marc-André Lureau]

Use a common self._print_members() to print enum members details.

Signed-off-by: Marc-André Lureau 
---
 tests/qapi-schema/comments.out   | 14 +++--
 tests/qapi-schema/doc-bad-section.out| 13 +++--
 tests/qapi-schema/doc-good.out   | 17 +--
 tests/qapi-schema/empty.out  |  9 +-
 tests/qapi-schema/event-case.out |  9 +-
 tests/qapi-schema/ident-with-escape.out  |  9 +-
 tests/qapi-schema/include-relpath.out| 14 +++--
 tests/qapi-schema/include-repetition.out | 14 +++--
 tests/qapi-schema/include-simple.out | 14 +++--
 tests/qapi-schema/indented-expr.out  |  9 +-
 tests/qapi-schema/qapi-schema-test.out   | 49 ++--
 tests/qapi-schema/test-qapi.py   | 15 +++---
 12 files changed, 156 insertions(+), 30 deletions(-)

diff --git a/tests/qapi-schema/comments.out b/tests/qapi-schema/comments.out
index 17e652535c..7c243b246b 100644
--- a/tests/qapi-schema/comments.out
+++ b/tests/qapi-schema/comments.out
@@ -1,4 +1,14 @@
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+enum QType
 prefix QTYPE
-enum Status ['good', 'bad', 'ugly']
+member none
+member qnull
+member qnum
+member qstring
+member qdict
+member qlist
+member qbool
+enum Status
+member good
+member bad
+member ugly
 object q_empty
diff --git a/tests/qapi-schema/doc-bad-section.out 
b/tests/qapi-schema/doc-bad-section.out
index 089bde1381..77e223ba41 100644
--- a/tests/qapi-schema/doc-bad-section.out
+++ b/tests/qapi-schema/doc-bad-section.out
@@ -1,6 +1,15 @@
-enum Enum ['one', 'two']
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+enum Enum
+member one
+member two
+enum QType
 prefix QTYPE
+member none
+member qnull
+member qnum
+member qstring
+member qdict
+member qlist
+member qbool
 object q_empty
 doc symbol=Enum
 body=
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index 1d2c250527..e615b04281 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -1,19 +1,30 @@
 object Base
 member base1: Enum optional=False
-enum Enum ['one', 'two']
+enum Enum
+member one
+member two
 object Object
 base Base
 tag base1
 case one: Variant1
 case two: Variant2
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+enum QType
 prefix QTYPE
+member none
+member qnull
+member qnum
+member qstring
+member qdict
+member qlist
+member qbool
 object SugaredUnion
 member type: SugaredUnionKind optional=False
 tag type
 case one: q_obj_Variant1-wrapper
 case two: q_obj_Variant2-wrapper
-enum SugaredUnionKind ['one', 'two']
+enum SugaredUnionKind
+member one
+member two
 object Variant1
 member var1: str optional=False
 object Variant2
diff --git a/tests/qapi-schema/empty.out b/tests/qapi-schema/empty.out
index 40b886ddae..0bd6d1421d 100644
--- a/tests/qapi-schema/empty.out
+++ b/tests/qapi-schema/empty.out
@@ -1,3 +1,10 @@
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+enum QType
 prefix QTYPE
+member none
+member qnull
+member qnum
+member qstring
+member qdict
+member qlist
+member qbool
 object q_empty
diff --git a/tests/qapi-schema/event-case.out b/tests/qapi-schema/event-case.out
index 313c0fe7be..fc64383edb 100644
--- a/tests/qapi-schema/event-case.out
+++ b/tests/qapi-schema/event-case.out
@@ -1,5 +1,12 @@
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+enum QType
 prefix QTYPE
+member none
+member qnull
+member qnum
+member qstring
+member qdict
+member qlist
+member qbool
 event oops None
boxed=False
 object q_empty
diff --git a/tests/qapi-schema/ident-with-escape.out 
b/tests/qapi-schema/ident-with-escape.out
index b5637cb2e0..6458ffb28c 100644
--- a/tests/qapi-schema/ident-with-escape.out
+++ b/tests/qapi-schema/ident-with-escape.out
@@ -1,5 +1,12 @@
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+enum QType
 prefix QTYPE
+member none
+member qnull
+member qnum
+member qstring
+member qdict
+member qlist
+member qbool
 command fooA q_obj_fooA-arg -> None
gen=True success_response=True boxed=False
 object q_empty
diff --git a/tests/qapi-schema/include-relpath.out 
b/tests/qapi-schema/include-relpath.out
index 17e652535c..7c243b246b 100644
--- a/tests/qapi-schema/include-relpath.out
+++ b/tests/qapi-schema/include-relpath.out
@@ -1,4 +1,14 @@
-enum QType ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist', 'qbool']
+enum QType
 prefix QTYPE
-enum Status ['good', 'bad', 'ugly']
+member none
+member qnull
+member qnum
+member qstring
+member qdict
+member qlist
+member 

[Qemu-devel] [PATCH v4 03/51] qapi: generate a literal qobject for introspection

[Marc-André Lureau]

Replace the generated json string with a literal qobject. The later is
easier to deal with, at run time as well as compile time: adding #if
conditionals will be easier than in a json string.

Signed-off-by: Marc-André Lureau 
Reviewed-by: Markus Armbruster 
---
 scripts/qapi-introspect.py | 83 ++
 monitor.c  |  2 +-
 tests/test-qobject-input-visitor.c | 11 +++--
 docs/devel/qapi-code-gen.txt   | 29 -
 4 files changed, 76 insertions(+), 49 deletions(-)

diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 032bcea491..0002bc1a68 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -12,26 +12,36 @@
 from qapi import *
 
 
-# Caveman's json.dumps() replacement (we're stuck at Python 2.4)
-# TODO try to use json.dumps() once we get unstuck
-def to_json(obj, level=0):
+def to_qlit(obj, level=0, suppress_first_indent=False):
+
+def indent(level):
+return level * 4 * ' '
+
+ret = ''
+if not suppress_first_indent:
+ret += indent(level)
 if obj is None:
-ret = 'null'
+ret += 'QLIT_QNULL'
 elif isinstance(obj, str):
-ret = '"' + obj.replace('"', r'\"') + '"'
+ret += 'QLIT_QSTR(' + to_c_string(obj) + ')'
 elif isinstance(obj, list):
-elts = [to_json(elt, level + 1)
+elts = [to_qlit(elt, level + 1)
 for elt in obj]
-ret = '[' + ', '.join(elts) + ']'
+elts.append(indent(level + 1) + "{}")
+ret += 'QLIT_QLIST(((QLitObject[]) {\n'
+ret += ',\n'.join(elts) + '\n'
+ret += indent(level) + '}))'
 elif isinstance(obj, dict):
-elts = ['"%s": %s' % (key.replace('"', r'\"'),
-  to_json(obj[key], level + 1))
-for key in sorted(obj.keys())]
-ret = '{' + ', '.join(elts) + '}'
+elts = []
+for key, value in sorted(obj.iteritems()):
+elts.append(indent(level + 1) + '{ %s, %s }' %
+(to_c_string(key), to_qlit(value, level + 1, True)))
+elts.append(indent(level + 1) + '{}')
+ret += 'QLIT_QDICT(((QLitDictEntry[]) {\n'
+ret += ',\n'.join(elts) + '\n'
+ret += indent(level) + '}))'
 else:
 assert False# not implemented
-if level == 1:
-ret = '\n' + ret
 return ret
 
 
@@ -45,39 +55,37 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaVisitor):
 self.defn = None
 self.decl = None
 self._schema = None
-self._jsons = None
+self._qlits = None
 self._used_types = None
 self._name_map = None
 
 def visit_begin(self, schema):
 self._schema = schema
-self._jsons = []
+self._qlits = []
 self._used_types = []
 self._name_map = {}
 
 def visit_end(self):
 # visit the types that are actually used
-jsons = self._jsons
-self._jsons = []
+qlits = self._qlits
+self._qlits = []
 for typ in self._used_types:
 typ.visit(self)
 # generate C
 # TODO can generate awfully long lines
-jsons.extend(self._jsons)
-name = c_name(prefix, protect=False) + 'qmp_schema_json'
+qlits.extend(self._qlits)
+name = c_name(prefix, protect=False) + 'qmp_schema_qlit'
 self.decl = mcgen('''
-extern const char %(c_name)s[];
+extern const QLitObject %(c_name)s;
 ''',
   c_name=c_name(name))
-lines = to_json(jsons).split('\n')
-c_string = '\n'.join([to_c_string(line) for line in lines])
 self.defn = mcgen('''
-const char %(c_name)s[] = %(c_string)s;
+const QLitObject %(c_name)s = %(c_string)s;
 ''',
   c_name=c_name(name),
-  c_string=c_string)
+  c_string=to_qlit(qlits))
 self._schema = None
-self._jsons = None
+self._qlits = None
 self._used_types = None
 self._name_map = None
 
@@ -111,12 +119,12 @@ const char %(c_name)s[] = %(c_string)s;
 return '[' + self._use_type(typ.element_type) + ']'
 return self._name(typ.name)
 
-def _gen_json(self, name, mtype, obj):
+def _gen_qlit(self, name, mtype, obj):
 if mtype not in ('command', 'event', 'builtin', 'array'):
 name = self._name(name)
 obj['name'] = name
 obj['meta-type'] = mtype
-self._jsons.append(obj)
+self._qlits.append(obj)
 
 def _gen_member(self, member):
 ret = {'name': member.name, 'type': self._use_type(member.type)}
@@ -132,24 +140,24 @@ const char %(c_name)s[] = %(c_string)s;
 return {'case': variant.name, 'type': self._use_type(variant.type)}
 
 def visit_builtin_type(self, name, info, json_type):
-self._gen_json(name, 'builtin', 

[Qemu-devel] [PATCH v4 10/51] qapi: add #if/#endif helpers

[Marc-André Lureau]

Add helpers to wrap generated code with #if/#endif lines.

Add a function decorator that will be used to wrap visitor methods.
The decorator will check if code was generated before adding #if/#endif
lines. Used in the following patches.

Signed-off-by: Marc-André Lureau 
---
 scripts/qapi.py | 47 +++
 1 file changed, 47 insertions(+)

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 3d33ed7d76..71f28fc6d8 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -1911,6 +1911,53 @@ def guardend(name):
  name=guardname(name))
 
 
+def gen_if(ifcond):
+ret = ''
+for ifc in ifcond:
+ret += mcgen('''
+#if %(cond)s
+''', cond=ifc)
+return ret
+
+
+def gen_endif(ifcond):
+ret = ''
+for ifc in reversed(ifcond):
+ret += mcgen('''
+#endif /* %(cond)s */
+''', cond=ifc)
+return ret
+
+
+# Wrap a method to add #if / #endif to generated code, only if some
+# code was generated. The method must have an 'ifcond' argument.
+# self must have 'if_members' listing the attributes to wrap.
+def ifcond_decorator(func):
+
+def func_wrapper(self, *args, **kwargs):
+import inspect
+idx = inspect.getargspec(func).args.index('ifcond')
+ifcond = args[idx - 1]
+save = {}
+for mem in self.if_members:
+save[mem] = getattr(self, mem)
+func(self, *args, **kwargs)
+for mem, val in save.items():
+newval = getattr(self, mem)
+if newval != val:
+assert newval.startswith(val)
+newval = newval[len(val):]
+if newval[0] == '\n':
+val += '\n'
+newval = newval[1:]
+val += gen_if(ifcond)
+val += newval
+val += gen_endif(ifcond)
+setattr(self, mem, val)
+
+return func_wrapper
+
+
 def gen_enum_lookup(name, values, prefix=None):
 ret = mcgen('''
 
-- 
2.16.0.rc1.1.gef27df75a1

Re: [Qemu-devel] [PATCH v3 24/50] qapi: add some struct member tests

[Marc-André Lureau]

Hi

On Sat, Dec 9, 2017 at 10:07 AM, Markus Armbruster  wrote:
> Marc-André Lureau  writes:
>
>> Signed-off-by: Marc-André Lureau 
>> ---
>>  tests/Makefile.include|  2 ++
>>  tests/qapi-schema/struct-if-invalid.err   |  1 +
>>  tests/qapi-schema/struct-if-invalid.exit  |  1 +
>>  tests/qapi-schema/struct-if-invalid.json  |  3 +++
>>  tests/qapi-schema/struct-if-invalid.out   |  0
>>  tests/qapi-schema/struct-member-type.err  |  0
>>  tests/qapi-schema/struct-member-type.exit |  1 +
>>  tests/qapi-schema/struct-member-type.json |  2 ++
>>  tests/qapi-schema/struct-member-type.out  | 12 
>>  9 files changed, 22 insertions(+)
>>  create mode 100644 tests/qapi-schema/struct-if-invalid.err
>>  create mode 100644 tests/qapi-schema/struct-if-invalid.exit
>>  create mode 100644 tests/qapi-schema/struct-if-invalid.json
>>  create mode 100644 tests/qapi-schema/struct-if-invalid.out
>>  create mode 100644 tests/qapi-schema/struct-member-type.err
>>  create mode 100644 tests/qapi-schema/struct-member-type.exit
>>  create mode 100644 tests/qapi-schema/struct-member-type.json
>>  create mode 100644 tests/qapi-schema/struct-member-type.out
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index 0aa532f029..44a3d8895e 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -520,7 +520,9 @@ qapi-schema += returns-whitelist.json
>>  qapi-schema += struct-base-clash-deep.json
>>  qapi-schema += struct-base-clash.json
>>  qapi-schema += struct-data-invalid.json
>> +qapi-schema += struct-if-invalid.json
>>  qapi-schema += struct-member-invalid.json
>> +qapi-schema += struct-member-type.json
>>  qapi-schema += trailing-comma-list.json
>>  qapi-schema += trailing-comma-object.json
>>  qapi-schema += type-bypass-bad-gen.json
>> diff --git a/tests/qapi-schema/struct-if-invalid.err 
>> b/tests/qapi-schema/struct-if-invalid.err
>> new file mode 100644
>> index 00..42245262a9
>> --- /dev/null
>> +++ b/tests/qapi-schema/struct-if-invalid.err
>> @@ -0,0 +1 @@
>> +tests/qapi-schema/struct-if-invalid.json:2: Member 'bar' of 'data' for 
>> struct 'TestIfStruct' should be a type name
>> diff --git a/tests/qapi-schema/struct-if-invalid.exit 
>> b/tests/qapi-schema/struct-if-invalid.exit
>> new file mode 100644
>> index 00..d00491fd7e
>> --- /dev/null
>> +++ b/tests/qapi-schema/struct-if-invalid.exit
>> @@ -0,0 +1 @@
>> +1
>> diff --git a/tests/qapi-schema/struct-if-invalid.json 
>> b/tests/qapi-schema/struct-if-invalid.json
>> new file mode 100644
>> index 00..466cdb61e1
>> --- /dev/null
>> +++ b/tests/qapi-schema/struct-if-invalid.json
>> @@ -0,0 +1,3 @@
>> +# check missing 'type'
>> +{ 'struct': 'TestIfStruct', 'data':
>> +  { 'foo': 'int', 'bar': { 'if': 'defined(TEST_IF_STRUCT_BAR)' } } }
>
> Hmm.  This tests the previous patch's change to check_type().  It
> demonstrates that the "should be a type name" error message can be
> suboptimal: it gets triggered when
>
> not isinstance(value, str)
> and not (isinstance(value, dict) and 'type' in value)
>
> Fine when the value is neither str not dict.  Suboptimal when it's dict
> and 'type' is missing.  A more obvious example of suboptimality would be
>
>'bar': { 'tvpe': 'int' }
>
> The previous patch's
>
> if isinstance(value, dict) and 'type' in value:
> check_type(info, source, value['type'], allow_array,
>allow_dict, allow_optional, allow_metas)
> if 'if' in value:
> check_if(value, info)
> check_unknown_keys(info, value, {'type', 'if'})
> return
> else:
> raise QAPISemError(info, "%s should be a type name" % source)
>
> should be improved to something like
>
> if not isinstance(value, dict):
> raise QAPISemError(
> info, "%s should be a type name or dictionary" % source)
> if 'type' not in value:
> raise QAPISemError(
> info, "%s must have a member 'type'" % source)
> check_type(info, source, value['type'], allow_array,
>allow_dict, allow_optional, allow_metas)
> if 'if' in value:
> check_if(value, info)
> check_unknown_keys(info, value, {'type', 'if'})
> return
>
> producing
>
> struct-if-invalid.json:2: Member 'bar' of 'data' for struct 
> 'TestIfStruct' must have a member 'type'
>

Fixed differently in v4

> The fact that I missed this in review of the code, but noticed it in the
> tests is why I want tests added together with the code they test.
>

Changed in v4

> Nitpick: the file name struct-if-invalid makes me expect an invalid if.
> Not the case.  It's a missing type.  Let's rename accordingly.

Done

>
>> diff --git a/tests/qapi-schema/struct-if-invalid.out 
>> b/tests/qapi-schema/struct-if-invalid.out
>> new file mode 100644
>> index 

[Qemu-devel] [PATCH v4 05/51] qapi: add 'if' to top-level expressions

[Marc-André Lureau]

Accept 'if' key in top-level elements, accepted as string or list of
string type. The following patches will modify the test visitor to
check the value is correctly saved, and generate #if/#endif code (as a
single #if/endif line or a series for a list).

Example of 'if' key:
{ 'struct': 'TestIfStruct', 'data': { 'foo': 'int' },
  'if': 'defined(TEST_IF_STRUCT)' }

The generated code is for now *unconditional*. Later patches generate
the conditionals.

A following patch for qapi-code-gen.txt will provide more complete
documentation for 'if' usage.

Signed-off-by: Marc-André Lureau 
Reviewed-by: Markus Armbruster 
---
 scripts/qapi.py  | 36 ++--
 tests/test-qmp-commands.c|  6 ++
 tests/Makefile.include   |  4 
 tests/qapi-schema/bad-if-empty-list.err  |  1 +
 tests/qapi-schema/bad-if-empty-list.exit |  1 +
 tests/qapi-schema/bad-if-empty-list.json |  3 +++
 tests/qapi-schema/bad-if-empty-list.out  |  0
 tests/qapi-schema/bad-if-empty.err   |  1 +
 tests/qapi-schema/bad-if-empty.exit  |  1 +
 tests/qapi-schema/bad-if-empty.json  |  3 +++
 tests/qapi-schema/bad-if-empty.out   |  0
 tests/qapi-schema/bad-if-list.err|  1 +
 tests/qapi-schema/bad-if-list.exit   |  1 +
 tests/qapi-schema/bad-if-list.json   |  3 +++
 tests/qapi-schema/bad-if-list.out|  0
 tests/qapi-schema/bad-if.err |  1 +
 tests/qapi-schema/bad-if.exit|  1 +
 tests/qapi-schema/bad-if.json|  3 +++
 tests/qapi-schema/bad-if.out |  0
 tests/qapi-schema/qapi-schema-test.json  | 20 ++
 tests/qapi-schema/qapi-schema-test.out   | 22 +++
 21 files changed, 102 insertions(+), 6 deletions(-)
 create mode 100644 tests/qapi-schema/bad-if-empty-list.err
 create mode 100644 tests/qapi-schema/bad-if-empty-list.exit
 create mode 100644 tests/qapi-schema/bad-if-empty-list.json
 create mode 100644 tests/qapi-schema/bad-if-empty-list.out
 create mode 100644 tests/qapi-schema/bad-if-empty.err
 create mode 100644 tests/qapi-schema/bad-if-empty.exit
 create mode 100644 tests/qapi-schema/bad-if-empty.json
 create mode 100644 tests/qapi-schema/bad-if-empty.out
 create mode 100644 tests/qapi-schema/bad-if-list.err
 create mode 100644 tests/qapi-schema/bad-if-list.exit
 create mode 100644 tests/qapi-schema/bad-if-list.json
 create mode 100644 tests/qapi-schema/bad-if-list.out
 create mode 100644 tests/qapi-schema/bad-if.err
 create mode 100644 tests/qapi-schema/bad-if.exit
 create mode 100644 tests/qapi-schema/bad-if.json
 create mode 100644 tests/qapi-schema/bad-if.out

diff --git a/scripts/qapi.py b/scripts/qapi.py
index 43a54bf40f..27df0fcf48 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -630,6 +630,27 @@ def add_name(name, info, meta, implicit=False):
 all_names[name] = meta
 
 
+def check_if(expr, info):
+
+def check_if_str(ifcond, info):
+if not isinstance(ifcond, str):
+raise QAPISemError(
+info, "'if' condition must be a string or a list of strings")
+if ifcond == '':
+raise QAPISemError(info, "'if' condition '' makes no sense")
+
+ifcond = expr.get('if')
+if ifcond is None:
+return
+elif isinstance(ifcond, list):
+if ifcond == []:
+raise QAPISemError(info, "'if' condition [] is useless")
+for elt in ifcond:
+check_if_str(elt, info)
+else:
+check_if_str(ifcond, info)
+
+
 def check_type(info, source, value, allow_array=False,
allow_dict=False, allow_optional=False,
allow_metas=[]):
@@ -869,6 +890,8 @@ def check_keys(expr_elem, meta, required, optional=[]):
 raise QAPISemError(info,
"'%s' of %s '%s' should only use true value"
% (key, meta, name))
+if key == 'if':
+check_if(expr, info)
 for key in required:
 if key not in expr:
 raise QAPISemError(info, "Key '%s' is missing from %s '%s'"
@@ -894,27 +917,28 @@ def check_exprs(exprs):
 
 if 'enum' in expr:
 meta = 'enum'
-check_keys(expr_elem, 'enum', ['data'], ['prefix'])
+check_keys(expr_elem, 'enum', ['data'], ['if', 'prefix'])
 enum_types[expr[meta]] = expr
 elif 'union' in expr:
 meta = 'union'
 check_keys(expr_elem, 'union', ['data'],
-   ['base', 'discriminator'])
+   ['base', 'discriminator', 'if'])
 union_types[expr[meta]] = expr
 elif 'alternate' in expr:
 meta = 'alternate'
-check_keys(expr_elem, 'alternate', ['data'])
+check_keys(expr_elem, 'alternate', ['data'], ['if'])
 elif 'struct' in expr:
 meta = 'struct'
-check_keys(expr_elem, 'struct',