Re: [Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

2018-02-26 Thread Eric Blake 

On 02/23/2018 11:20 AM, Markus Armbruster wrote:

Eric Blake  writes:


On 02/11/2018 03:35 AM, Markus Armbruster wrote:

Whenever qapi-schema.json changes, we run six programs eleven times to
update eleven files.  Similar for qga/qapi-schema.json.  This is
silly.  Replace the six programs by a single program that spits out
all eleven files.

The programs become modules in new Python package qapi, along with the
helper library.  This requires moving them to scripts/qapi/.

Signed-off-by: Markus Armbruster 
Reviewed-by: Marc-André Lureau 
---



+++ b/docs/devel/qapi-code-gen.txt



   @@ -1302,23 +1296,22 @@ Example:
   }
 const char *const example_QAPIEvent_lookup[] = {
-[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
+
+[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
   [EXAMPLE_QAPI_EVENT__MAX] = NULL,
   };


Looks like our generated code indentation has slightly regressed from
what we would write by hand, but it's still okay for generated code
(and the commit message in 5/29 did call that out)


I'd prefer to have this tidied up.


Actually, we currently output with good indentation, but with different 
text altogether:


const QEnumLookup example_QAPIEvent_lookup = {
.array = (const char *const[]) {
[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
},
.size = EXAMPLE_QAPI_EVENT__MAX
};

I'm fixing that up as part of prepping this series for PULL.



+def gen_commands(schema, output_dir, prefix):
+blurb = ' * Schema-defined QAPI/QMP commands'


We discussed whether to make the assignment to blurb be a one-liner in
an earlier patch, but I'm also fine with the churn you have over the
course of the series.


I ran out of time, and decided to leave this one as is.


It wasn't too difficult (although the ripple effects hit 3 other 
patches), so I did this as well.


--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

2018-02-23 Thread Markus Armbruster 
Eric Blake  writes:

> On 02/11/2018 03:35 AM, Markus Armbruster wrote:
>> Whenever qapi-schema.json changes, we run six programs eleven times to
>> update eleven files.  Similar for qga/qapi-schema.json.  This is
>> silly.  Replace the six programs by a single program that spits out
>> all eleven files.
>>
>> The programs become modules in new Python package qapi, along with the
>> helper library.  This requires moving them to scripts/qapi/.
>>
>> Signed-off-by: Markus Armbruster 
>> Reviewed-by: Marc-André Lureau 
>> ---
>>   .gitignore |  2 +
>>   Makefile   | 86 +--
>>   docs/devel/qapi-code-gen.txt   | 97 
>> ++
>>   monitor.c  |  2 +-
>>   qapi-schema.json   |  2 +-
>>   scripts/qapi-gen.py| 41 +
>>   scripts/qapi/__init__.py   |  0
>>   scripts/{qapi-commands.py => qapi/commands.py} | 23 ++---
>>   scripts/{qapi.py => qapi/common.py}| 18 +---
>>   scripts/{qapi2texi.py => qapi/doc.py}  | 29 ++-
>>   scripts/{qapi-event.py => qapi/events.py}  | 23 ++---
>>   scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++-
>>   scripts/{qapi-types.py => qapi/types.py}   | 34 ++--
>>   scripts/{qapi-visit.py => qapi/visit.py}   | 34 ++--
>>   tests/Makefile.include | 56 ++---
>>   tests/qapi-schema/test-qapi.py |  4 +-
>>   16 files changed, 193 insertions(+), 290 deletions(-)
>>   create mode 100755 scripts/qapi-gen.py
>>   create mode 100644 scripts/qapi/__init__.py
>>   rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
>>   rename scripts/{qapi.py => qapi/common.py} (99%)
>>   rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
>>   mode change 100755 => 100644
>
> Still forgot mention that the mode bit change was intentional, but not
> worth a respin for just that.
>
>>   rename scripts/{qapi-event.py => qapi/events.py} (92%)
>>   rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
>>   rename scripts/{qapi-types.py => qapi/types.py} (90%)
>>   rename scripts/{qapi-visit.py => qapi/visit.py} (92%)
>
> Reviewed-by: Eric Blake 
>
>> +++ b/docs/devel/qapi-code-gen.txt
>
>> -$ python scripts/qapi-event.py --output-dir="qapi-generated"
>> ---prefix="example-" example-schema.json
>>   $ cat qapi-generated/example-qapi-event.h
>>   [Uninteresting stuff omitted...]
>>   @@ -1302,23 +1296,22 @@ Example:
>>   }
>> const char *const example_QAPIEvent_lookup[] = {
>> -[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
>> +
>> +[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
>>   [EXAMPLE_QAPI_EVENT__MAX] = NULL,
>>   };
>
> Looks like our generated code indentation has slightly regressed from
> what we would write by hand, but it's still okay for generated code
> (and the commit message in 5/29 did call that out)

I'd prefer to have this tidied up.

>> +++ b/scripts/qapi-gen.py
>
>> +++ b/scripts/qapi/commands.py
>> @@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, 
>> version 2.
>>   See the COPYING file in the top-level directory.
>>   """
>>   -from qapi import *
>> +from qapi.common import *
>>   def gen_command_decl(name, arg_type, boxed, ret_type):
>> @@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
>>   self._regy += gen_register_command(name, success_response)
>> -def main(argv):
>> -(input_file, output_dir, do_c, do_h, prefix, opts) = 
>> parse_command_line()
>> -
>> -blurb = '''
>> - * Schema-defined QAPI/QMP commands
>> -'''
>> -
>> +def gen_commands(schema, output_dir, prefix):
>> +blurb = ' * Schema-defined QAPI/QMP commands'
>
> We discussed whether to make the assignment to blurb be a one-liner in
> an earlier patch, but I'm also fine with the churn you have over the
> course of the series.

I ran out of time, and decided to leave this one as is.

Re: [Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

2018-02-19 Thread Michael Roth 
Quoting Markus Armbruster (2018-02-11 03:35:46)
> Whenever qapi-schema.json changes, we run six programs eleven times to
> update eleven files.  Similar for qga/qapi-schema.json.  This is
> silly.  Replace the six programs by a single program that spits out
> all eleven files.
> 
> The programs become modules in new Python package qapi, along with the
> helper library.  This requires moving them to scripts/qapi/.
> 
> Signed-off-by: Markus Armbruster 
> Reviewed-by: Marc-André Lureau 

Reviewed-by: Michael Roth 

> ---
>  .gitignore |  2 +
>  Makefile   | 86 +--
>  docs/devel/qapi-code-gen.txt   | 97 
> ++
>  monitor.c  |  2 +-
>  qapi-schema.json   |  2 +-
>  scripts/qapi-gen.py| 41 +
>  scripts/qapi/__init__.py   |  0
>  scripts/{qapi-commands.py => qapi/commands.py} | 23 ++---
>  scripts/{qapi.py => qapi/common.py}| 18 +---
>  scripts/{qapi2texi.py => qapi/doc.py}  | 29 ++-
>  scripts/{qapi-event.py => qapi/events.py}  | 23 ++---
>  scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++-
>  scripts/{qapi-types.py => qapi/types.py}   | 34 ++--
>  scripts/{qapi-visit.py => qapi/visit.py}   | 34 ++--
>  tests/Makefile.include | 56 ++---
>  tests/qapi-schema/test-qapi.py |  4 +-
>  16 files changed, 193 insertions(+), 290 deletions(-)
>  create mode 100755 scripts/qapi-gen.py
>  create mode 100644 scripts/qapi/__init__.py
>  rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
>  rename scripts/{qapi.py => qapi/common.py} (99%)
>  rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
>  mode change 100755 => 100644
>  rename scripts/{qapi-event.py => qapi/events.py} (92%)
>  rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
>  rename scripts/{qapi-types.py => qapi/types.py} (90%)
>  rename scripts/{qapi-visit.py => qapi/visit.py} (92%)
> 
> diff --git a/.gitignore b/.gitignore
> index 704b22285d..2f9a92f6cc 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -28,9 +28,11 @@
>  /linux-headers/asm
>  /qga/qapi-generated
>  /qapi-generated
> +/qapi-gen-timestamp
>  /qapi-types.[ch]
>  /qapi-visit.[ch]
>  /qapi-event.[ch]
> +/qapi-doc.texi
>  /qmp-commands.h
>  /qmp-introspect.[ch]
>  /qmp-marshal.c
> diff --git a/Makefile b/Makefile
> index 4ec7a3cb82..bd781c6aad 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -94,6 +94,7 @@ GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h 
> qapi-event.h
>  GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
>  GENERATED_FILES += qmp-introspect.h
>  GENERATED_FILES += qmp-introspect.c
> +GENERATED_FILES += qapi-doc.texi
> 
>  GENERATED_FILES += trace/generated-tcg-tracers.h
> 
> @@ -483,25 +484,26 @@ qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated
>  qemu-keymap$(EXESUF): LIBS += $(XKBCOMMON_LIBS)
>  qemu-keymap$(EXESUF): QEMU_CFLAGS += $(XKBCOMMON_CFLAGS)
> 
> -gen-out-type = $(subst .,-,$(suffix $@))
> +qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
> +$(SRC_PATH)/scripts/qapi/events.py \
> +$(SRC_PATH)/scripts/qapi/introspect.py \
> +$(SRC_PATH)/scripts/qapi/types.py \
> +$(SRC_PATH)/scripts/qapi/visit.py \
> +$(SRC_PATH)/scripts/qapi/common.py \
> +$(SRC_PATH)/scripts/qapi/doc.py \
> +$(SRC_PATH)/scripts/ordereddict.py \
> +$(SRC_PATH)/scripts/qapi-gen.py
> 
> -qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
> -
> -qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\
> -$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
> -   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-types.py 
> \
> -   $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
> -   "GEN","$@")
> -qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h :\
> -$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
> -   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-visit.py 
> \
> -   $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
> -   "GEN","$@")
> -qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c :\
> -$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py 
> $(qapi-py)
> -   $(call quiet-command,$(PYTHON_UTF8) 
> $(SRC_PATH)/scripts/qapi-commands.py \
> -   $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
> -   "GEN","$@")
> +qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
> +qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
> +qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
> +qga/qapi-generated/qga-q

Re: [Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

2018-02-12 Thread Eric Blake 

On 02/11/2018 03:35 AM, Markus Armbruster wrote:

Whenever qapi-schema.json changes, we run six programs eleven times to
update eleven files.  Similar for qga/qapi-schema.json.  This is
silly.  Replace the six programs by a single program that spits out
all eleven files.

The programs become modules in new Python package qapi, along with the
helper library.  This requires moving them to scripts/qapi/.

Signed-off-by: Markus Armbruster 
Reviewed-by: Marc-André Lureau 
---
  .gitignore |  2 +
  Makefile   | 86 +--
  docs/devel/qapi-code-gen.txt   | 97 ++
  monitor.c  |  2 +-
  qapi-schema.json   |  2 +-
  scripts/qapi-gen.py| 41 +
  scripts/qapi/__init__.py   |  0
  scripts/{qapi-commands.py => qapi/commands.py} | 23 ++---
  scripts/{qapi.py => qapi/common.py}| 18 +---
  scripts/{qapi2texi.py => qapi/doc.py}  | 29 ++-
  scripts/{qapi-event.py => qapi/events.py}  | 23 ++---
  scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++-
  scripts/{qapi-types.py => qapi/types.py}   | 34 ++--
  scripts/{qapi-visit.py => qapi/visit.py}   | 34 ++--
  tests/Makefile.include | 56 ++---
  tests/qapi-schema/test-qapi.py |  4 +-
  16 files changed, 193 insertions(+), 290 deletions(-)
  create mode 100755 scripts/qapi-gen.py
  create mode 100644 scripts/qapi/__init__.py
  rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
  rename scripts/{qapi.py => qapi/common.py} (99%)
  rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
  mode change 100755 => 100644


Still forgot mention that the mode bit change was intentional, but not 
worth a respin for just that.



  rename scripts/{qapi-event.py => qapi/events.py} (92%)
  rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
  rename scripts/{qapi-types.py => qapi/types.py} (90%)
  rename scripts/{qapi-visit.py => qapi/visit.py} (92%)


Reviewed-by: Eric Blake 


+++ b/docs/devel/qapi-code-gen.txt



-$ python scripts/qapi-event.py --output-dir="qapi-generated"
---prefix="example-" example-schema.json
  $ cat qapi-generated/example-qapi-event.h
  [Uninteresting stuff omitted...]
  
@@ -1302,23 +1296,22 @@ Example:

  }
  
  const char *const example_QAPIEvent_lookup[] = {

-[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
+
+[EXAMPLE_QAPI_EVENT_MY_EVENT] = "MY_EVENT",
  [EXAMPLE_QAPI_EVENT__MAX] = NULL,
  };


Looks like our generated code indentation has slightly regressed from 
what we would write by hand, but it's still okay for generated code (and 
the commit message in 5/29 did call that out)



+++ b/scripts/qapi-gen.py



+++ b/scripts/qapi/commands.py
@@ -13,7 +13,7 @@ This work is licensed under the terms of the GNU GPL, version 
2.
  See the COPYING file in the top-level directory.
  """
  
-from qapi import *

+from qapi.common import *
  
  
  def gen_command_decl(name, arg_type, boxed, ret_type):

@@ -255,13 +255,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
  self._regy += gen_register_command(name, success_response)
  
  
-def main(argv):

-(input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
-
-blurb = '''
- * Schema-defined QAPI/QMP commands
-'''
-
+def gen_commands(schema, output_dir, prefix):
+blurb = ' * Schema-defined QAPI/QMP commands'


We discussed whether to make the assignment to blurb be a one-liner in 
an earlier patch, but I'm also fine with the churn you have over the 
course of the series.



--
Eric Blake, Principal Software Engineer
Red Hat, Inc.   +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

[Qemu-devel] [PATCH v2 08/29] qapi-gen: New common driver for code and doc generators

2018-02-11 Thread Markus Armbruster 
Whenever qapi-schema.json changes, we run six programs eleven times to
update eleven files.  Similar for qga/qapi-schema.json.  This is
silly.  Replace the six programs by a single program that spits out
all eleven files.

The programs become modules in new Python package qapi, along with the
helper library.  This requires moving them to scripts/qapi/.

Signed-off-by: Markus Armbruster 
Reviewed-by: Marc-André Lureau 
---
 .gitignore |  2 +
 Makefile   | 86 +--
 docs/devel/qapi-code-gen.txt   | 97 ++
 monitor.c  |  2 +-
 qapi-schema.json   |  2 +-
 scripts/qapi-gen.py| 41 +
 scripts/qapi/__init__.py   |  0
 scripts/{qapi-commands.py => qapi/commands.py} | 23 ++---
 scripts/{qapi.py => qapi/common.py}| 18 +---
 scripts/{qapi2texi.py => qapi/doc.py}  | 29 ++-
 scripts/{qapi-event.py => qapi/events.py}  | 23 ++---
 scripts/{qapi-introspect.py => qapi/introspect.py} | 32 ++-
 scripts/{qapi-types.py => qapi/types.py}   | 34 ++--
 scripts/{qapi-visit.py => qapi/visit.py}   | 34 ++--
 tests/Makefile.include | 56 ++---
 tests/qapi-schema/test-qapi.py |  4 +-
 16 files changed, 193 insertions(+), 290 deletions(-)
 create mode 100755 scripts/qapi-gen.py
 create mode 100644 scripts/qapi/__init__.py
 rename scripts/{qapi-commands.py => qapi/commands.py} (94%)
 rename scripts/{qapi.py => qapi/common.py} (99%)
 rename scripts/{qapi2texi.py => qapi/doc.py} (92%)
 mode change 100755 => 100644
 rename scripts/{qapi-event.py => qapi/events.py} (92%)
 rename scripts/{qapi-introspect.py => qapi/introspect.py} (90%)
 rename scripts/{qapi-types.py => qapi/types.py} (90%)
 rename scripts/{qapi-visit.py => qapi/visit.py} (92%)

diff --git a/.gitignore b/.gitignore
index 704b22285d..2f9a92f6cc 100644
--- a/.gitignore
+++ b/.gitignore
@@ -28,9 +28,11 @@
 /linux-headers/asm
 /qga/qapi-generated
 /qapi-generated
+/qapi-gen-timestamp
 /qapi-types.[ch]
 /qapi-visit.[ch]
 /qapi-event.[ch]
+/qapi-doc.texi
 /qmp-commands.h
 /qmp-introspect.[ch]
 /qmp-marshal.c
diff --git a/Makefile b/Makefile
index 4ec7a3cb82..bd781c6aad 100644
--- a/Makefile
+++ b/Makefile
@@ -94,6 +94,7 @@ GENERATED_FILES += qmp-commands.h qapi-types.h qapi-visit.h 
qapi-event.h
 GENERATED_FILES += qmp-marshal.c qapi-types.c qapi-visit.c qapi-event.c
 GENERATED_FILES += qmp-introspect.h
 GENERATED_FILES += qmp-introspect.c
+GENERATED_FILES += qapi-doc.texi
 
 GENERATED_FILES += trace/generated-tcg-tracers.h
 
@@ -483,25 +484,26 @@ qemu-ga$(EXESUF): QEMU_CFLAGS += -I qga/qapi-generated
 qemu-keymap$(EXESUF): LIBS += $(XKBCOMMON_LIBS)
 qemu-keymap$(EXESUF): QEMU_CFLAGS += $(XKBCOMMON_CFLAGS)
 
-gen-out-type = $(subst .,-,$(suffix $@))
+qapi-py = $(SRC_PATH)/scripts/qapi/commands.py \
+$(SRC_PATH)/scripts/qapi/events.py \
+$(SRC_PATH)/scripts/qapi/introspect.py \
+$(SRC_PATH)/scripts/qapi/types.py \
+$(SRC_PATH)/scripts/qapi/visit.py \
+$(SRC_PATH)/scripts/qapi/common.py \
+$(SRC_PATH)/scripts/qapi/doc.py \
+$(SRC_PATH)/scripts/ordereddict.py \
+$(SRC_PATH)/scripts/qapi-gen.py
 
-qapi-py = $(SRC_PATH)/scripts/qapi.py $(SRC_PATH)/scripts/ordereddict.py
-
-qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-types.py $(qapi-py)
-   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-types.py \
-   $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-   "GEN","$@")
-qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-visit.py $(qapi-py)
-   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-visit.py \
-   $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-   "GEN","$@")
-qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c :\
-$(SRC_PATH)/qga/qapi-schema.json $(SRC_PATH)/scripts/qapi-commands.py 
$(qapi-py)
-   $(call quiet-command,$(PYTHON_UTF8) 
$(SRC_PATH)/scripts/qapi-commands.py \
-   $(gen-out-type) -o qga/qapi-generated -p "qga-" $<, \
-   "GEN","$@")
+qga/qapi-generated/qga-qapi-types.c qga/qapi-generated/qga-qapi-types.h \
+qga/qapi-generated/qga-qapi-visit.c qga/qapi-generated/qga-qapi-visit.h \
+qga/qapi-generated/qga-qmp-commands.h qga/qapi-generated/qga-qmp-marshal.c \
+qga/qapi-generated/qga-qapi-doc.texi: \
+qga/qapi-generated/qapi-gen-timestamp ;
+qga/qapi-generated/qapi-gen-timestamp: $(SRC_PATH)/qga/qapi-schema.json 
$(qapi-py)
+   $(call quiet-command,$(PYTHON_UTF8) $(SRC_PATH)/scripts/qapi-gen.py \
+   -o qga/qapi-generated -p "qga-" $<, \
+   "GEN