Re: [libvirt] [PATCH v3 02/22] build-aux: rewrite augest test generator in Python

2019-09-26 Thread Daniel P . Berrangé 
On Thu, Sep 26, 2019 at 05:58:47PM +0200, Peter Krempa wrote:
> On Thu, Sep 26, 2019 at 16:40:54 +0100, Daniel Berrange wrote:
> > On Thu, Sep 26, 2019 at 05:35:46PM +0200, Ján Tomko wrote:
> > > On Wed, Sep 25, 2019 at 03:17:10PM +0100, Daniel P. Berrangé wrote:
> > > > On Wed, Sep 25, 2019 at 03:25:39PM +0200, Ján Tomko wrote:
> > > > > On Tue, Sep 24, 2019 at 03:58:43PM +0100, Daniel P. Berrangé wrote:
> 
> [...]
> 
> > > > If we create 'scripts/' should we put everything in there, or just
> > > > stuff that's related to the top level, and keep everything else in
> > > > their current subdirs ?
> > > 
> > > We put all the driver-specific tests into one tests/ directory, I think
> > > doing it for scripts makes sense too.
> > 
> > I'd really like the move all the tests into their respective driver
> > directories at some point :-)
> 
> I hope you intend to stash them into a 'tests' subdirectory or
> something. Otherwise I'll object as the driver directories themselves
> are getting too clobbered with non-test stuff already.

I've not thought that far ahead to be honest - its just a random wishlist
item at number 783 on the todo list.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 02/22] build-aux: rewrite augest test generator in Python

2019-09-26 Thread Peter Krempa 
On Thu, Sep 26, 2019 at 16:40:54 +0100, Daniel Berrange wrote:
> On Thu, Sep 26, 2019 at 05:35:46PM +0200, Ján Tomko wrote:
> > On Wed, Sep 25, 2019 at 03:17:10PM +0100, Daniel P. Berrangé wrote:
> > > On Wed, Sep 25, 2019 at 03:25:39PM +0200, Ján Tomko wrote:
> > > > On Tue, Sep 24, 2019 at 03:58:43PM +0100, Daniel P. Berrangé wrote:

[...]

> > > If we create 'scripts/' should we put everything in there, or just
> > > stuff that's related to the top level, and keep everything else in
> > > their current subdirs ?
> > 
> > We put all the driver-specific tests into one tests/ directory, I think
> > doing it for scripts makes sense too.
> 
> I'd really like the move all the tests into their respective driver
> directories at some point :-)

I hope you intend to stash them into a 'tests' subdirectory or
something. Otherwise I'll object as the driver directories themselves
are getting too clobbered with non-test stuff already.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 02/22] build-aux: rewrite augest test generator in Python

2019-09-26 Thread Daniel P . Berrangé 
On Thu, Sep 26, 2019 at 05:35:46PM +0200, Ján Tomko wrote:
> On Wed, Sep 25, 2019 at 03:17:10PM +0100, Daniel P. Berrangé wrote:
> > On Wed, Sep 25, 2019 at 03:25:39PM +0200, Ján Tomko wrote:
> > > On Tue, Sep 24, 2019 at 03:58:43PM +0100, Daniel P. Berrangé wrote:
> > > > As part of an goal to eliminate Perl from libvirt build tools,
> > > > rewrite the augeas-gentest.pl tool in Python.
> > > >
> > > > This was a straight conversion, manually going line-by-line to
> > > > change the syntax from Perl to Python. Thus the overall structure
> > > > of the file and approach is the same.
> > > >
> > > > The use of $(AUG_GENTEST) as a dependancy in the makefiles needed
> > > 
> > > s/dependancy/dependency/
> > > 
> > > > to be fixed, because this was assumed to be the filename of the
> > > > script, but is in fact a full shell command line.
> > > >
> > > 
> > > This is the case regardless of the Perl->Python conversion
> > > and can be done upfront to reduce the churn in this patch.
> > > Introduced by commit fb59cf7a5824b9c876737dcbf6aac97c29b1444a
> > > 
> > > > Signed-off-by: Daniel P. Berrangé 
> > > > ---
> > > > Makefile.am |  2 +-
> > > > build-aux/augeas-gentest.pl | 60 ---
> > > > build-aux/augeas-gentest.py | 72 +
> > > > src/Makefile.am |  3 +-
> > > > src/bhyve/Makefile.inc.am   |  4 +-
> > > > src/interface/Makefile.inc.am   |  2 +-
> > > > src/libxl/Makefile.inc.am   |  4 +-
> > > > src/locking/Makefile.inc.am |  6 +--
> > > > src/logging/Makefile.inc.am |  2 +-
> > > > src/lxc/Makefile.inc.am |  4 +-
> > > > src/network/Makefile.inc.am |  2 +-
> > > > src/node_device/Makefile.inc.am |  2 +-
> > > > src/nwfilter/Makefile.inc.am|  2 +-
> > > > src/qemu/Makefile.inc.am|  4 +-
> > > > src/remote/Makefile.inc.am  |  4 +-
> > > > src/secret/Makefile.inc.am  |  2 +-
> > > > src/storage/Makefile.inc.am |  2 +-
> > > > src/vbox/Makefile.inc.am|  2 +-
> > > > src/vz/Makefile.inc.am  |  2 +-
> > > > 19 files changed, 97 insertions(+), 84 deletions(-)
> > > > delete mode 100755 build-aux/augeas-gentest.pl
> > > > create mode 100755 build-aux/augeas-gentest.py
> > > 
> > > Since this is a new file with clean history, it might actually deserve
> > > a better location than build-aux and we can leave this directory to
> > > Automake and gnulib to do whatever magic they do there.
> > > Also note that the directory is in .gitignore. (I added the exception
> > > for .pl files back when I added files here)
> > > 
> > > Would 'scripts' be too vague? Could be a good place to put the helper
> > > scripts for generating QEMU caps files since I never seem to remember
> > > its name and tests/ is growing quite big.
> > 
> > That's a good question. As you see from this series, we've got random
> > scripts scattered all over the sub-dirs. build-aux/ was in some sense
> > to avoid polluting the top level dir.
> > 
> > If we create 'scripts/' should we put everything in there, or just
> > stuff that's related to the top level, and keep everything else in
> > their current subdirs ?
> 
> We put all the driver-specific tests into one tests/ directory, I think
> doing it for scripts makes sense too.

I'd really like the move all the tests into their respective driver
directories at some point :-)

We don't have an enourmous number of scripts though, so having them
all in one place is fine.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 02/22] build-aux: rewrite augest test generator in Python

2019-09-26 Thread Ján Tomko 

On Wed, Sep 25, 2019 at 03:17:10PM +0100, Daniel P. Berrangé wrote:

On Wed, Sep 25, 2019 at 03:25:39PM +0200, Ján Tomko wrote:

On Tue, Sep 24, 2019 at 03:58:43PM +0100, Daniel P. Berrangé wrote:
> As part of an goal to eliminate Perl from libvirt build tools,
> rewrite the augeas-gentest.pl tool in Python.
>
> This was a straight conversion, manually going line-by-line to
> change the syntax from Perl to Python. Thus the overall structure
> of the file and approach is the same.
>
> The use of $(AUG_GENTEST) as a dependancy in the makefiles needed

s/dependancy/dependency/

> to be fixed, because this was assumed to be the filename of the
> script, but is in fact a full shell command line.
>

This is the case regardless of the Perl->Python conversion
and can be done upfront to reduce the churn in this patch.
Introduced by commit fb59cf7a5824b9c876737dcbf6aac97c29b1444a

> Signed-off-by: Daniel P. Berrangé 
> ---
> Makefile.am |  2 +-
> build-aux/augeas-gentest.pl | 60 ---
> build-aux/augeas-gentest.py | 72 +
> src/Makefile.am |  3 +-
> src/bhyve/Makefile.inc.am   |  4 +-
> src/interface/Makefile.inc.am   |  2 +-
> src/libxl/Makefile.inc.am   |  4 +-
> src/locking/Makefile.inc.am |  6 +--
> src/logging/Makefile.inc.am |  2 +-
> src/lxc/Makefile.inc.am |  4 +-
> src/network/Makefile.inc.am |  2 +-
> src/node_device/Makefile.inc.am |  2 +-
> src/nwfilter/Makefile.inc.am|  2 +-
> src/qemu/Makefile.inc.am|  4 +-
> src/remote/Makefile.inc.am  |  4 +-
> src/secret/Makefile.inc.am  |  2 +-
> src/storage/Makefile.inc.am |  2 +-
> src/vbox/Makefile.inc.am|  2 +-
> src/vz/Makefile.inc.am  |  2 +-
> 19 files changed, 97 insertions(+), 84 deletions(-)
> delete mode 100755 build-aux/augeas-gentest.pl
> create mode 100755 build-aux/augeas-gentest.py

Since this is a new file with clean history, it might actually deserve
a better location than build-aux and we can leave this directory to
Automake and gnulib to do whatever magic they do there.
Also note that the directory is in .gitignore. (I added the exception
for .pl files back when I added files here)

Would 'scripts' be too vague? Could be a good place to put the helper
scripts for generating QEMU caps files since I never seem to remember
its name and tests/ is growing quite big.


That's a good question. As you see from this series, we've got random
scripts scattered all over the sub-dirs. build-aux/ was in some sense
to avoid polluting the top level dir.

If we create 'scripts/' should we put everything in there, or just
stuff that's related to the top level, and keep everything else in
their current subdirs ?


We put all the driver-specific tests into one tests/ directory, I think
doing it for scripts makes sense too.

Jano




Regards,
Daniel


signature.asc
Description: PGP signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 02/22] build-aux: rewrite augest test generator in Python

2019-09-25 Thread Daniel P . Berrangé 
On Wed, Sep 25, 2019 at 03:25:39PM +0200, Ján Tomko wrote:
> On Tue, Sep 24, 2019 at 03:58:43PM +0100, Daniel P. Berrangé wrote:
> > As part of an goal to eliminate Perl from libvirt build tools,
> > rewrite the augeas-gentest.pl tool in Python.
> > 
> > This was a straight conversion, manually going line-by-line to
> > change the syntax from Perl to Python. Thus the overall structure
> > of the file and approach is the same.
> > 
> > The use of $(AUG_GENTEST) as a dependancy in the makefiles needed
> 
> s/dependancy/dependency/
> 
> > to be fixed, because this was assumed to be the filename of the
> > script, but is in fact a full shell command line.
> > 
> 
> This is the case regardless of the Perl->Python conversion
> and can be done upfront to reduce the churn in this patch.
> Introduced by commit fb59cf7a5824b9c876737dcbf6aac97c29b1444a
> 
> > Signed-off-by: Daniel P. Berrangé 
> > ---
> > Makefile.am |  2 +-
> > build-aux/augeas-gentest.pl | 60 ---
> > build-aux/augeas-gentest.py | 72 +
> > src/Makefile.am |  3 +-
> > src/bhyve/Makefile.inc.am   |  4 +-
> > src/interface/Makefile.inc.am   |  2 +-
> > src/libxl/Makefile.inc.am   |  4 +-
> > src/locking/Makefile.inc.am |  6 +--
> > src/logging/Makefile.inc.am |  2 +-
> > src/lxc/Makefile.inc.am |  4 +-
> > src/network/Makefile.inc.am |  2 +-
> > src/node_device/Makefile.inc.am |  2 +-
> > src/nwfilter/Makefile.inc.am|  2 +-
> > src/qemu/Makefile.inc.am|  4 +-
> > src/remote/Makefile.inc.am  |  4 +-
> > src/secret/Makefile.inc.am  |  2 +-
> > src/storage/Makefile.inc.am |  2 +-
> > src/vbox/Makefile.inc.am|  2 +-
> > src/vz/Makefile.inc.am  |  2 +-
> > 19 files changed, 97 insertions(+), 84 deletions(-)
> > delete mode 100755 build-aux/augeas-gentest.pl
> > create mode 100755 build-aux/augeas-gentest.py
> 
> Since this is a new file with clean history, it might actually deserve
> a better location than build-aux and we can leave this directory to
> Automake and gnulib to do whatever magic they do there.
> Also note that the directory is in .gitignore. (I added the exception
> for .pl files back when I added files here)
> 
> Would 'scripts' be too vague? Could be a good place to put the helper
> scripts for generating QEMU caps files since I never seem to remember
> its name and tests/ is growing quite big.

That's a good question. As you see from this series, we've got random
scripts scattered all over the sub-dirs. build-aux/ was in some sense
to avoid polluting the top level dir.

If we create 'scripts/' should we put everything in there, or just
stuff that's related to the top level, and keep everything else in
their current subdirs ?


Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v3 02/22] build-aux: rewrite augest test generator in Python

2019-09-25 Thread Ján Tomko 

On Tue, Sep 24, 2019 at 03:58:43PM +0100, Daniel P. Berrangé wrote:

As part of an goal to eliminate Perl from libvirt build tools,
rewrite the augeas-gentest.pl tool in Python.

This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.

The use of $(AUG_GENTEST) as a dependancy in the makefiles needed


s/dependancy/dependency/


to be fixed, because this was assumed to be the filename of the
script, but is in fact a full shell command line.



This is the case regardless of the Perl->Python conversion
and can be done upfront to reduce the churn in this patch.
Introduced by commit fb59cf7a5824b9c876737dcbf6aac97c29b1444a


Signed-off-by: Daniel P. Berrangé 
---
Makefile.am |  2 +-
build-aux/augeas-gentest.pl | 60 ---
build-aux/augeas-gentest.py | 72 +
src/Makefile.am |  3 +-
src/bhyve/Makefile.inc.am   |  4 +-
src/interface/Makefile.inc.am   |  2 +-
src/libxl/Makefile.inc.am   |  4 +-
src/locking/Makefile.inc.am |  6 +--
src/logging/Makefile.inc.am |  2 +-
src/lxc/Makefile.inc.am |  4 +-
src/network/Makefile.inc.am |  2 +-
src/node_device/Makefile.inc.am |  2 +-
src/nwfilter/Makefile.inc.am|  2 +-
src/qemu/Makefile.inc.am|  4 +-
src/remote/Makefile.inc.am  |  4 +-
src/secret/Makefile.inc.am  |  2 +-
src/storage/Makefile.inc.am |  2 +-
src/vbox/Makefile.inc.am|  2 +-
src/vz/Makefile.inc.am  |  2 +-
19 files changed, 97 insertions(+), 84 deletions(-)
delete mode 100755 build-aux/augeas-gentest.pl
create mode 100755 build-aux/augeas-gentest.py


Since this is a new file with clean history, it might actually deserve
a better location than build-aux and we can leave this directory to
Automake and gnulib to do whatever magic they do there.
Also note that the directory is in .gitignore. (I added the exception
for .pl files back when I added files here)

Would 'scripts' be too vague? Could be a good place to put the helper
scripts for generating QEMU caps files since I never seem to remember
its name and tests/ is growing quite big.

[...]


+def expand_config(config):
+with open(config) as fh:
+optprog = re.compile(r'''^#\w.*''')
+groupstartprog = re.compile(r'''.*\[\s$''')
+groupendprog = re.compile(r'''#\s*\].*$''')
+
+group = False
+for line in fh:
+if optprog.match(line) is not None:
+line = line[1:]
+line = line.replace('"', '\\"')
+print(line, end='')
+if groupstartprog.match(line):
+group = True
+elif group:
+line = line.replace('"', '\\"')
+
+if groupendprog.match(line):
+group = False
+
+if line[0] == '#':
+line = line[1:]
+print(line, end='')
+
+
+def expand_template(template, config):
+with open(template) as fh:
+markerprog = re.compile(r'''\s*@CONFIG@\s*''')
+for line in fh:
+if markerprog.match(line) is not None:


Simpler as:
 if '@CONFIG@' in line:
There's no need to use a regex here


+print('   let conf = "', end='')


This uses three spaces instead of two like the original Perl script.
Three match the rest of the .aug files, but by including that change
here, this rewrite generates a different output.


+expand_config(config)
+print('"')
+else:
+print(line, end='')
+
+
+expand_template(template, config)


Reviewed-by: Ján Tomko 

Jano


signature.asc
Description: PGP signature
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v3 02/22] build-aux: rewrite augest test generator in Python

2019-09-24 Thread Daniel P . Berrangé 
As part of an goal to eliminate Perl from libvirt build tools,
rewrite the augeas-gentest.pl tool in Python.

This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.

The use of $(AUG_GENTEST) as a dependancy in the makefiles needed
to be fixed, because this was assumed to be the filename of the
script, but is in fact a full shell command line.

Signed-off-by: Daniel P. Berrangé 
---
 Makefile.am |  2 +-
 build-aux/augeas-gentest.pl | 60 ---
 build-aux/augeas-gentest.py | 72 +
 src/Makefile.am |  3 +-
 src/bhyve/Makefile.inc.am   |  4 +-
 src/interface/Makefile.inc.am   |  2 +-
 src/libxl/Makefile.inc.am   |  4 +-
 src/locking/Makefile.inc.am |  6 +--
 src/logging/Makefile.inc.am |  2 +-
 src/lxc/Makefile.inc.am |  4 +-
 src/network/Makefile.inc.am |  2 +-
 src/node_device/Makefile.inc.am |  2 +-
 src/nwfilter/Makefile.inc.am|  2 +-
 src/qemu/Makefile.inc.am|  4 +-
 src/remote/Makefile.inc.am  |  4 +-
 src/secret/Makefile.inc.am  |  2 +-
 src/storage/Makefile.inc.am |  2 +-
 src/vbox/Makefile.inc.am|  2 +-
 src/vz/Makefile.inc.am  |  2 +-
 19 files changed, 97 insertions(+), 84 deletions(-)
 delete mode 100755 build-aux/augeas-gentest.pl
 create mode 100755 build-aux/augeas-gentest.py

diff --git a/Makefile.am b/Makefile.am
index 711f365504..17448a914e 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -41,7 +41,7 @@ EXTRA_DIST = \
   run.in \
   README.md \
   AUTHORS.in \
-  build-aux/augeas-gentest.pl \
+  build-aux/augeas-gentest.py \
   build-aux/check-spacing.pl \
   build-aux/gitlog-to-changelog \
   build-aux/header-ifdef.pl \
diff --git a/build-aux/augeas-gentest.pl b/build-aux/augeas-gentest.pl
deleted file mode 100755
index 65834b533b..00
--- a/build-aux/augeas-gentest.pl
+++ /dev/null
@@ -1,60 +0,0 @@
-#!/usr/bin/env perl
-#
-# augeas-gentest.pl: Generate an augeas test file, from an
-#example config file + test file template
-#
-# This library is free software; you can redistribute it and/or
-# modify it under the terms of the GNU Lesser General Public
-# License as published by the Free Software Foundation; either
-# version 2.1 of the License, or (at your option) any later version.
-#
-# This library is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
-# Lesser General Public License for more details.
-#
-# You should have received a copy of the GNU Lesser General Public
-# License along with this library.  If not, see
-# .
-
-use strict;
-use warnings;
-
-die "syntax: $0 CONFIG TEMPLATE\n" unless @ARGV == 2;
-
-my $config = shift @ARGV;
-my $template = shift @ARGV;
-
-open CONFIG, "<", $config or die "cannot read $config: $!";
-open TEMPLATE, "<", $template or die "cannot read $template: $!";
-
-my $group = 0;
-while () {
-if (/\@CONFIG\@/) {
-my $group = 0;
-print "  let conf = \"";
-while () {
-if (/^#\w/) {
-s/^#//;
-s/\"/\\\"/g;
-print $_;
-$group = /\[\s$/;
-} elsif ($group) {
-s/\"/\\\"/g;
-if (/#\s*\]/) {
-$group = 0;
-}
-if (/^#/) {
-s/^#//;
-print $_;
-}
-}
-}
-print "\"\n";
-} else {
-print $_;
-}
-}
-
-close TEMPLATE;
-close CONFIG;
diff --git a/build-aux/augeas-gentest.py b/build-aux/augeas-gentest.py
new file mode 100755
index 00..18c6490a08
--- /dev/null
+++ b/build-aux/augeas-gentest.py
@@ -0,0 +1,72 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2012-2019 Red Hat, Inc.
+#
+# augeas-gentest.py: Generate an augeas test file, from an
+#example config file + test file template
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library.  If not, see
+# .
+
+from __future__ import print_function
+
+import re
+import sys
+
+if len(sys.argv) != 3:
+print("syntax: %s CONFIG TEMPLATE" % sys.argv[0], file=sys.stderr)