Re: [libvirt] [PATCH v5 17/23] tests: rewrite file access checker in Python

2019-12-04 Thread Michal Privoznik 

On 12/4/19 2:19 PM, Daniel P. Berrangé wrote:

On Mon, Nov 18, 2019 at 02:11:33PM -0500, Cole Robinson wrote:

On 11/11/19 9:38 AM, Daniel P. Berrangé wrote:

As part of an goal to eliminate Perl from libvirt build tools,
rewrite the check-file-access.pl tool in Python.

This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.

Signed-off-by: Daniel P. Berrangé 
---
  Makefile.am |   1 +
  scripts/check-file-access.py| 123 +++
  tests/Makefile.am   |   3 +-
  tests/check-file-access.pl  | 126 
  tests/file_access_whitelist.txt |   2 +-
  5 files changed, 126 insertions(+), 129 deletions(-)
  create mode 100755 scripts/check-file-access.py
  delete mode 100755 tests/check-file-access.pl



`make -C builddir check-access` fails like:

LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 /usr/bin/python3
/home/crobinso/src/libvirt/scripts/check-file-access.py | sort -u
Traceback (most recent call last):
   File "/home/crobinso/src/libvirt/scripts/check-file-access.py", line
53, in 
 with open(whitelist_file, "r") as fh:
FileNotFoundError: [Errno 2] No such file or directory:
'file_access_whitelist.txt'
make[1]: Leaving directory '/home/crobinso/src/libvirt/builddir/tests'
make: Leaving directory '/home/crobinso/src/libvirt/builddir'


Looks like this is unrelated to this patch - the original script
has the same flaw wrt VPATH builds, and indeed even the Makefile.am
is currently broken.


Yes, I guess it never worked with VPATH. Anyway, patch proposed here:

https://www.redhat.com/archives/libvir-list/2019-December/msg00204.html

Michal

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v5 17/23] tests: rewrite file access checker in Python

2019-12-04 Thread Daniel P . Berrangé 
On Mon, Nov 18, 2019 at 02:11:33PM -0500, Cole Robinson wrote:
> On 11/11/19 9:38 AM, Daniel P. Berrangé wrote:
> > As part of an goal to eliminate Perl from libvirt build tools,
> > rewrite the check-file-access.pl tool in Python.
> > 
> > This was a straight conversion, manually going line-by-line to
> > change the syntax from Perl to Python. Thus the overall structure
> > of the file and approach is the same.
> > 
> > Signed-off-by: Daniel P. Berrangé 
> > ---
> >  Makefile.am |   1 +
> >  scripts/check-file-access.py| 123 +++
> >  tests/Makefile.am   |   3 +-
> >  tests/check-file-access.pl  | 126 
> >  tests/file_access_whitelist.txt |   2 +-
> >  5 files changed, 126 insertions(+), 129 deletions(-)
> >  create mode 100755 scripts/check-file-access.py
> >  delete mode 100755 tests/check-file-access.pl
> > 
> 
> `make -C builddir check-access` fails like:
> 
> LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 /usr/bin/python3
> /home/crobinso/src/libvirt/scripts/check-file-access.py | sort -u
> Traceback (most recent call last):
>   File "/home/crobinso/src/libvirt/scripts/check-file-access.py", line
> 53, in 
> with open(whitelist_file, "r") as fh:
> FileNotFoundError: [Errno 2] No such file or directory:
> 'file_access_whitelist.txt'
> make[1]: Leaving directory '/home/crobinso/src/libvirt/builddir/tests'
> make: Leaving directory '/home/crobinso/src/libvirt/builddir'

Looks like this is unrelated to this patch - the original script
has the same flaw wrt VPATH builds, and indeed even the Makefile.am
is currently broken.

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Re: [libvirt] [PATCH v5 17/23] tests: rewrite file access checker in Python

2019-11-18 Thread Cole Robinson 
On 11/11/19 9:38 AM, Daniel P. Berrangé wrote:
> As part of an goal to eliminate Perl from libvirt build tools,
> rewrite the check-file-access.pl tool in Python.
> 
> This was a straight conversion, manually going line-by-line to
> change the syntax from Perl to Python. Thus the overall structure
> of the file and approach is the same.
> 
> Signed-off-by: Daniel P. Berrangé 
> ---
>  Makefile.am |   1 +
>  scripts/check-file-access.py| 123 +++
>  tests/Makefile.am   |   3 +-
>  tests/check-file-access.pl  | 126 
>  tests/file_access_whitelist.txt |   2 +-
>  5 files changed, 126 insertions(+), 129 deletions(-)
>  create mode 100755 scripts/check-file-access.py
>  delete mode 100755 tests/check-file-access.pl
> 

`make -C builddir check-access` fails like:

LC_ALL= LANG=C LC_CTYPE=en_US.UTF-8 /usr/bin/python3
/home/crobinso/src/libvirt/scripts/check-file-access.py | sort -u
Traceback (most recent call last):
  File "/home/crobinso/src/libvirt/scripts/check-file-access.py", line
53, in 
with open(whitelist_file, "r") as fh:
FileNotFoundError: [Errno 2] No such file or directory:
'file_access_whitelist.txt'
make[1]: Leaving directory '/home/crobinso/src/libvirt/builddir/tests'
make: Leaving directory '/home/crobinso/src/libvirt/builddir'

- Cole

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

[libvirt] [PATCH v5 17/23] tests: rewrite file access checker in Python

2019-11-11 Thread Daniel P . Berrangé 
As part of an goal to eliminate Perl from libvirt build tools,
rewrite the check-file-access.pl tool in Python.

This was a straight conversion, manually going line-by-line to
change the syntax from Perl to Python. Thus the overall structure
of the file and approach is the same.

Signed-off-by: Daniel P. Berrangé 
---
 Makefile.am |   1 +
 scripts/check-file-access.py| 123 +++
 tests/Makefile.am   |   3 +-
 tests/check-file-access.pl  | 126 
 tests/file_access_whitelist.txt |   2 +-
 5 files changed, 126 insertions(+), 129 deletions(-)
 create mode 100755 scripts/check-file-access.py
 delete mode 100755 tests/check-file-access.pl

diff --git a/Makefile.am b/Makefile.am
index 769cd4ce64..19114069e3 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -50,6 +50,7 @@ EXTRA_DIST = \
   scripts/check-aclrules.py \
   scripts/check-drivername.py \
   scripts/check-driverimpls.py \
+  scripts/check-file-access.py \
   scripts/check-remote-protocol.py \
   scripts/check-spacing.py \
   scripts/check-symfile.py \
diff --git a/scripts/check-file-access.py b/scripts/check-file-access.py
new file mode 100755
index 00..cdcbf2666f
--- /dev/null
+++ b/scripts/check-file-access.py
@@ -0,0 +1,123 @@
+#!/usr/bin/env python
+#
+# Copyright (C) 2016-2019 Red Hat, Inc.
+#
+# This library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+#
+# This library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this library.  If not, see
+# .
+#
+# This script is supposed to check test_file_access.txt file and
+# warn about file accesses outside our working tree.
+#
+#
+
+from __future__ import print_function
+
+import re
+import sys
+
+access_file = "test_file_access.txt"
+whitelist_file = "file_access_whitelist.txt"
+
+known_actions = ["open", "fopen", "access", "stat", "lstat", "connect"]
+
+files = []
+whitelist = []
+
+with open(access_file, "r") as fh:
+for line in fh:
+line = line.rstrip("\n")
+
+m = re.search(r'''^(\S*):\s*(\S*):\s*(\S*)(\s*:\s*(.*))?$''', line)
+if m is not None:
+rec = {
+"path": m.group(1),
+"action": m.group(2),
+"progname": m.group(3),
+"testname": m.group(5),
+}
+files.append(rec)
+else:
+raise Exception("Malformed line %s" % line)
+
+with open(whitelist_file, "r") as fh:
+for line in fh:
+line = line.rstrip("\n")
+
+if re.search(r'''^\s*#.*$''', line):
+continue  # comment
+if line == "":
+continue
+
+m = re.search(r'''^(\S*):\s*(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$''', line)
+if m is not None and m.group(2) in known_actions:
+# $path: $action: $progname: $testname
+rec = {
+"path": m.group(1),
+"action": m.group(3),
+"progname": m.group(4),
+"testname": m.group(6),
+}
+whitelist.append(rec)
+else:
+m = re.search(r'''^(\S*)(:\s*(\S*)(\s*:\s*(.*))?)?$''', line)
+if m is not None:
+# $path: $progname: $testname
+rec = {
+"path": m.group(1),
+"action": None,
+"progname": m.group(3),
+"testname": m.group(5),
+}
+whitelist.append(rec)
+else:
+raise Exception("Malformed line %s" % line)
+
+
+# Now we should check if %traces is included in $whitelist. For
+# now checking just keys is sufficient
+err = False
+for file in files:
+match = False
+
+for rule in whitelist:
+if not re.search("^" + rule["path"], file["path"]):
+continue
+
+if (rule["action"] is not None and
+not re.search("^" + rule["action"], file["action"])):
+continue
+
+if (rule["progname"] is not None and
+not re.search("^" + rule["progname"], file["progname"])):
+continue
+
+if (rule["testname"] is not None and
+file["testname"] is not None and
+not re.search("^" + rule["testname"], file["testname"])):
+continue
+
+match = True
+
+if not match:
+err = True
+print("%s: %s: %s" %
+  (file["path"], file["action"], file["progname"]),
+