Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers
On Mon, 03/12 18:17, Marc-André Lureau wrote: > Hi > > On Mon, Mar 12, 2018 at 6:11 PM, Fam Zheng wrote: > > On Mon, 03/12 15:11, Marc-André Lureau wrote: > >> Hi Fam, > >> > >> It would be great if patchew could run the debug/sanitizer build. > >> Unfortunately, except x86 target, the run is full of ASAN leaks or > >> warnings. > >> > >> Do you think patchew could learn to do regression testing? By that I > >> mean that patchew could compare a patchset output with the current > >> master (that output should be cached for multiple tests). The output > >> to compare here would be the errors reported by ASAN during the build. > >> (eventually, this could be used for other metrics some day). If so, > >> would you be willing to help me implementing it? > > > > The question is can we simply rely on the exit code of cmp or diff for that > > kind > > of comparison? If there is any out-of-order texts, that simple diff won't > > work. > > If the tests are run sequentially, i think it should produce the same > output when run multiple times (to be verified). In that case the test will last for a long time. Currently patchew isn't very good as multi-tasking. But that is one option we can explore (like we can devote one VM for asan test). > > However, if there are leaks already, and the address of allocation or > symbols change, this may produce diff on existing lines. Maybe we can come up with a 'make docker-test-debug-filtered@fedora' or a relatively simple 'make docker-test-debug@fedora | sed -e "..."' command that can be fed to diff. > > What I would like to see is the diff of added (or removed) lines. > > I suppose that should be possible, I'll do some experiments. > > > Anything more intelligent/sensible should go as a policy, and patchew should > > continue focus on mechanisms. > > hmm ok :) Like said above, I think it's possible to postprocess the output to make it easy to diff/cmp. By mechanism/policy I mean the place to draw a line to code Patchew. However patchew has a database where we can write test script snippets (as configurations) that is "the policy". Fam
Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers
Hi On Mon, Mar 12, 2018 at 6:11 PM, Fam Zheng wrote: > On Mon, 03/12 15:11, Marc-André Lureau wrote: >> Hi Fam, >> >> It would be great if patchew could run the debug/sanitizer build. >> Unfortunately, except x86 target, the run is full of ASAN leaks or >> warnings. >> >> Do you think patchew could learn to do regression testing? By that I >> mean that patchew could compare a patchset output with the current >> master (that output should be cached for multiple tests). The output >> to compare here would be the errors reported by ASAN during the build. >> (eventually, this could be used for other metrics some day). If so, >> would you be willing to help me implementing it? > > The question is can we simply rely on the exit code of cmp or diff for that > kind > of comparison? If there is any out-of-order texts, that simple diff won't > work. If the tests are run sequentially, i think it should produce the same output when run multiple times (to be verified). However, if there are leaks already, and the address of allocation or symbols change, this may produce diff on existing lines. What I would like to see is the diff of added (or removed) lines. I suppose that should be possible, I'll do some experiments. > Anything more intelligent/sensible should go as a policy, and patchew should > continue focus on mechanisms. hmm ok :) > I'll send a pull for the patch. Thanks! > thanks -- Marc-André Lureau
Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers
On Mon, 03/12 15:11, Marc-André Lureau wrote: > Hi Fam, > > It would be great if patchew could run the debug/sanitizer build. > Unfortunately, except x86 target, the run is full of ASAN leaks or > warnings. > > Do you think patchew could learn to do regression testing? By that I > mean that patchew could compare a patchset output with the current > master (that output should be cached for multiple tests). The output > to compare here would be the errors reported by ASAN during the build. > (eventually, this could be used for other metrics some day). If so, > would you be willing to help me implementing it? The question is can we simply rely on the exit code of cmp or diff for that kind of comparison? If there is any out-of-order texts, that simple diff won't work. Anything more intelligent/sensible should go as a policy, and patchew should continue focus on mechanisms. I'll send a pull for the patch. Thanks! Thanks.
Re: [Qemu-devel] [PATCH] tests: make docker-test-debug@fedora run sanitizers
Hi Fam, It would be great if patchew could run the debug/sanitizer build. Unfortunately, except x86 target, the run is full of ASAN leaks or warnings. Do you think patchew could learn to do regression testing? By that I mean that patchew could compare a patchset output with the current master (that output should be cached for multiple tests). The output to compare here would be the errors reported by ASAN during the build. (eventually, this could be used for other metrics some day). If so, would you be willing to help me implementing it? On Mon, Mar 12, 2018 at 1:08 PM, Marc-André Lureau wrote: > Since --enable-debug no longer enable sanitizers, we need explicit > --enable-sanitizers. > > llvm package is required for llvm-symbolizer, to get symbols in > backtraces. > > Add make V=1 to get details about failing tests. > > Signed-off-by: Marc-André Lureau > --- > tests/docker/dockerfiles/fedora.docker | 2 +- > tests/docker/test-debug| 6 +++--- > 2 files changed, 4 insertions(+), 4 deletions(-) > > diff --git a/tests/docker/dockerfiles/fedora.docker > b/tests/docker/dockerfiles/fedora.docker > index 994a35a332..a4319e 100644 > --- a/tests/docker/dockerfiles/fedora.docker > +++ b/tests/docker/dockerfiles/fedora.docker > @@ -2,7 +2,7 @@ FROM fedora:27 > ENV PACKAGES \ > ccache gettext git tar PyYAML sparse flex bison python3 bzip2 hostname \ > glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel \ > -gcc gcc-c++ clang make perl which bc findutils libaio-devel \ > +gcc gcc-c++ llvm clang make perl which bc findutils libaio-devel \ > nettle-devel libasan libubsan \ > mingw32-pixman mingw32-glib2 mingw32-gmp mingw32-SDL mingw32-pkg-config \ > mingw32-gtk2 mingw32-gtk3 mingw32-gnutls mingw32-nettle mingw32-libtasn1 > \ > diff --git a/tests/docker/test-debug b/tests/docker/test-debug > index d020b06917..d3f9f70d01 100755 > --- a/tests/docker/test-debug > +++ b/tests/docker/test-debug > @@ -1,6 +1,6 @@ > #!/bin/bash -e > # > -# Compile and check with clang & --enable-debug. > +# Compile and check with clang & --enable-debug --enable-sanitizers. > # > # Copyright (c) 2016-2018 Red Hat Inc. > # > @@ -19,8 +19,8 @@ requires clang asan > cd "$BUILD_DIR" > > OPTS="--cxx=clang++ --cc=clang --host-cc=clang" > -OPTS="--enable-debug $OPTS" > +OPTS="--enable-debug --enable-sanitizers $OPTS" > > build_qemu $OPTS > -make $MAKEFLAGS check > +make $MAKEFLAGS V=1 check > install_qemu > -- > 2.16.2.346.g9779355e34 > > -- Marc-André Lureau