On 07/25/2013 06:21 PM, Joseph S. Myers wrote:
> I was interested to watch the video of the DejaGnu BOF at the Cauldron. A
> few issues with DejaGnu for toolchain testing that I've noted but I don't
> think were covered there include:
Thanks for the thoughtful comments, they're useful as I start
considering refactoring DejaGnu to keep it working for the next 22
years... There is a lot of crusty old code in DejaGnu, I admit it.
DejaGnu was never truly designed, it was just built and debugged while
we were using it, and it shows.
I'm not sure if this discussion is better on the GCC list or the
DejaGnu list, but I would like to keep this thread going. Course GCC
developers are the main users of DejaGnu anyway.
> * DejaGnu has a lot of hardcoded logic to try to find various files in a
> toolchain build directory. A lot of it is actually for very old toolchain
> versions (using GCC version 2 or older, for example). The first issue
> with this is that it doesn't belong in DejaGnu: the toolchain should be
> free to rearrange its build directories without needing changes to DejaGnu
> itself (which in practice means there's lots of such logic in the
> toolchain's own testsuites *as well*, duplicating the DejaGnu code to a
> greater or lesser extent). The second issue is that "make install"
DejaGnu is a testing framework, so it makes sense that much of the GCC
testing logic is in gcc/testsuite/{lib,config}. It was also a decision
at the time that having a testsuite override existing procs in DejaGnu
core was a good idea. Now many years later, I think I'd move most what
GCC needs into the core, especially all the "dg* style of tests.
At one time the thought was DejaGnu was a general purpose test
framework, but I think at this point in time, it's really just used for
toolchain testing. (although my Gnash project also uses it) So I think
tweaking DejaGnu core to be mainly toolchain testing oriented is
probably a good idea.
> * Similarly, DejaGnu has hardcoded prune_warnings - and again GCC adds
> lots of its own prunes; it's not clear hardcoding this in DejaGnu is a
> particularly good idea either.
The DejaGnu pruning is older than GCC's. :-)
> * Another piece of unfortunate hardcoding in DejaGnu is how remote-host
> testing uses "-o a.out" when running tools on the remote host - such a
> difference from how they are run on a local host results in lots of issue
This is historical, a.out being common at the time.
> * A key feature of QMTest that I like but I don't think got mentioned is
> that you can *statically enumerate the set of tests* without running them.
> That is, a testsuite has a well-defined set of tests, and that set does
> not depend on what the results of the tests are - whereas it's very easy
> and common for a DejaGnu test to have test names (the text after PASS: or
> FAIL: ) depending on whether the test passed or failed, or how the test
> passed or failed (no doubt the testsuite authors had reasons for doing
> this, but it conflicts with any automatic comparison of results). The
One of my other ideas for DejaGnu 2.0 is improved test result output.
I'm currently importing all test results into a database (see the mysql
branch on savannah), and find text parsing painful and lacking more fine
grained details. The text field for PASS/FAIL is overloaded. Since I
want to improve the ability to analyze results, ie... comparing what
happens with differing configure or command line options, I think the
output format has to change. One thought is to only add new fields into
the --xml output, as that is database specific, and leave the current
text output unchanged.
> * People in the BOF seemed happy with expect. I think expect has caused
> quite a few problems for toolchain testing. In particular, there are or
I don't think it was that we were happy with expect, but at least for
GDB testing, nobody has any alternatives. I thought I mentioned that a
refactored DejaGnu would only use expect for GDB testing, everything
else wouldn't require it. That also means all the remote execution procs
would need to work without expect as well.
- rob -
