Pedro Alves wrote:
On 6/26/20 11:58 PM, Jacob Bachmeyer wrote:
The issue for me is that the "gdb Summary" tally did not
mention the aborted testcase. I would like to see something
like this instead:
# of expected passes 208
# of nasty OMG FIX! tcl errors 1
:-)
Your UNRESOLVED is of course better than the status quo.
UNRESOLVED is a step in the right direction, but I agree that more is needed. We are
somewhat limited by POSIX, which does not define a separate result type for "test
case crashed" and seems to roll that into UNRESOLVED, although POSIX appears to
require a testsuite to have a strictly-defined set of tests and that a run must produce
results for exactly all of them, which requires information DejaGnu does not normally
have about the running testsuite. So, while UNRESOLVED is needed, we could also maintain
a separate count of UNRESOLVED-due-to-crash, or simply store away the error
message/errorCode/errorInfo tuples and emit them a second time at the end of the run,
also easy in Tcl 8.
I'm not sure how much is POSIX compliance important these days. I would hope
that that
would not limit DejaGnu's development and limit its potential to be more usable.
We already have extensions; POSIX only allowed
PASS/FAIL/UNTESTED/UNRESOLVED/UNSUPPORTED. [KX]{PASS,FAIL} are GNU
extensions.
Even
if POSIX compliance is important, I would think that that could be put behind
a --posixly_correct option, and/or POSIXLY_CORRECT environment variable, like
other
GNU tools do:
https://www.gnu.org/prep/standards/html_node/Non_002dGNU-Standards.html
Or conversely, we could have a --gnu mode, though I would say that a GNU tool
should behave in GNU mode by default.
POSIX compliance in DejaGnu depended on the testsuite being run. Any
results generated by the framework itself need to be in the POSIX set,
though testsuites can use extensions, as GDB has done. Actually adding
more test result codes is dubious, as that will interact badly with XML
output and efficient database processing, both of which can reasonably
enumerate the nine result codes. ({,K,X}{PASS,FAIL},
UN{TEST,RESOLV,SUPPORT}ED)
I really don't buy that adding different result types is an issue, as
hinted earlier. We've actually done it in GDB, for a loooong while. We
have distinct XFAIL vs KFAIL output messages show up in the .sum file (though
those aren't distinguished in the summary file, doing so would be convenient,
hence I see that as a GDB testsuite bug).
And just very recently, we've added PATH and DUPLICATE output messages,
and those do show up in the summary:
# of expected passes 89593
# of unexpected failures 547
# of expected failures 146
# of known failures 104
# of untested testcases 20
# of unresolved testcases 5
# of unsupported tests 97
# of paths in test names 1 <<<<<
# of duplicate test names 370 <<<<<
I would think that even these two would be useful to have in
DejaGnu proper for all testsuites.
v1/rationale -
https://sourceware.org/pipermail/gdb-patches/2020-April/167847.html
v3/final version -
https://sourceware.org/pipermail/gdb-patches/2020-April/168105.html
I will look at the exact details later, but this sounds like it is
probably a monkeypatch that will stop working in some future version
when the procedures you are overriding disappear into an internal
namespace, so we need to be thinking about what would be needed to
provide extensible summary line sets, just as we need some way to extend
language support in default_target_compile without overriding the entire
procedure.
-- Jacob
_______________________________________________
Bug-dejagnu mailing list
Bug-dejagnu@gnu.org
https://lists.gnu.org/mailman/listinfo/bug-dejagnu
