On 8/29/25 04:40, Richard Earnshaw wrote:
On 29/08/2025 10:36, Iain Sandoe wrote:
On 29 Aug 2025, at 10:32, Richard Earnshaw <richard.earns...@arm.com>
wrote:
On 29/08/2025 04:08, Jacob Bachmeyer wrote:
[...]
The problem with detecting duplicate names in the DejaGnu framework
is that it would add memory overhead that scales with the number of
tests and DejaGnu tries to avoid that kind of unbounded space
requirement. (OK, it *is* bounded for any finite testsuite, but the
idea of a steadily growing memory footprint during a test run still
bothers me.)
I suggest that the comparison script GCC uses is probably the best
place to check for duplicate test names, since that seems to also
be the script that can be confused by them.
That's exactly what I was suggesting. Trying to do it in dejagnu
would be a nightmare given that we run multiple instances of it to
get parallel testing.
For the record, I’ve now proposed a BoF for the cauldron on
‘improving the raw output’ from the testsuite, since that seems at
least one place we can make progress - by making the input to the
post-processing tools more machine-friendly. Hopefully the concerns
and ideas from this thread can contribute there.
I don't know how much maintenance DejaGNU is getting these days, but
certainly something like a --json option to runtest that caused the
.sum files to be written as structured data would make some
post-processing significantly easier
There is an option for XML logs that is currently a bit "half-baked"
and, as far as I can tell, what it currently tries to do is not actually
possible with Expect. (There seems to be no good way to reliably
distinguish text to and from child processes in the log. The fatal flaw
is that text "to" a child is typically echoed by the pty driver and
therefore is actually logged as "from" the child.)
The "least ridiculous" option may be to divide the log into per-test
segments with result tags as segment delimiters---XML syntactic sugar
atop the plain text log. Removing XML log support entirely and only
using XML for a structured summary might be a better option. Annotating
text sent by DejaGnu before it appears as echoed by the pty driver is
another option that could be a legitimate use of XML.
There are plans to radically overhaul XML support for the DejaGnu 1.7
release. There is some preliminary code on the "psql" branch but I am
unsure if that code will ultimately be merged or prove to be a dead
end. The revised XML support might simply provide an XML summary file
alongside the plain log and summary files.
The DejaGnu 1.6.3 release introduced the first elements of support for
test groups. In 1.6.3, all that the {testcase group ...} commands do is
validate group nesting and report the current group if [testcase group]
is used. The framework will eventually implicitly group tests; explicit
groups will nest among the implicit groups. In 1.7, the new XML output
will record groups as XML container tags in the XML log and/or summary.
All that said, once the infrastructure for the revised XML support is
complete, reusing it to also produce .sum.json files would be relatively
simple, really only a matter of output syntax. (Note that that would
mean the resulting JSON structure would mirror the XML tree structure.)
DejaGnu maintenance in general is: bugs get fixed quickly in Git
master, feature advances occur as I collect the right "round tuits", and
there is a roadmap for features to land in at least the next few
releases but no timeline planned for when those releases will be made.
For example, the currently planned major highlight for 1.6.4 will be a
rewritten table-driven default_target_compile because that procedure is
teetering on the edge of "BigBallOfMud" unmaintainability.
There are also plans for (eventual) native parallel testing support, but
these are far off at the moment and will require major refactoring of
how DejaGnu runs tests. Expect limitations mean while that parallel
testing support is possible, implementing it reliably will not be easy.
(A first experimental attempt has already foundered on limitations of
Expect.)
The refactoring required for parallel testing will not affect
well-behaved testsuites, and any testsuites it does affect will be
unavoidably broken if run in parallel anyway. Using slave interpreters
to isolate test cases has been in the TODO file for a very long time.
(I guess probably since Tcl introduced the feature...)
For now, the best option for parallel testing is to run parallel
instances of DejaGnu on different subsets of the testsuite.
I should probably also give advance notice that if you are interested in
this, you should be subscribed to the DejaGnu mailing list
(<deja...@gnu.org>). It is generally low traffic, and there are some
currently undocumented dubious behaviors that will be changed in the
future. Such changes will be announced on that list to check if anyone
is actually depending on the current behavior.
For example, the (undocumented) procedure findfile can return a
directory, despite its name. (Yes, directories *are* technically files
in POSIX...) This is likely to change in the future.
-- Jacob
