A couple thoughts I have are to reduce the enormous directory names the test results go in (when that works again), and also to make it easy to get the relevant details from a failure. I think I've said this somewhere else, but I think the debugging loop might look like:
1. Run tests. 2. See a failing test. 3. Easily see what failed and how to reproduce the failure with information printed alongside the fact that it failed. 4. Try to fix things, possibly without using the test framework. It might be necessary in some cases, but it gets in the way of other tools like gdb, etc. 5. Rerun that test, go to 3 or 1 depending on the results. Right now it's (as you said) very unclear what failed when something fails, where the artifacts are like binaries, logs, etc., and also how to reproduce the failure so you can debug. I also don't like how it scans for tests since it will potentially run into all kinds of random things. If you run it from the root of a checkout and don't tell it to scan in tests only, it will currently fail because it finds some random thing from somewhere that looks like a test but isn't. This isn't a great user experience, and it took me a little while to figure out why it was upset. Gabe On Mon, Aug 24, 2020 at 2:03 AM Gabe Black <[email protected]> wrote: > This at least seems reasonable, although I haven't thought about it long > enough to commit to too strongly :-). I *used* to be able to find what I > would need (with enough digging) in .testing-results/blah-blah, but now > there are only two files in there, results.xml and results.pickle. Did this > break recently? > > Gabe > > On Mon, Aug 24, 2020 at 1:43 AM Bobby Bruce <[email protected]> wrote: > >> I agree the TestLib framework is broken when it comes to reporting >> output, which makes problems pretty difficult to figure things out. As far >> as I know, it's been this way for some time. My workflow when i get a >> failing test is to use `./main.py run --uid <UID>` on the failing test >> suite which, when run (at least on my machine), will print the location of >> the `simout` and `simerr` files (not deleted). >> >> E.g.: >> >> ``` >> $ ./main.py run --uid >> SuiteUID:tests/gem5/hello_se/test_hello_se.py:test-hello-linux-TimingSimpleCPU-RISCV-x86_64-opt >> Running the new gem5 testing script. >> For more information see TESTING.md. >> To see details as the testing scripts are running, use the option -v, >> -vv, or -vvv >> Discovered 198 tests and 99 suites in >> /home/bobbyrbruce/Documents/gem5/tests/gem5/hello_se/test_hello_se.py >> >> ======================================================================================================= >> Running Tests from 1 suites >> Results will be stored in >> /home/bobbyrbruce/Documents/gem5/tests/.testing-results >> >> ======================================================================================================= >> Building the following targets. This may take a while. >> /home/bobbyrbruce/Documents/gem5/build/RISCV/gem5.opt >> You may want to run with only a single ISA(--isa=), use --skip-build, or >> use 'rerun'. >> warn: CheckedInt already exists in allParams. This may be caused by the >> Python 2.7 compatibility layer. >> warn: Enum already exists in allParams. This may be caused by the Python >> 2.7 compatibility layer. >> warn: ScopedEnum already exists in allParams. This may be caused by the >> Python 2.7 compatibility layer. >> warn: CheckedInt already exists in allParams. This may be caused by the >> Python 2.7 compatibility layer. >> warn: Enum already exists in allParams. This may be caused by the Python >> 2.7 compatibility layer. >> warn: ScopedEnum already exists in allParams. This may be caused by the >> Python 2.7 compatibility layer. >> Redirecting stdout to /tmp/gem5outaiXmNQ/simout >> Redirecting stderr to /tmp/gem5outaiXmNQ/simerr >> Test: test-hello-linux-TimingSimpleCPU-RISCV-x86_64-opt Passed >> Test: test-hello-linux-TimingSimpleCPU-RISCV-x86_64-opt-MatchStdoutNoPerf >> Passed >> ============================== Results: 2 Passed in 9.7e+02 seconds >> ================================== >> ``` >> >> I'd recommend three fixes for the TestLib: >> >> 1) `.testing-results` shouldn't be a hidden directory. >> 2) Stdout and stderr should be output to the >> `testing-results/results.xml` file. >> 3) For the `MatchStdoutNoPerf` tests (where the stdout is compared >> against some reference), the diff should be recorded upon failure. The >> `tests/gem5/verifier.py` script contains most of the functionality for >> these `MatchStdoutNoPerf` tests. >> >> The TestLib code (in `ext/testlib`) could definitely use a cleanup. I >> think, frankly, it's way over-engineered given all we really want to do is >> compile different gem5 ISAs and run them with different configurations. The >> testing-results functionality is mostly handled by` >> ext/testlib/results.py,` though the methods here are clearly broken or not >> being called when they are supposed to be. If you're looking for something >> to do, I'd recommend starting there. >> >> Kind regards, >> Bobby >> -- >> Dr. Bobby R. Bruce >> Room 2235, >> Kemper Hall, UC Davis >> Davis, >> CA, 95616 >> >> web: https://www.bobbybruce.net >> >> >> On Sun, Aug 23, 2020 at 8:10 PM Gabe Black <[email protected]> wrote: >> >>> Adding Bobby and Giacomo specifically, since I see their fingerprints on >>> the testing stuff. >>> >>> Gabe >>> >>> On Sun, Aug 23, 2020 at 8:03 PM Gabe Black <[email protected]> wrote: >>> >>>> At the risk of sounding over dramatic, this is a major problem. Before >>>> it was difficult to see what happened when a test failed when running the >>>> test automation, but now it's impossible. As far as I can tell, the results >>>> end up in a temp directory and are then wiped out and not copied anywhere. >>>> I can't even determine what the test ran to run myself and see what >>>> happened. >>>> >>>> What code is supposed to copy the results out? I'm willing to help >>>> figure this out, but I don't even know where to start. >>>> >>>> Gabe >>>> >>>> On Sun, Aug 23, 2020 at 6:00 PM Gabe Black <[email protected]> >>>> wrote: >>>> >>>>> Hey folks, I'm trying to run tests on a change I'm working on, and the >>>>> test infrastructure is not actually saving any test results where it >>>>> claims >>>>> it will. I also tried selecting what ISAs to use, and after selecting all >>>>> of them the run script claimed no tests would be run. >>>>> >>>>> Is this something other people are seeing that's just generally >>>>> broken, or is there something out of whack on my machine? >>>>> >>>>> Gabe >>>>> >>>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
