This at least seems reasonable, although I haven't thought about it long enough to commit to too strongly :-). I *used* to be able to find what I would need (with enough digging) in .testing-results/blah-blah, but now there are only two files in there, results.xml and results.pickle. Did this break recently?
Gabe On Mon, Aug 24, 2020 at 1:43 AM Bobby Bruce <[email protected]> wrote: > I agree the TestLib framework is broken when it comes to reporting output, > which makes problems pretty difficult to figure things out. As far as I > know, it's been this way for some time. My workflow when i get a failing > test is to use `./main.py run --uid <UID>` on the failing test suite which, > when run (at least on my machine), will print the location of the `simout` > and `simerr` files (not deleted). > > E.g.: > > ``` > $ ./main.py run --uid > SuiteUID:tests/gem5/hello_se/test_hello_se.py:test-hello-linux-TimingSimpleCPU-RISCV-x86_64-opt > Running the new gem5 testing script. > For more information see TESTING.md. > To see details as the testing scripts are running, use the option -v, -vv, > or -vvv > Discovered 198 tests and 99 suites in > /home/bobbyrbruce/Documents/gem5/tests/gem5/hello_se/test_hello_se.py > > ======================================================================================================= > Running Tests from 1 suites > Results will be stored in > /home/bobbyrbruce/Documents/gem5/tests/.testing-results > > ======================================================================================================= > Building the following targets. This may take a while. > /home/bobbyrbruce/Documents/gem5/build/RISCV/gem5.opt > You may want to run with only a single ISA(--isa=), use --skip-build, or > use 'rerun'. > warn: CheckedInt already exists in allParams. This may be caused by the > Python 2.7 compatibility layer. > warn: Enum already exists in allParams. This may be caused by the Python > 2.7 compatibility layer. > warn: ScopedEnum already exists in allParams. This may be caused by the > Python 2.7 compatibility layer. > warn: CheckedInt already exists in allParams. This may be caused by the > Python 2.7 compatibility layer. > warn: Enum already exists in allParams. This may be caused by the Python > 2.7 compatibility layer. > warn: ScopedEnum already exists in allParams. This may be caused by the > Python 2.7 compatibility layer. > Redirecting stdout to /tmp/gem5outaiXmNQ/simout > Redirecting stderr to /tmp/gem5outaiXmNQ/simerr > Test: test-hello-linux-TimingSimpleCPU-RISCV-x86_64-opt Passed > Test: test-hello-linux-TimingSimpleCPU-RISCV-x86_64-opt-MatchStdoutNoPerf > Passed > ============================== Results: 2 Passed in 9.7e+02 seconds > ================================== > ``` > > I'd recommend three fixes for the TestLib: > > 1) `.testing-results` shouldn't be a hidden directory. > 2) Stdout and stderr should be output to the `testing-results/results.xml` > file. > 3) For the `MatchStdoutNoPerf` tests (where the stdout is compared against > some reference), the diff should be recorded upon failure. The > `tests/gem5/verifier.py` script contains most of the functionality for > these `MatchStdoutNoPerf` tests. > > The TestLib code (in `ext/testlib`) could definitely use a cleanup. I > think, frankly, it's way over-engineered given all we really want to do is > compile different gem5 ISAs and run them with different configurations. The > testing-results functionality is mostly handled by` > ext/testlib/results.py,` though the methods here are clearly broken or not > being called when they are supposed to be. If you're looking for something > to do, I'd recommend starting there. > > Kind regards, > Bobby > -- > Dr. Bobby R. Bruce > Room 2235, > Kemper Hall, UC Davis > Davis, > CA, 95616 > > web: https://www.bobbybruce.net > > > On Sun, Aug 23, 2020 at 8:10 PM Gabe Black <[email protected]> wrote: > >> Adding Bobby and Giacomo specifically, since I see their fingerprints on >> the testing stuff. >> >> Gabe >> >> On Sun, Aug 23, 2020 at 8:03 PM Gabe Black <[email protected]> wrote: >> >>> At the risk of sounding over dramatic, this is a major problem. Before >>> it was difficult to see what happened when a test failed when running the >>> test automation, but now it's impossible. As far as I can tell, the results >>> end up in a temp directory and are then wiped out and not copied anywhere. >>> I can't even determine what the test ran to run myself and see what >>> happened. >>> >>> What code is supposed to copy the results out? I'm willing to help >>> figure this out, but I don't even know where to start. >>> >>> Gabe >>> >>> On Sun, Aug 23, 2020 at 6:00 PM Gabe Black <[email protected]> wrote: >>> >>>> Hey folks, I'm trying to run tests on a change I'm working on, and the >>>> test infrastructure is not actually saving any test results where it claims >>>> it will. I also tried selecting what ISAs to use, and after selecting all >>>> of them the run script claimed no tests would be run. >>>> >>>> Is this something other people are seeing that's just generally broken, >>>> or is there something out of whack on my machine? >>>> >>>> Gabe >>>> >>>
_______________________________________________ gem5-dev mailing list -- [email protected] To unsubscribe send an email to [email protected] %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
