Sounds good. We can give it a week or so before we push. I won't have any time to work on much between now and then anyway.
Cheers, Jason On Fri, Aug 3, 2018 at 2:50 AM Andreas Sandberg <[email protected]> wrote: > OK, that sounds reasonable. We shouldn't have any issues with that. > > I'd like someone from us to have a look as well, most likely Giacomo since > I won't be able to find the time myself. However, don't let that block > anything. Last time I looked at the code, it looked reasonable. I'm sure we > can fix any issues once we start integrating it into our test setup. > > Thanks, > Andreas > > On 03/08/2018 03:26, Sean Wilson wrote: > > My attempt to address your concerns, Andreas.. > > > Your issue with the baked-in test server was definitely reasonable. I've > actually taken that part out of the library since it was so unreliable. A > reliable test server would probably be a whole library in itself. > > My suggestion for running distributed tests would be to do basically the > same thing you do now: list all the suites and kick off a distributed task > for each suite uid. Each test is individually labeled with a UID. When > using the `./main.py run` command you can supply the test suite uid with > `--uid <SUITE-UID>`. That test and that test only will run. > > Your current xml aggregation solution sounds reasonable to me. The only > issue you might run into is that a single result.xml is written to, so if > all of gem5 is served over NFS you might get conflicts writing to the file. > I can add an output path flag if you think this might be an issue for you. > > > Cheers, > Sean > > On Thu, Aug 2, 2018 at 6:21 PM, Jason Lowe-Power <[email protected]> > wrote: > >> Hi all, >> >> Some more follow up. Sorry that this is going so slowly. >> >> Sean just did a pretty major reworking of the main library. See >> https://gem5-review.googlesource.com/4421. I'll let him address >> Andreas's questions below. I think we're pretty close to having all three. >> >> I would really like to get this pushed in ASAP so someone (me, a student, >> or others in the community) can start migrating all of the current tests >> over to the new framework. >> >> As a quick preview, this is what the output looks like now. >> >> > cd tests >> > ./main.py run >> Running the new gem5 testing script. >> For more information see TESTING.md. >> To see details as the testing scripts are running, use the option -v, >> -vv, or -vvv >> >> ====================================================================================================================== >> Loading Tests >> Discovered 0 tests and 0 suites in /home/jlp/Code/gem5/gem5/tests/tests.py >> Discovered 30 tests and 15 suites in >> /home/jlp/Code/gem5/gem5/tests/gem5/hello_se/test_hello_se.py >> Discovered 6 tests and 3 suites in >> /home/jlp/Code/gem5/gem5/tests/gem5/m5_util/test_exit.py >> Discovered 21 tests and 21 suites in >> /home/jlp/Code/gem5/gem5/tests/gem5/test_build/test_build.py >> ... <a few errors while we are transitioning> >> >> ====================================================================================================================== >> Running Tests from 9 suites >> Results will be stored in /home/jlp/Code/gem5/gem5/tests/.testing-results >> >> ====================================================================================================================== >> Building the following targets. This may take a while. >> /home/jlp/Code/gem5/gem5/build/RISCV/gem5.opt, >> /home/jlp/Code/gem5/gem5/build/X86/gem5.opt, >> /home/jlp/Code/gem5/gem5/build/ARM/gem5.opt >> You may want to run with only a single ISA(--isa=), use --skip-build, or >> use 'retest'. >> Test: testhello64-static-X86-opt Passed >> Test: testhello64-static-X86-opt-MatchStdoutNoPerf Passed >> Test: testhello64-dynamic-X86-opt Passed >> Test: testhello64-dynamic-X86-opt-MatchStdoutNoPerf Passed >> Test: testhello32-static-X86-opt Passed >> Test: testhello32-static-X86-opt-MatchStdoutNoPerf Passed >> Test: testhello64-static-ARM-opt Passed >> Test: testhello64-static-ARM-opt-MatchStdoutNoPerf Passed >> Test: testhello32-static-ARM-opt Passed >> Test: testhello32-static-ARM-opt-MatchStdoutNoPerf Passed >> Test: m5_exit_test-X86-opt Passed >> Test: m5_exit_test-X86-opt-MatchRegex Passed >> Test: build-X86-opt Passed >> Test: build-RISCV-opt Passed >> Test: build-ARM-opt Passed >> ====================================== Results: 15 Passed in 2.6e+01 >> seconds ======================================== >> >> Cheers, >> Jason >> >> On Fri, Jul 13, 2018 at 1:37 AM Andreas Sandberg < >> [email protected]> wrote: >> >>> Hi Jason, >>> >>> Thanks for doing this. I haven't had a chance to look at the code in >>> details myself, but from what I have seen it looks like a real improvement. >>> Ideally, I'd like someone from our side (probably Giacomo T) to have a look >>> at it. However, Giacomo is currently travelling and I have other pressing >>> matters to tend to. :( >>> >>> I have had a quick look at the code and it all looks reasonable. There >>> are three things we need for CI integration: >>> >>> 1. Ability to separate build and test. >>> 2. Ability to schedule individual (long-running) tests separately in >>> a cluster environment. >>> 3. Ability to get a JUnit XML file with per-test results. >>> >>> >>> With the potential exception of 2, I think the new framework supports >>> all of the above. We could, maybe, use the support for distributed test >>> using the test server in the framework, but I'm a bit reluctant to deploy >>> it for two reasons. 1) It requires credentials to be hard-coded. It would >>> be much more convenient to use an auto-generated authentication file (with >>> suitable permissions) in a shared file location. 2) Having a custom server >>> is potentially very unreliable since the network tends to be noisy (e.g., >>> port scanners, IDS software). The static allocation of a port will also be >>> troublesome (e.g., when two servers run on the same node). >>> >>> In the current framework, we typically list the available tests and then >>> create one cluster job per test. Once the tests complete, the CI framework >>> aggregates all of the JUnit files (it also stores the pickle files for >>> debugging) and generates a report. I'd like to be able to do something with >>> the new framework. Is that possible? >>> >>> Cheers, >>> Andreas >>> >>> On 05/07/2018 21:22, Jason Lowe-Power wrote: >>> >>> Hi all, >>> >>> I'm hoping that we can finalize the new testing framework, and I can push >>> it in within the next week or so. All of the important changes have at >>> least one +2, but I want to give everyone one last chance to object or >>> suggest changes. I've been talking to Sean, and while there's more that >>> could be cleaned up in ext/testlib, we think it's good as it is. At a >>> minimum, this will give us a starting point. I'd like to get this in soon >>> so I can start adding more tests to it in the coming weeks. >>> >>> The patches are linked below: >>> https://gem5-review.googlesource.com/c/public/gem5/+/4880/7https://gem5-review.googlesource.com/c/public/gem5/+/4881/7https://gem5-review.googlesource.com/c/public/gem5/+/4421/12https://gem5-review.googlesource.com/c/public/gem5/+/4882/13https://gem5-review.googlesource.com/c/public/gem5/+/4883/13https://gem5-review.googlesource.com/c/public/gem5/+/4422/13https://gem5-review.googlesource.com/c/public/gem5/+/4423/13https://gem5-review.googlesource.com/c/public/gem5/+/10121/4 >>> >>> Thanks, >>> Jason >>> _______________________________________________ >>> gem5-dev mailing >>> [email protected]http://m5sim.org/mailman/listinfo/gem5-dev >>> >>> >>> IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended >>> recipient, please notify the sender immediately and do not disclose the >>> contents to any other person, use it for any purpose, or store or copy the >>> information in any medium. Thank you. >>> >> > > IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
