More problems in paradise. It turns out, on a full regression run, that the particular changeset causes a massive increase in the run-time (30%) and instruction count for build/ARM/tests/opt/long/fs/10.linux-boot/arm/linux/realview64-o3 and build/ARM/tests/opt/long/fs/10.linux-boot/arm/linux/realview64-o3-checker. I suspect some interrupt/quiesce oddity, or even bug. Note that these are already some of our longest running regressions (3+ hours), and this change just added 1 hour to each of them.
I am not sure what to suggest. The run-time increase is troublesome, the fact that such a small change suddenly adds 30% to the instruction count of a single-core regression is even more alarming. Joel, any ideas? Anyone else? Andreas On 27/08/2015 22:12, "gem5-dev on behalf of Andreas Hansson" <[email protected] on behalf of [email protected]> wrote: >Hi Joel, > >I always run util/regress with ‘all'. This will build all ISAs and run all >tests. You shouldn’t need anything else imho. > >Preferably make sure you have protobuf so that the tgen tests are run as >well. Eio I agree we should probably nuke, although that’s a separate >discussion. > >I would suggest the stats update should always be pushed at the same time >as the change that would otherwise break the regressions. Anytime someone >does a pull all regressions should pass. As simple as that. > >The challenge with distributing test binaries is licenses, and sheer size. >I agree it would be good to have something openly available, and have >precompiled binaries in a separate repository or a tarball. Any >suggestions are welcome. Until we have something to replace the current >ones we need to stick to what we have... > >For the FS ones you don’t have to modify anything, just set M5_PATH. > >Andreas > > >On 27/08/2015 20:10, "gem5-dev on behalf of Joel Hestness" ><[email protected] on behalf of [email protected]> wrote: > >>> On zizzer you should be able to run it without any issues. For systems >>> other than zizzer, the FS regressions need the FS files from the >>>download >>> section on the wiki. As you point out, the cpu2000 tests rely on >>>binaries >>> we cannot distribute :-(. >>> >> >>I see. A big part of my misunderstanding then is that I've never had >>access >>to Zizzer, though I've had gem5 commit access since 2011. Is there a >>process in place for getting access to Zizzer and running regressions >>there? >> >> >> >>> A full regression run covers all ISAs and tests, not just quick. This >>>is >>> around 50-60 host CPU hours. >> >> >>Hm, I was not aware that this was the community's plan. This seems pretty >>unclear to me. For instance, the gem5 submission wiki page ( >>http://gem5.org/Submitting_Contributions) lacks precise information about >>which tests must be working and updated, and only states that quick tests >>must be run. It doesn't include anything about updating the regressions, >>which ones, or when the updates need to take place. >> >>Assuming all committers are on the same page about what constitutes "full >>regressions", I'll volunteer to update the page to clarify that the >>committer is responsible for updating all regressions as appropriate when >>a >>change is committed. Do we want to set a specific timeline for this (e.g. >>within the same set of changesets that change regressions, or within the >>next X days)? >> >> >>Can I also suggest a couple things?: >> 1) It is also currently unclear which tests are required based on the >>daily cron regressions emails, and especially since the do-regression >>script isn't available in gem5. At a minimum, we should update scons or >>do-regression on Zizzer to print all regression test names daily. Another >>good thing to do is to include do-regression in gem5, so all contributors >>can see how regressions are run. If we decide not to run full regressions >>daily, those that are not run (but need to be as part of a "full >>regression") should be marked with something more descriptive than >>"skipped" like the currently tracing regressions (eio, tgen). >> >> 2) ::Expanding on my prior email:: If we formalize "full regressions" to >>include all tests, then we *really* need to include full, out-of-the-box >>runable regressions within the gem5 repo or at least using nicely >>packaged >>binaries/disks in a tar file (NOTE: even the FS files tarball requires >>modifying configs/common/SysPaths.py so the simulator can find the >>files). >>It should be possible for both committers AND patch contributors to run >>the >>full regression suite (not just those committers with access to Zizzer). >>Otherwise, it seems like an unreasonable expectation for committers with >>access to Zizzer to completely manage regression test updates. >> >> Probably also makes sense to firm up and document committer/Zizzer >>access >>and regression plans while I go through as a partial guinea pig. >> >> Joel >> >> >> >>On 27/08/2015 14:34, "gem5-dev on behalf of Joel Hestness" >>> <[email protected] on behalf of [email protected]> wrote: >>> >>> >Hi Andreas, >>> > Yes, I agree that committers should run full regressions (i.e. all >>>quick >>> >tests that are verified daily on Zizzer, right?). I ran quick >>>regressions >>> >that are configured correctly, and they pass with this patch. >>> > >>> > The problem with running all quick regressions currently is that >>>many >>> >tests don't run out-of-the-box, and I haven't found instructions on >>>how to >>> >set up tests like the cpu2000 and FS tests. I made a request for such >>> >information in this thread, though I haven't yet seen response: >>> >https://www.mail-archive.com/[email protected]/msg16328.html >>> > >>> > I'm totally willing to run all the quick regressions and update as >>> >appropriate, but I'd like more information on these. How does one set >>>up >>> >FS >>> >and cpu2000 tests? Can we include Linux binaries in the tests/ >>>directory >>> >so >>> >those tests can run out-of-the-box? Can we change the cpu2000 tests to >>> >some >>> >other benchmarks without distribution restrictions, so they too can be >>> >included in tests/? >>> > >>> > Thanks, >>> > Joel >>> > >>> >On Thu, Aug 27, 2015 at 7:19 AM, Andreas Hansson >>><[email protected] >>> > >>> >wrote: >>> > >>> >> Hi Joel, >>> >> >>> >> It seems the patch from Emilio changes a few of the full system >>> >> regressions. I merely wanted to check if you simply forgot to push a >>> >>stats >>> >> update? >>> >> >>> >> In general I would expect anyone pushing patches to do a full >>>regression >>> >> run (and sanity check any changes). Agreed? >>> >> >>> >> Thanks, >>> >> >>> >> Andreas >>> >> >>> >> >>> >> -- IMPORTANT NOTICE: The contents of this email and any attachments >>>are >>> >> confidential and may also be privileged. If you are not the intended >>> >> recipient, please notify the sender immediately and do not disclose >>>the >>> >> contents to any other person, use it for any purpose, or store or >>>copy >>> >>the >>> >> information in any medium. Thank you. >>> >> >>> >> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >>> >> Registered in England & Wales, Company No: 2557590 >>> >> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 >>> >>9NJ, >>> >> Registered in England & Wales, Company No: 2548782 >>> >> _______________________________________________ >>> >> gem5-dev mailing list >>> >> [email protected] >>> >> http://m5sim.org/mailman/listinfo/gem5-dev >>> >> >>> > >>> > >>> > >>> >-- >>> > Joel Hestness >>> > PhD Candidate, Computer Architecture >>> > Dept. of Computer Science, University of Wisconsin - Madison >>> > http://pages.cs.wisc.edu/~hestness/ >>> >_______________________________________________ >>> >gem5-dev mailing list >>> >[email protected] >>> >http://m5sim.org/mailman/listinfo/gem5-dev >>> >>> >>> -- IMPORTANT NOTICE: The contents of this email and any attachments are >>> confidential and may also be privileged. If you are not the intended >>> recipient, please notify the sender immediately and do not disclose the >>> contents to any other person, use it for any purpose, or store or copy >>>the >>> information in any medium. Thank you. >>> >>> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >>> Registered in England & Wales, Company No: 2557590 >>> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 >>>9NJ, >>> Registered in England & Wales, Company No: 2548782 >>> _______________________________________________ >>> gem5-dev mailing list >>> [email protected] >>> http://m5sim.org/mailman/listinfo/gem5-dev >>> >> >>-- >> Joel Hestness >> PhD Candidate, Computer Architecture >> Dept. of Computer Science, University of Wisconsin - Madison >> http://pages.cs.wisc.edu/~hestness/ >>_______________________________________________ >>gem5-dev mailing list >>[email protected] >>http://m5sim.org/mailman/listinfo/gem5-dev > > >-- IMPORTANT NOTICE: The contents of this email and any attachments are >confidential and may also be privileged. If you are not the intended >recipient, please notify the sender immediately and do not disclose the >contents to any other person, use it for any purpose, or store or copy >the information in any medium. Thank you. > >ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >Registered in England & Wales, Company No: 2557590 >ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >Registered in England & Wales, Company No: 2548782 >_______________________________________________ >gem5-dev mailing list >[email protected] >http://m5sim.org/mailman/listinfo/gem5-dev -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782 _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
