Hey guys, For our gem5-gpu regressions, we have many regression runs that dump statistics multiple times, once each for different unit-like tests in the overall run (this just simplifies the set of runs and setting up each of the tests). The current gem5 stats regression script prints even more output than just the differences (including these non-deterministic stats), so the output can become fairly long and finding the appropriate portion of the regression run that changed takes some parsing. Because this output ends up being pretty clunky, I typically just use diff instead of the gem5 script and then grep out the non-deterministic stats.
Overall, my preference would be to eliminate the non-deterministic stats simply because it streamlines the process a bit. For something as important and detailed as regression testing, I would expect that every bit of improved user experience will result in faster development and/or more reliable code. All that said, this change isn't something that I feel a burning need for, just a nice-to-have. Joel On Thu, Oct 3, 2013 at 7:04 AM, Steve Reinhardt <[email protected]> wrote: > (Switching to email...) > > Since we're in agreement on the Ruby part, I'll go ahead an push that when > I get home (traveling now). > > As far as the non-deterministic parts of stats.txt, I'm not really opposed > to moving them, but I'm curious why it's bothering you. From my > perspective, the only time output determinism matters is when comparing > outputs across different runs, and those comparisons are nearly always > scripted (e.g., to pull out the metrics you care about if you're doing > experiments, or using tests/diff-out for regressions). It's pretty easy to > ignore these stats or treat them specially -- diff-out already does so, and > a script that's just selecting metrics of interest can just not select > them. In the worst case, just ignore all stats that start with 'host_'. > > The reason the Ruby non-determinism bothers me is solely for the > regression testing aspect, and then only because we don't already have a > special script to diff ruby stats (like we do for stats.txt) and it doesn't > make sense to write one when all the information that's getting output is > redundant. If we didn't have the diff-out script, I'd agree that it would > be easier to have put those host-related stats somewhere else, but since we > have the script, I'm curious what your motivation is. > > Steve > > > > On Thu, Oct 3, 2013 at 3:05 AM, Andreas Hansson > <[email protected]>wrote: > >> This is an automatically generated e-mail. To reply, visit: >> http://reviews.gem5.org/r/2028/ >> >> On September 29th, 2013, 2:15 a.m. UTC, *Joel Hestness* wrote: >> >> I've had a personal patch for a while that comments out the code you're >> eliminating here, and I sympathize with wanting to eliminate this to ease >> regression parsing. I've always wondered why we even print a version of this >> data to the stats.txt and not just a separate file for simulator-specific >> data (i.e. separate from simulated system statistics). That would completely >> eliminate the non-deterministic stats problem. >> >> Minor comment below, but otherwise, this looks good to me. >> >> I'd like to put a vote in for removing the men-usage, simulation speed etc >> from the stats.txt. Let's just print this out the stdout (simout) rather. >> Thoughts? >> >> >> - Andreas >> >> On September 28th, 2013, 2:18 a.m. UTC, Steve Reinhardt wrote: >> Review request for Default. >> By Steve Reinhardt. >> >> *Updated Sept. 28, 2013, 2:18 a.m.* >> *Repository: * gem5 >> Description >> >> Changeset 9896:dae3245884f3 >> --------------------------- >> ruby: eliminate non-determinism from ruby.stats output >> >> Get rid of non-deterministic "stats" in ruby.stats output >> such as time & date of run, elapsed & CPU time used, >> and memory usage. These values cause spurious >> miscomparisons when looking at output diffs (though >> they don't affect regressions, since the regressions >> pass/fail status currently ignores ruby.stats entirely). >> >> Most of this information is already captured in other >> places (time & date in stdout, elapsed time & mem usage >> in stats.txt), where the regression script is smart >> enough to filter it out. It seems easier to get rid of >> the redundant output rather than teaching the >> regression tester to ignore the same information in >> two different places. >> >> Diffs >> >> - src/mem/ruby/profiler/Profiler.cc >> (d1a5e147e72db4c64f7d489bb14069c62215471a) >> - src/mem/ruby/system/System.cc >> (d1a5e147e72db4c64f7d489bb14069c62215471a) >> >> View Diff <http://reviews.gem5.org/r/2028/diff/> >> > > -- Joel Hestness PhD Student, Computer Architecture Dept. of Computer Science, University of Wisconsin - Madison http://pages.cs.wisc.edu/~hestness/ _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
