Hi Steve, The 00.hello tests are below 10 seconds and have too high SNR to even make it into my report :-), so yes you are right in that they are included in the ‘short’ regressions.
This is definitely an intermediate step, but in any case we benefit from having a more sensible classification. Thanks for the feedback. Andreas On 22/12/2014 21:21, "Steve Reinhardt via gem5-dev" <[email protected]> wrote: >Sounds reasonable to me. I'm not too particular about the naming. > >I am surprised that even the o3 "hello world" tests wouldn't be < 180 >seconds though. It would be nice to have the quick/short/zippy/whatever >test category exercise o3 at least a little bit. > >As far as composing regression paths, I agree it's awkward, but in general >I use the util/regress script to run batches of tests, then just >copy/paste >the ones that fail if I need to re-run them individually. > >Of course, all this should still be considered merely stopgap until we get >a better test system. > >Steve > > > >On Mon, Dec 22, 2014 at 12:45 PM, Gabe Black via gem5-dev ><[email protected] >> wrote: >> >> I mean quick, medium, slow, not quick, medium, fast. >> >> On Mon, Dec 22, 2014 at 12:44 PM, Gabe Black <[email protected]> >>wrote: >> >> > I complained about those names a long time ago, and I still think they >> > aren't very good. "quick" and "long" aren't really on the same scale, >>to >> > start with. Something can be quick (a rate) and still take a long >>time. >> > Medium is very generic and so isn't on a different axis, but since the >> > others aren't lined up it's not as clear as it could be. I would >>suggest >> > either: >> > >> > short, medium, long >> > >> > or >> > >> > quick, medium, fast >> > >> > Preferably the first. We have another collection of options the second >> > would collide with, namely fast, opt, debug, etc. >> > >> > If somebody new came along and saw there were fast/quick and opt/long >> > regressions, it wouldn't be obvious what that meant. I also think it's >> not >> > easy to compose one of those regression paths since I can never >>remember >> > what all the parts are or what order they go in and it's not >>documented >> > anywhere obvious. That's a separate problem though. >> > >> > Gabe >> > >> > On Mon, Dec 22, 2014 at 2:39 AM, Andreas Hansson via gem5-dev < >> > [email protected]> wrote: >> > >> >> Hi all, >> >> >> >> At the moment we run roughly 120 regressions, and divide them into >>quick >> >> and long somewhat arbitrarily. Anyone doing active development and >>using >> >> quick as their “quick” way of checking that nothing is broken has to >> wait >> >> more than 10 minutes for some of these regressions to finish, which >> seems a >> >> bit of a stretch. It turns out the actual regression run times >>follow an >> >> exponential distribution, ranging from a few seconds up to >10k >>seconds >> >> (almost 3 hours). I propose we also start using medium (mentioned in >>a >> few >> >> places), and use a slightly more structured approach in dividing >>them up >> >> into quick, medium and long. >> >> >> >> Here is what I propose: >> >> >> >> Quick – anything below 180 seconds, resulting in roughly 40 >>regressions >> >> across all ISAs. The turn around for a quick regression run for NULL, >> >> ALPHA, ARM and X86 (what I would deem the minimum to run) should >>thus be >> >> below 5 minutes of wall-clock time. Note that there are plenty >> >> configurations not covered by this (o3, realview64 etc). >> >> >> >> Medium – anything above 180 seconds, but below 1800 seconds, also >> >> resulting in roughly 40 regressions. >> >> >> >> Long – anything >1800 seconds. >> >> >> >> With this split, quick could be used as part of any development, to >>get >> >> an indication that everything is ok. For a sensible coverage before >> posting >> >> any patch, quick and medium should do the job. The cronjobs we have >> running >> >> at the moment could thus do 'quick,medium' for the daily one, and >> >> 'quick,medium,long’ for the weekly one. >> >> >> >> Thoughts? Ideas? Additional comments? >> >> >> >> Thanks, >> >> >> >> Andreas >> >> >> >> >> >> -- IMPORTANT NOTICE: The contents of this email and any attachments >>are >> >> confidential and may also be privileged. If you are not the intended >> >> recipient, please notify the sender immediately and do not disclose >>the >> >> contents to any other person, use it for any purpose, or store or >>copy >> the >> >> information in any medium. Thank you. >> >> >> >> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >> >> Registered in England & Wales, Company No: 2557590 >> >> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 >> 9NJ, >> >> Registered in England & Wales, Company No: 2548782 >> >> _______________________________________________ >> >> gem5-dev mailing list >> >> [email protected] >> >> http://m5sim.org/mailman/listinfo/gem5-dev >> >> >> > >> > >> _______________________________________________ >> gem5-dev mailing list >> [email protected] >> http://m5sim.org/mailman/listinfo/gem5-dev >> >_______________________________________________ >gem5-dev mailing list >[email protected] >http://m5sim.org/mailman/listinfo/gem5-dev -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782 _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
