Sounds reasonable to me. I'm not too particular about the naming. I am surprised that even the o3 "hello world" tests wouldn't be < 180 seconds though. It would be nice to have the quick/short/zippy/whatever test category exercise o3 at least a little bit.
As far as composing regression paths, I agree it's awkward, but in general I use the util/regress script to run batches of tests, then just copy/paste the ones that fail if I need to re-run them individually. Of course, all this should still be considered merely stopgap until we get a better test system. Steve On Mon, Dec 22, 2014 at 12:45 PM, Gabe Black via gem5-dev <[email protected] > wrote: > > I mean quick, medium, slow, not quick, medium, fast. > > On Mon, Dec 22, 2014 at 12:44 PM, Gabe Black <[email protected]> wrote: > > > I complained about those names a long time ago, and I still think they > > aren't very good. "quick" and "long" aren't really on the same scale, to > > start with. Something can be quick (a rate) and still take a long time. > > Medium is very generic and so isn't on a different axis, but since the > > others aren't lined up it's not as clear as it could be. I would suggest > > either: > > > > short, medium, long > > > > or > > > > quick, medium, fast > > > > Preferably the first. We have another collection of options the second > > would collide with, namely fast, opt, debug, etc. > > > > If somebody new came along and saw there were fast/quick and opt/long > > regressions, it wouldn't be obvious what that meant. I also think it's > not > > easy to compose one of those regression paths since I can never remember > > what all the parts are or what order they go in and it's not documented > > anywhere obvious. That's a separate problem though. > > > > Gabe > > > > On Mon, Dec 22, 2014 at 2:39 AM, Andreas Hansson via gem5-dev < > > [email protected]> wrote: > > > >> Hi all, > >> > >> At the moment we run roughly 120 regressions, and divide them into quick > >> and long somewhat arbitrarily. Anyone doing active development and using > >> quick as their “quick” way of checking that nothing is broken has to > wait > >> more than 10 minutes for some of these regressions to finish, which > seems a > >> bit of a stretch. It turns out the actual regression run times follow an > >> exponential distribution, ranging from a few seconds up to >10k seconds > >> (almost 3 hours). I propose we also start using medium (mentioned in a > few > >> places), and use a slightly more structured approach in dividing them up > >> into quick, medium and long. > >> > >> Here is what I propose: > >> > >> Quick – anything below 180 seconds, resulting in roughly 40 regressions > >> across all ISAs. The turn around for a quick regression run for NULL, > >> ALPHA, ARM and X86 (what I would deem the minimum to run) should thus be > >> below 5 minutes of wall-clock time. Note that there are plenty > >> configurations not covered by this (o3, realview64 etc). > >> > >> Medium – anything above 180 seconds, but below 1800 seconds, also > >> resulting in roughly 40 regressions. > >> > >> Long – anything >1800 seconds. > >> > >> With this split, quick could be used as part of any development, to get > >> an indication that everything is ok. For a sensible coverage before > posting > >> any patch, quick and medium should do the job. The cronjobs we have > running > >> at the moment could thus do 'quick,medium' for the daily one, and > >> 'quick,medium,long’ for the weekly one. > >> > >> Thoughts? Ideas? Additional comments? > >> > >> Thanks, > >> > >> Andreas > >> > >> > >> -- IMPORTANT NOTICE: The contents of this email and any attachments are > >> confidential and may also be privileged. If you are not the intended > >> recipient, please notify the sender immediately and do not disclose the > >> contents to any other person, use it for any purpose, or store or copy > the > >> information in any medium. Thank you. > >> > >> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, > >> Registered in England & Wales, Company No: 2557590 > >> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 > 9NJ, > >> Registered in England & Wales, Company No: 2548782 > >> _______________________________________________ > >> gem5-dev mailing list > >> [email protected] > >> http://m5sim.org/mailman/listinfo/gem5-dev > >> > > > > > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
