Thanks for the clarification, Andreas. Yes, it's a good step; thanks for doing it.
Steve On Tue, Dec 23, 2014 at 12:55 AM, Andreas Hansson via gem5-dev < [email protected]> wrote: > > Hi Steve, > > The 00.hello tests are below 10 seconds and have too high SNR to even make > it into my report :-), so yes you are right in that they are included in > the ‘short’ regressions. > > This is definitely an intermediate step, but in any case we benefit from > having a more sensible classification. > > Thanks for the feedback. > > Andreas > > On 22/12/2014 21:21, "Steve Reinhardt via gem5-dev" <[email protected]> > wrote: > > >Sounds reasonable to me. I'm not too particular about the naming. > > > >I am surprised that even the o3 "hello world" tests wouldn't be < 180 > >seconds though. It would be nice to have the quick/short/zippy/whatever > >test category exercise o3 at least a little bit. > > > >As far as composing regression paths, I agree it's awkward, but in general > >I use the util/regress script to run batches of tests, then just > >copy/paste > >the ones that fail if I need to re-run them individually. > > > >Of course, all this should still be considered merely stopgap until we get > >a better test system. > > > >Steve > > > > > > > >On Mon, Dec 22, 2014 at 12:45 PM, Gabe Black via gem5-dev > ><[email protected] > >> wrote: > >> > >> I mean quick, medium, slow, not quick, medium, fast. > >> > >> On Mon, Dec 22, 2014 at 12:44 PM, Gabe Black <[email protected]> > >>wrote: > >> > >> > I complained about those names a long time ago, and I still think they > >> > aren't very good. "quick" and "long" aren't really on the same scale, > >>to > >> > start with. Something can be quick (a rate) and still take a long > >>time. > >> > Medium is very generic and so isn't on a different axis, but since the > >> > others aren't lined up it's not as clear as it could be. I would > >>suggest > >> > either: > >> > > >> > short, medium, long > >> > > >> > or > >> > > >> > quick, medium, fast > >> > > >> > Preferably the first. We have another collection of options the second > >> > would collide with, namely fast, opt, debug, etc. > >> > > >> > If somebody new came along and saw there were fast/quick and opt/long > >> > regressions, it wouldn't be obvious what that meant. I also think it's > >> not > >> > easy to compose one of those regression paths since I can never > >>remember > >> > what all the parts are or what order they go in and it's not > >>documented > >> > anywhere obvious. That's a separate problem though. > >> > > >> > Gabe > >> > > >> > On Mon, Dec 22, 2014 at 2:39 AM, Andreas Hansson via gem5-dev < > >> > [email protected]> wrote: > >> > > >> >> Hi all, > >> >> > >> >> At the moment we run roughly 120 regressions, and divide them into > >>quick > >> >> and long somewhat arbitrarily. Anyone doing active development and > >>using > >> >> quick as their “quick” way of checking that nothing is broken has to > >> wait > >> >> more than 10 minutes for some of these regressions to finish, which > >> seems a > >> >> bit of a stretch. It turns out the actual regression run times > >>follow an > >> >> exponential distribution, ranging from a few seconds up to >10k > >>seconds > >> >> (almost 3 hours). I propose we also start using medium (mentioned in > >>a > >> few > >> >> places), and use a slightly more structured approach in dividing > >>them up > >> >> into quick, medium and long. > >> >> > >> >> Here is what I propose: > >> >> > >> >> Quick – anything below 180 seconds, resulting in roughly 40 > >>regressions > >> >> across all ISAs. The turn around for a quick regression run for NULL, > >> >> ALPHA, ARM and X86 (what I would deem the minimum to run) should > >>thus be > >> >> below 5 minutes of wall-clock time. Note that there are plenty > >> >> configurations not covered by this (o3, realview64 etc). > >> >> > >> >> Medium – anything above 180 seconds, but below 1800 seconds, also > >> >> resulting in roughly 40 regressions. > >> >> > >> >> Long – anything >1800 seconds. > >> >> > >> >> With this split, quick could be used as part of any development, to > >>get > >> >> an indication that everything is ok. For a sensible coverage before > >> posting > >> >> any patch, quick and medium should do the job. The cronjobs we have > >> running > >> >> at the moment could thus do 'quick,medium' for the daily one, and > >> >> 'quick,medium,long’ for the weekly one. > >> >> > >> >> Thoughts? Ideas? Additional comments? > >> >> > >> >> Thanks, > >> >> > >> >> Andreas > >> >> > >> >> > >> >> -- IMPORTANT NOTICE: The contents of this email and any attachments > >>are > >> >> confidential and may also be privileged. If you are not the intended > >> >> recipient, please notify the sender immediately and do not disclose > >>the > >> >> contents to any other person, use it for any purpose, or store or > >>copy > >> the > >> >> information in any medium. Thank you. > >> >> > >> >> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, > >> >> Registered in England & Wales, Company No: 2557590 > >> >> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 > >> 9NJ, > >> >> Registered in England & Wales, Company No: 2548782 > >> >> _______________________________________________ > >> >> gem5-dev mailing list > >> >> [email protected] > >> >> http://m5sim.org/mailman/listinfo/gem5-dev > >> >> > >> > > >> > > >> _______________________________________________ > >> gem5-dev mailing list > >> [email protected] > >> http://m5sim.org/mailman/listinfo/gem5-dev > >> > >_______________________________________________ > >gem5-dev mailing list > >[email protected] > >http://m5sim.org/mailman/listinfo/gem5-dev > > > -- IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > > ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, > Registered in England & Wales, Company No: 2557590 > ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, > Registered in England & Wales, Company No: 2548782 > _______________________________________________ > gem5-dev mailing list > [email protected] > http://m5sim.org/mailman/listinfo/gem5-dev > _______________________________________________ gem5-dev mailing list [email protected] http://m5sim.org/mailman/listinfo/gem5-dev
