Hi Andreas, Thanks for the response. I will script the benchmark execution so that the human intervention is eliminated. I will ensure that i the benchmarks multiple times and observe near-identical results.
I will keep you posted. Thanks and regards, Nizam On Thu, Sep 11, 2014 at 1:35 PM, Andreas Hansson <andreas.hans...@arm.com> wrote: > Hi Nizam, > > Ah, that explains it. > > By waiting different amounts of time before you start the benchmark, you > essentially have different system states. This difference in state in turn > (the Monday vs Tuesday effect), can blow up tremendously due to the OS, > locks etc. There are lots of publications on this, and I’m afraid it’s just > a fact of life. > > I’d suggest to script the running of the benchmark to ensure that it is > launched at the same time, with the same state. Even with this in place, > I’d suggest to run multiple runs per experiment to ensure that you have > some statistical rigour in your results. Gone are the days when computer > engineers/scientists could get away just dividing two numbers :-) > > Andreas > > From: Nizamudheen Ahmed via gem5-users <gem5-users@gem5.org> > Reply-To: Nizamudheen Ahmed <nizam.ah...@gmail.com>, gem5 users mailing > list <gem5-users@gem5.org> > Date: Thursday, 11 September 2014 04:50 > To: Steve Reinhardt <ste...@gmail.com>, gem5 users mailing list < > gem5-users@gem5.org> > Subject: Re: [gem5-users] non-determinism in GEM5 stats > > Hi all, > > Thanks for the replies. > > The input to the PARSEC is identical. There is no change in any > parameters. I run the same GEM5 invocation command (to invoke GEM5 and > restore a checkpoint for Linux boot ) and the run the same PARSEC > executable (blackscholes) with the same the same parameters. However, i > notice different results. > > Yes. The simulation should be deterministic. But, i am not sure why i > notice the difference. One point i would like to bring to notice. Of > course, there is a human intervention in the step. I manually type the > command (./blackscholes 4 ../input/in_16K.txt res.txt) on the restored > Linux console. However, the statistics are collected for the ROI as > determined (deterministically) by the PARSEC executable. I would not expect > this manual intervention should cause anomaly. Kindly correct me if i am > wrong on this. > > And, the blackscholes is a multithreaded application. As Biswa pointed, > could this be because of the synchronization primitives that causing > determinism? > > Should i change my experiment to remove the human intervention? Kindly > recommend. > > Thanks and Regards, > Nizam > > > > > On Thu, Sep 11, 2014 at 8:05 AM, Steve Reinhardt via gem5-users < > gem5-users@gem5.org> wrote: > >> Even FS simulation should be deterministic. Although slight changes in >> inputs can have a significant effect if they cause changes in the order of >> locks etc., with *identical* inputs the simulation should produce >> *identical* results. >> >> Steve >> >> On Wed, Sep 10, 2014 at 6:42 PM, biswabandan panda via gem5-users < >> gem5-users@gem5.org> wrote: >> >>> Non-determinism comes from the FS simulation. >>> You could try pinning the software threads to the >>> hardware threads. The miss rate varies because >>> of the dynamic behaviour of the synchronization primitives such as >>> barriers and locks. >>> >>> >>> On Wed, Sep 10, 2014 at 9:50 PM, Andreas Hansson via gem5-users < >>> gem5-users@gem5.org> wrote: >>> >>>> Hi Nizam, >>>> >>>> Could you elaborate on what is changing between the runs? >>>> >>>> The simulator is deterministic, so if everything stays the same you >>>> will get exactly the same output (otherwise we would never be able to run >>>> any regressions the way we do). >>>> >>>> Andreas >>>> >>>> From: Nizamudheen Ahmed via gem5-users <gem5-users@gem5.org> >>>> Reply-To: Nizamudheen Ahmed <nizam.ah...@gmail.com>, gem5 users >>>> mailing list <gem5-users@gem5.org> >>>> Date: Wednesday, 10 September 2014 16:50 >>>> To: gem5 users mailing list <gem5-users@gem5.org> >>>> Subject: [gem5-users] non-determinism in GEM5 stats >>>> >>>> Hi champs, >>>> >>>> I observe the stats.txt contain alarmingly different data across >>>> multiple run of the same benchmark under same target configuration. For >>>> example, i get 17% miss-rate and 5% miss-rate across two different runs. >>>> >>>> Can you help me identifying what could be going wrong? >>>> >>>> I have checkpointed the Linux boot and i am attempting to run PARSEC >>>> benchmark (blackscholes to be precise) ARM ISA. I also use m5::reset_stats >>>> and m5::dump_and_reset_stats around the ROI (i instrumented the hook.c in >>>> the PARSEC to invoke these calls). >>>> >>>> Thanks. >>>> >>>> Best regards, >>>> Nizam >>>> >>>> >>>> -- IMPORTANT NOTICE: The contents of this email and any attachments are >>>> confidential and may also be privileged. If you are not the intended >>>> recipient, please notify the sender immediately and do not disclose the >>>> contents to any other person, use it for any purpose, or store or copy the >>>> information in any medium. Thank you. >>>> >>>> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >>>> Registered in England & Wales, Company No: 2557590 >>>> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 >>>> 9NJ, Registered in England & Wales, Company No: 2548782 >>>> >>>> _______________________________________________ >>>> gem5-users mailing list >>>> gem5-users@gem5.org >>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>> >>> >>> >>> >>> -- >>> >>> >>> *thanks®ards * >>> *BISWABANDAN* >>> http://www.cse.iitm.ac.in/~biswa/ >>> >>> “We might fall down, but we will never lay down. We might not be the >>> best, but we will beat the best! We might not be at the top, but we will >>> rise.” >>> >>> >>> >>> _______________________________________________ >>> gem5-users mailing list >>> gem5-users@gem5.org >>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>> >> >> >> _______________________________________________ >> gem5-users mailing list >> gem5-users@gem5.org >> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >> > > > -- IMPORTANT NOTICE: The contents of this email and any attachments are > confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the > contents to any other person, use it for any purpose, or store or copy the > information in any medium. Thank you. > > ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, > Registered in England & Wales, Company No: 2557590 > ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, > Registered in England & Wales, Company No: 2548782 >
_______________________________________________ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
