Hi Andreas, all, I am able to get identical results for multiple runs, with the elimination of manual intervention in my experiments. Thanks for the help.
BR/Nizam On Thu, Sep 11, 2014 at 5:01 PM, Nizamudheen Ahmed <nizam.ah...@gmail.com> wrote: > Hi Andreas, > > Thanks for the response. I will script the benchmark execution so that the > human intervention is eliminated. I will ensure that i the benchmarks > multiple times and observe near-identical results. > > I will keep you posted. > > Thanks and regards, > Nizam > > On Thu, Sep 11, 2014 at 1:35 PM, Andreas Hansson <andreas.hans...@arm.com> > wrote: > >> Hi Nizam, >> >> Ah, that explains it. >> >> By waiting different amounts of time before you start the benchmark, >> you essentially have different system states. This difference in state in >> turn (the Monday vs Tuesday effect), can blow up tremendously due to the >> OS, locks etc. There are lots of publications on this, and I’m afraid it’s >> just a fact of life. >> >> I’d suggest to script the running of the benchmark to ensure that it is >> launched at the same time, with the same state. Even with this in place, >> I’d suggest to run multiple runs per experiment to ensure that you have >> some statistical rigour in your results. Gone are the days when computer >> engineers/scientists could get away just dividing two numbers :-) >> >> Andreas >> >> From: Nizamudheen Ahmed via gem5-users <gem5-users@gem5.org> >> Reply-To: Nizamudheen Ahmed <nizam.ah...@gmail.com>, gem5 users mailing >> list <gem5-users@gem5.org> >> Date: Thursday, 11 September 2014 04:50 >> To: Steve Reinhardt <ste...@gmail.com>, gem5 users mailing list < >> gem5-users@gem5.org> >> Subject: Re: [gem5-users] non-determinism in GEM5 stats >> >> Hi all, >> >> Thanks for the replies. >> >> The input to the PARSEC is identical. There is no change in any >> parameters. I run the same GEM5 invocation command (to invoke GEM5 and >> restore a checkpoint for Linux boot ) and the run the same PARSEC >> executable (blackscholes) with the same the same parameters. However, i >> notice different results. >> >> Yes. The simulation should be deterministic. But, i am not sure why i >> notice the difference. One point i would like to bring to notice. Of >> course, there is a human intervention in the step. I manually type the >> command (./blackscholes 4 ../input/in_16K.txt res.txt) on the restored >> Linux console. However, the statistics are collected for the ROI as >> determined (deterministically) by the PARSEC executable. I would not expect >> this manual intervention should cause anomaly. Kindly correct me if i am >> wrong on this. >> >> And, the blackscholes is a multithreaded application. As Biswa pointed, >> could this be because of the synchronization primitives that causing >> determinism? >> >> Should i change my experiment to remove the human intervention? Kindly >> recommend. >> >> Thanks and Regards, >> Nizam >> >> >> >> >> On Thu, Sep 11, 2014 at 8:05 AM, Steve Reinhardt via gem5-users < >> gem5-users@gem5.org> wrote: >> >>> Even FS simulation should be deterministic. Although slight changes in >>> inputs can have a significant effect if they cause changes in the order of >>> locks etc., with *identical* inputs the simulation should produce >>> *identical* results. >>> >>> Steve >>> >>> On Wed, Sep 10, 2014 at 6:42 PM, biswabandan panda via gem5-users < >>> gem5-users@gem5.org> wrote: >>> >>>> Non-determinism comes from the FS simulation. >>>> You could try pinning the software threads to the >>>> hardware threads. The miss rate varies because >>>> of the dynamic behaviour of the synchronization primitives such as >>>> barriers and locks. >>>> >>>> >>>> On Wed, Sep 10, 2014 at 9:50 PM, Andreas Hansson via gem5-users < >>>> gem5-users@gem5.org> wrote: >>>> >>>>> Hi Nizam, >>>>> >>>>> Could you elaborate on what is changing between the runs? >>>>> >>>>> The simulator is deterministic, so if everything stays the same you >>>>> will get exactly the same output (otherwise we would never be able to run >>>>> any regressions the way we do). >>>>> >>>>> Andreas >>>>> >>>>> From: Nizamudheen Ahmed via gem5-users <gem5-users@gem5.org> >>>>> Reply-To: Nizamudheen Ahmed <nizam.ah...@gmail.com>, gem5 users >>>>> mailing list <gem5-users@gem5.org> >>>>> Date: Wednesday, 10 September 2014 16:50 >>>>> To: gem5 users mailing list <gem5-users@gem5.org> >>>>> Subject: [gem5-users] non-determinism in GEM5 stats >>>>> >>>>> Hi champs, >>>>> >>>>> I observe the stats.txt contain alarmingly different data across >>>>> multiple run of the same benchmark under same target configuration. For >>>>> example, i get 17% miss-rate and 5% miss-rate across two different runs. >>>>> >>>>> Can you help me identifying what could be going wrong? >>>>> >>>>> I have checkpointed the Linux boot and i am attempting to run PARSEC >>>>> benchmark (blackscholes to be precise) ARM ISA. I also use m5::reset_stats >>>>> and m5::dump_and_reset_stats around the ROI (i instrumented the hook.c in >>>>> the PARSEC to invoke these calls). >>>>> >>>>> Thanks. >>>>> >>>>> Best regards, >>>>> Nizam >>>>> >>>>> >>>>> -- IMPORTANT NOTICE: The contents of this email and any attachments >>>>> are confidential and may also be privileged. If you are not the intended >>>>> recipient, please notify the sender immediately and do not disclose the >>>>> contents to any other person, use it for any purpose, or store or copy the >>>>> information in any medium. Thank you. >>>>> >>>>> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >>>>> Registered in England & Wales, Company No: 2557590 >>>>> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 >>>>> 9NJ, Registered in England & Wales, Company No: 2548782 >>>>> >>>>> _______________________________________________ >>>>> gem5-users mailing list >>>>> gem5-users@gem5.org >>>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> >>>> *thanks®ards * >>>> *BISWABANDAN* >>>> http://www.cse.iitm.ac.in/~biswa/ >>>> >>>> “We might fall down, but we will never lay down. We might not be the >>>> best, but we will beat the best! We might not be at the top, but we will >>>> rise.” >>>> >>>> >>>> >>>> _______________________________________________ >>>> gem5-users mailing list >>>> gem5-users@gem5.org >>>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>>> >>> >>> >>> _______________________________________________ >>> gem5-users mailing list >>> gem5-users@gem5.org >>> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users >>> >> >> >> -- IMPORTANT NOTICE: The contents of this email and any attachments are >> confidential and may also be privileged. If you are not the intended >> recipient, please notify the sender immediately and do not disclose the >> contents to any other person, use it for any purpose, or store or copy the >> information in any medium. Thank you. >> >> ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >> Registered in England & Wales, Company No: 2557590 >> ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, >> Registered in England & Wales, Company No: 2548782 >> > >
_______________________________________________ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
