Hi Nizam, Ah, that explains it.
By waiting different amounts of time before you start the benchmark, you essentially have different system states. This difference in state in turn (the Monday vs Tuesday effect), can blow up tremendously due to the OS, locks etc. There are lots of publications on this, and I’m afraid it’s just a fact of life. I’d suggest to script the running of the benchmark to ensure that it is launched at the same time, with the same state. Even with this in place, I’d suggest to run multiple runs per experiment to ensure that you have some statistical rigour in your results. Gone are the days when computer engineers/scientists could get away just dividing two numbers :-) Andreas From: Nizamudheen Ahmed via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Reply-To: Nizamudheen Ahmed <nizam.ah...@gmail.com<mailto:nizam.ah...@gmail.com>>, gem5 users mailing list <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Date: Thursday, 11 September 2014 04:50 To: Steve Reinhardt <ste...@gmail.com<mailto:ste...@gmail.com>>, gem5 users mailing list <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Subject: Re: [gem5-users] non-determinism in GEM5 stats Hi all, Thanks for the replies. The input to the PARSEC is identical. There is no change in any parameters. I run the same GEM5 invocation command (to invoke GEM5 and restore a checkpoint for Linux boot ) and the run the same PARSEC executable (blackscholes) with the same the same parameters. However, i notice different results. Yes. The simulation should be deterministic. But, i am not sure why i notice the difference. One point i would like to bring to notice. Of course, there is a human intervention in the step. I manually type the command (./blackscholes 4 ../input/in_16K.txt res.txt) on the restored Linux console. However, the statistics are collected for the ROI as determined (deterministically) by the PARSEC executable. I would not expect this manual intervention should cause anomaly. Kindly correct me if i am wrong on this. And, the blackscholes is a multithreaded application. As Biswa pointed, could this be because of the synchronization primitives that causing determinism? Should i change my experiment to remove the human intervention? Kindly recommend. Thanks and Regards, Nizam On Thu, Sep 11, 2014 at 8:05 AM, Steve Reinhardt via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> wrote: Even FS simulation should be deterministic. Although slight changes in inputs can have a significant effect if they cause changes in the order of locks etc., with *identical* inputs the simulation should produce *identical* results. Steve On Wed, Sep 10, 2014 at 6:42 PM, biswabandan panda via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> wrote: Non-determinism comes from the FS simulation. You could try pinning the software threads to the hardware threads. The miss rate varies because of the dynamic behaviour of the synchronization primitives such as barriers and locks. On Wed, Sep 10, 2014 at 9:50 PM, Andreas Hansson via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> wrote: Hi Nizam, Could you elaborate on what is changing between the runs? The simulator is deterministic, so if everything stays the same you will get exactly the same output (otherwise we would never be able to run any regressions the way we do). Andreas From: Nizamudheen Ahmed via gem5-users <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Reply-To: Nizamudheen Ahmed <nizam.ah...@gmail.com<mailto:nizam.ah...@gmail.com>>, gem5 users mailing list <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Date: Wednesday, 10 September 2014 16:50 To: gem5 users mailing list <gem5-users@gem5.org<mailto:gem5-users@gem5.org>> Subject: [gem5-users] non-determinism in GEM5 stats Hi champs, I observe the stats.txt contain alarmingly different data across multiple run of the same benchmark under same target configuration. For example, i get 17% miss-rate and 5% miss-rate across two different runs. Can you help me identifying what could be going wrong? I have checkpointed the Linux boot and i am attempting to run PARSEC benchmark (blackscholes to be precise) ARM ISA. I also use m5::reset_stats and m5::dump_and_reset_stats around the ROI (i instrumented the hook.c in the PARSEC to invoke these calls). Thanks. Best regards, Nizam -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782 _______________________________________________ gem5-users mailing list gem5-users@gem5.org<mailto:gem5-users@gem5.org> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users -- thanks®ards BISWABANDAN http://www.cse.iitm.ac.in/~biswa/<http://www.cse.iitm.ac.in/%7Ebiswa/> “We might fall down, but we will never lay down. We might not be the best, but we will beat the best! We might not be at the top, but we will rise.” _______________________________________________ gem5-users mailing list gem5-users@gem5.org<mailto:gem5-users@gem5.org> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users _______________________________________________ gem5-users mailing list gem5-users@gem5.org<mailto:gem5-users@gem5.org> http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ARM Limited, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2557590 ARM Holdings plc, Registered office 110 Fulbourn Road, Cambridge CB1 9NJ, Registered in England & Wales, Company No: 2548782
_______________________________________________ gem5-users mailing list gem5-users@gem5.org http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
