Hello, The existence of variance across machines is true even with ASLR on. My point was regarding doing repetitive measurements on the same machine. Nevertheless, even if small variations may appear in certain circumstances, you can minimize them if you use identical machines, with identical software, settings, etc. And the most important, the deltas remain comparable.
For the dedicated Python CodeSpeed machine that does daily measurements, among others, this can be a good setting for a bit more reliable results. Thank you, Alecsandru > -----Original Message----- > From: Maciej Fijalkowski [mailto:[email protected]] > Sent: Sunday, February 14, 2016 2:20 PM > To: Patrascu, Alecsandru <[email protected]> > Cc: Victor Stinner <[email protected]>; [email protected] > Subject: Re: [Speed] Linux tip: use isolcpus to have (more) reliable > benchmark > > Hi. > > Disabling ASLR means you get more repeatable benchmarks, of course, but > also means that on another identical machine (or a bit different > circumstances), you can get different results, hence you moved statistical > error to a more systematic one. I don't think that's a win > > On Fri, Feb 12, 2016 at 8:42 AM, Patrascu, Alecsandru > <[email protected]> wrote: > > Hi, > > > > Some of the things we do here at Intel, in our Languages Performance Lab > [1,2], is to disable ASLR as you get more reliable results. This can be > achieved on Linux by running echo 0 > /proc/sys/kernel/randomize_va_space. > Also, setting the CPU frequency at a fixed frequency, disabling Turbo > Boost and Hyper Threading, also helps for benchmark stability. > > > > From my experience, the isolcpus feature is useful when you have a lot > of cores on your machine because the kernel will have other cores on which > it can schedule its work; furthermore, it is a best effort situation and > it is not an absolute guarantee that the kernel will not use the cores > specified if you have a lot of processes running (for example, if you > benchmark on a machine with 2 physical cores and you isolate one of the > cores, there is a big chance that the kernel will schedule processes on > this core also, even it is for a small amount of time). Nevertheless, for > machines with more physical cores, it can be good to have dedicated > core(s) on which we do benchmarking. > > > > [1] http://languagesperformance.intel.com/ > > [2] https://lists.01.org/pipermail/langperf/ > > > > Thank you, > > Alecsandru > > > >> -----Original Message----- > >> From: Speed [mailto:speed- > >> [email protected]] On Behalf Of Victor > >> Stinner > >> Sent: Friday, February 12, 2016 12:54 AM > >> To: [email protected] > >> Subject: [Speed] Linux tip: use isolcpus to have (more) reliable > >> benchmark > >> > >> Hi, > >> > >> I'm sharing with you my notes (tricks) to get more reliable > >> benchmarks on Linux if your CPU have multiple cores: > >> > >> https://haypo-notes.readthedocs.org/microbenchmark.html#reliable-micr > >> o- > >> benchmarks > >> > >> FYI perf.py recently got a new --affinity= optional parameter. I plan > >> to send a patch to automatically use /sys/devices/system/cpu/isolated > >> if it's not empty. > >> > >> What are your "tricks" to get reliable benchmarks? > >> > >> Victor > >> _______________________________________________ > >> Speed mailing list > >> [email protected] > >> https://mail.python.org/mailman/listinfo/speed > > _______________________________________________ > > Speed mailing list > > [email protected] > > https://mail.python.org/mailman/listinfo/speed _______________________________________________ Speed mailing list [email protected] https://mail.python.org/mailman/listinfo/speed
