I feel that ignoring gc can be a bit of a cheat since it does happen and it's quite expensive – and other systems may be better or worse at it. Of course, it can still be good to separate the cause of slowness explicitly into execution time and overhead for things like gc.
On Mon, Jun 2, 2014 at 5:21 PM, Kevin Squire <[email protected]> wrote: > Thanks John. His argument definitely makes sense (that algorithms that > cause more garbage collection won't get penalized by median, unless, of > course, they cause gc() to occur more than 50% of the time). > > Most benchmarks of Julia code that I've done (or seen) have made some > attempt to take gc() differences out of the equation, usually by explicitly > calling gc() before the timing begins. For most algorithms, that would > mean that the same number of gc() calls should occur for each repetition, > in which case, I would think that any measure of central tendency > (including mean and median) would be useful. > > Is there a problem with this reasoning? > > Cheers, > Kevin > > > On Mon, Jun 2, 2014 at 1:04 PM, John Myles White <[email protected] > > wrote: > >> For some reasons why one might want not to use the median, see >> http://radfordneal.wordpress.com/2014/02/02/inaccurate-results-from-microbenchmark/ >> >> -- John >> >> On Jun 2, 2014, at 11:06 AM, Kevin Squire <[email protected]> wrote: >> >> median is probably also useful. I like it a little better in cases where >> the code being tested triggers gc() more than half the time. >> >> On Monday, June 2, 2014, Steven G. Johnson <[email protected]> wrote: >> >>> >>> >>> On Monday, June 2, 2014 1:01:25 AM UTC-4, Jameson wrote: >>>> >>>> Therefore, for benchmarks, you should execute your code in a loop >>>> enough times that the measurement error (of the hardware and OS) is not too >>>> significant. >>>> >>> >>> You can also often benchmark multiple times and take the minimum (not >>> the mean!) time for reasonable results with fairly small time intervals. >>> >> >> >
