Hi there all, somehow I didn't see replies in my mail client before.
The reason I want to (optionally) turn off GC is as especially for micro benchmarking the GC will only occur for one of the runs and that run will then have a significant increase in run time. This causes outliers and makes the average less meaningful and the standard deviation way higher than it ought to be. I agree that for more "macro" style benchmarks I probably wouldn't want to turn off GC because it's an important point there. as one example when running a very fast benchmark the results are usually 80-100 nanoseconds, but some end up being 5000+ messing with the end result. I'll also try to fend those off using statistical methods (like median) but it's not exactly my strong suit (yet) :) also interesting to see the points here against turning off GC. In Ruby it's rather standard practice, benchmark-ips does it and the Ruby performance optimization book explicitly promotes it as a best practice. Thanks for the input everyone! Tobi On 05/31/2016 12:17 AM, Greg Young wrote: > They way people normally do this is to: > > a) measure at 3km instead of 25km or attempt to amortize the costs > over longer runs say 500km while recognizing them (histograms help > here) > b) measure the time of changing shoes and discount it from the measurements > > The nice thing about both of these answers is that it forces you to > recognize the problem with the shoes. > > Cheers, > > Greg > > On Tue, May 31, 2016 at 12:48 AM, Gleb Arshinov <[email protected]> wrote: >> That's a good analogy. And it's describes the reason why being able >> to turn off GC is useful for performance optimization. >> >> It's more obvious when you are profiling. If you don't turn off GC >> when profiling, the full cost of GC will be attributed to the last >> function that triggered GC, not functions that allocated memory >> before. The straw that broke camel's back, e.g. "left leg step at >> kilometer 5" in your analogy. While in reality it needs to allocated >> over left and right steps over the previous 5 kilometers. >> >> When doing performance work in a GC language the cost of a piece of >> code is a pair of: >> * CPU/clock time it take to run (in seconds) >> * and GC pressure it generates (in # of allocations, total size of >> allocations, etc.) >> >> You can convert GC pressure to seconds. But the conversion rate may >> differ greatly between a micro-benchmark and production environment. >> E.g. 10-100ms in Ruby <2.1. So you need to keep GC pressure >> separate, and have a good understanding of how GC works to compare >> these pairs. >> >> High level, comparing a benchmark w/ GC on and off is a good way to >> see if GC time is a significant portion of runtime. >> >> Having said this I have no idea how BEAM GC works and how significant >> it is for performance. >> >> Best regards, >> >> Gleb >> >> On Mon, May 30, 2016 at 9:44 AM, Greg Young <[email protected]> wrote: >>> Wouldn't this be a very bad idea for any kind of benchmarking? >>> >>> I imagine a runner in a race. He has special shoes that make him go a >>> bit faster but for every step he takes a piece of the shoe falls off, >>> occasionally (every 5km) he has to change his shoes. So for our test >>> of his speed we measure a 3km measurement. Unfortunately in the actual >>> race he has to run 25km. >>> >>> Cheers, >>> >>> Greg >>> >>> On Mon, May 30, 2016 at 8:34 AM, Tobias Pfeiffer <[email protected]> wrote: >>>> Hi everyone, >>>> >>>> is there a way in Elixir/Erlang to turn off the Garbage collection? I've >>>> searched and what I found so far is :erlang.garbage_collect to force >>>> garbage collection. http://erlang.org/doc/man/erlang.html#garbage_collect-0 >>>> >>>> Is there any way to turn it off completely? >>>> >>>> Why would I want to do that? I'm working on a benchmarking tool and I >>>> don't want garbage collection to mess with my measured execution times. >>>> >>>> Any hints welcome :) >>>> Tobi >>>> -- >>>> http://www.pragtob.info/ >>>> >>>> -- >>>> You received this message because you are subscribed to the Google Groups >>>> "elixir-lang-talk" group. >>>> To unsubscribe from this group and stop receiving emails from it, send an >>>> email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elixir-lang-talk/574BECF6.1060309%40gmail.com. >>>> For more options, visit https://groups.google.com/d/optout. >>> >>> >>> >>> -- >>> Studying for the Turing test >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "elixir-lang-talk" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elixir-lang-talk/CAC9RQthoVqFCjYunRX7eMr9nF7tybB5tyMKP43GkOT9WoaPSBQ%40mail.gmail.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-talk" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-talk/CACZNi59evCGuyTcs25QLb%3DScA%3DpabrX%2Bh__NqMf%3D%3DFHJNt_ydA%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. > > > -- http://www.pragtob.info/ -- You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/574D601B.3070500%40gmail.com. For more options, visit https://groups.google.com/d/optout.
