They way people normally do this is to: a) measure at 3km instead of 25km or attempt to amortize the costs over longer runs say 500km while recognizing them (histograms help here) b) measure the time of changing shoes and discount it from the measurements
The nice thing about both of these answers is that it forces you to recognize the problem with the shoes. Cheers, Greg On Tue, May 31, 2016 at 12:48 AM, Gleb Arshinov <[email protected]> wrote: > That's a good analogy. And it's describes the reason why being able > to turn off GC is useful for performance optimization. > > It's more obvious when you are profiling. If you don't turn off GC > when profiling, the full cost of GC will be attributed to the last > function that triggered GC, not functions that allocated memory > before. The straw that broke camel's back, e.g. "left leg step at > kilometer 5" in your analogy. While in reality it needs to allocated > over left and right steps over the previous 5 kilometers. > > When doing performance work in a GC language the cost of a piece of > code is a pair of: > * CPU/clock time it take to run (in seconds) > * and GC pressure it generates (in # of allocations, total size of > allocations, etc.) > > You can convert GC pressure to seconds. But the conversion rate may > differ greatly between a micro-benchmark and production environment. > E.g. 10-100ms in Ruby <2.1. So you need to keep GC pressure > separate, and have a good understanding of how GC works to compare > these pairs. > > High level, comparing a benchmark w/ GC on and off is a good way to > see if GC time is a significant portion of runtime. > > Having said this I have no idea how BEAM GC works and how significant > it is for performance. > > Best regards, > > Gleb > > On Mon, May 30, 2016 at 9:44 AM, Greg Young <[email protected]> wrote: >> Wouldn't this be a very bad idea for any kind of benchmarking? >> >> I imagine a runner in a race. He has special shoes that make him go a >> bit faster but for every step he takes a piece of the shoe falls off, >> occasionally (every 5km) he has to change his shoes. So for our test >> of his speed we measure a 3km measurement. Unfortunately in the actual >> race he has to run 25km. >> >> Cheers, >> >> Greg >> >> On Mon, May 30, 2016 at 8:34 AM, Tobias Pfeiffer <[email protected]> wrote: >>> Hi everyone, >>> >>> is there a way in Elixir/Erlang to turn off the Garbage collection? I've >>> searched and what I found so far is :erlang.garbage_collect to force >>> garbage collection. http://erlang.org/doc/man/erlang.html#garbage_collect-0 >>> >>> Is there any way to turn it off completely? >>> >>> Why would I want to do that? I'm working on a benchmarking tool and I >>> don't want garbage collection to mess with my measured execution times. >>> >>> Any hints welcome :) >>> Tobi >>> -- >>> http://www.pragtob.info/ >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "elixir-lang-talk" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elixir-lang-talk/574BECF6.1060309%40gmail.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> >> >> -- >> Studying for the Turing test >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elixir-lang-talk" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elixir-lang-talk/CAC9RQthoVqFCjYunRX7eMr9nF7tybB5tyMKP43GkOT9WoaPSBQ%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "elixir-lang-talk" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elixir-lang-talk/CACZNi59evCGuyTcs25QLb%3DScA%3DpabrX%2Bh__NqMf%3D%3DFHJNt_ydA%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. -- Studying for the Turing test -- You received this message because you are subscribed to the Google Groups "elixir-lang-talk" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-talk/CAC9RQti6xGfzrp9eytn1s5DEDi-mSy_%2Bs_LkkOws%3DxxGEhXLkA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
