On Wed, Dec 21, 2011 at 1:49 PM, Johan Hendriks <joh.hendr...@gmail.com> wrote: > Nice page, but one thing i do not get is the following. > > [quote] > If you compare FreeBSD / GCC 4.2.1 against, for example, Ubuntu / GCC 4.7 > then the results are unlikely to tell you anything meaningful about FreeBSD > vs Ubuntu. > [/quote] > > That is a little strange in my opinion. > It tells me that FreeBSD falls more and more behind on Linux. > The reason is or could be that FreeBSD cannot or will not include GCC 4.7 > and that FreeBSD will not be on par with Linux anymore.
When benchmarking two systems, you need to make sure that everything possible is the same (constants) and that the only differences between the two systems are what you want to benchmark (variables). For example, if you want to compare the performance of GCC-compiled binaries, then you would use the same hardware host, the same OS install, the same source code, and only change the compiler versions used to compile the "benchmark" binaries. That way, the only variable is "version of GCC", everything else is constant, and thus the benchmark is actually testing the performance of GCC. Likewise, if you want to benchmark the performance of two OSes, you need to eliminate as many variables as possible: - same hardware - running the same benchmark binaries - using the same versions of GCC - using the same filesystems - etc That gives you the starting point. Then, you modify one of the constants above, and re-run the benchmarks. Then you modify one more of the constants above, and re-run the benchmarks. Etc. Each time, you vary only 1 thing, so that you can measure the impact of that *ONE* thing. Comparing "random binary compile with GCC X on FreeBSD Y on filesystem Z on hardware config A" against "random binary built with GCC Q on Linux R on fileystem S on hardware config B" doesn't show anything. Was the performance difference due to hardware? Filesystem? OS? GCC version? Something else? You can't use a shotgun the thread a needle. :) > And what if in the future LLVM gets on par with Linux, is it stil fair to > compare FreeBSD with Linux? Then you add "compiler suite" to the list of variables, and you make it a constant in the first run, and then vary it one piece at a time in later runs, to isolate whether or not it affects performance. In order to do a proper comparison of any two "things", you have to first make them as equal as possible, and then vary things one bit at a time until you are at the "default" configuration for each. Only then can you really, truly, empirically say why A is better/faster/more-uber than B. Unfortunately, doing it "right" requires a lot of time, effort, time, and more time. -- Freddie Cash fjwc...@gmail.com _______________________________________________ firstname.lastname@example.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"