--- Stefan Nobis <[EMAIL PROTECTED]> wrote: > Isaac Gouy <[EMAIL PROTECTED]> writes: > > > Again, you seem happy to say they are overrated and dismiss them > > without actually having looked - that is not "scientific"! > > Yes, my critique is not up to the standards I measure the shootout > with. But my main point is: Performance has so many (sometimes hard > to recognize) parameters that's really easy to draw false conclusions > From some benchmark data.
Yes, and it's really really really easy to guess or assume false conclusions without looking at any data! :-) > If you are interested for the best performance for you task > at hand, you have to do benchmarks within your execution environment > and you have to construct benachmarks that really resemble your > task. Different OS, different CPU, even different cache sizes, > different alogrithms and so on. All these things really matter and if > you change any of these parameters the whole picture may change (on > OS > A, CPU B compiler S for language X may be produce code that executes > faster for a benchmark than the code that is produced by compiler T > for language Y on the same system, but on CPU C the combination T/Y > may be better). Yes, and people can see some of that by just by comparing the benchmarks game measurements of the same programs made on the AMD Sempron machine and on the Intel P4 machine. Some of them even read about Flawed Benchmarks http://shootout.alioth.debian.org/gp4/miscfile.php?file=benchmarking&title=Flawed%20Benchmarks > One extra problem: As far as I can see you only compare freely > available language implementations (and this point is overssen by > many > people -- I'm a big open source fan, but more often than not there is > one or more commercial language implementation that produce better > results than any of the freely available implementations). Yes, and if you had looked at the website you would know that we talk insistently about programming language /implementations/. How can we benchmark a programming language? We can't - we benchmark programming language implementations. How can we benchmark language implementations? We can't - we measure particular programs. http://shootout.alioth.debian.org/ > These games are funny and sometimes it's even possible to come to > helpful conclusions given the data. But many people I talked with > understand these shootouts as hard facts, they assume language X is > always better than language Y and don't take into account for example > that they are developing for a Sparc cluster while using benchmarks > for Intel/AMD. So when you talk with those people, point them to examples in the benchmarks game that contradict their simplistic understanding. > OK, so I would say I don't really have any problems with the shootout > but with the perception of these shootouts by many people. And that's > why I dislike these shootouts in general. You seem to be arguing that those people would have a more accurate understanding of "performance" if they had less data! :-) ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping _______________________________________________ computer-go mailing list [email protected] http://www.computer-go.org/mailman/listinfo/computer-go/
