On Fri, 2006-05-19 at 09:48 -0700, therandthem wrote: > I doubt the shootout is without merit. In the very > least, it shows the general speed of a language > implementation.
I doubt it. I was heavily involved with the Shootout for a while (until Isaac Gouy took over and tried to be more Fascist than Adolf Hitler). First, many of the tests give wildly different results run elsewhere. Second, many of the requirements are rubbish: implementing an algorithm which does array manipulations is nonsense in a functional programming language, ditto for loops (FPLs use recursion). Thirdly, as a more sane kind of measurement such as those done by Jon Harrop show, a language like Ocaml may be slower for a fixed algorithm, but when you take the expressiveness of the language into account, and fix the amount of code, Ocaml, for example, can be much faster than C, simply because you can include more optimisations. Fourthly, Micky Mouse benchmarks don't give any serious indication of real world performance. I can say that for sure, because on many tests my own programming languague Felix utterly creamed all the others. So much so said Fascist Isaac Gouy banned it. On the EH test, Felix ran in negative time, orders of magnitude faster than the nearest competitor! It trashes every available translator on Ackermann test, including Ocaml and even gcc -- despite generating C++ code processed by gcc :) On the 'threading' test it again wiped everyone else out of the sky. In all these cases, as the author of the compiler, I KNOW the reason was because the benchmarks were not reasonable. Felix is good, but not that good: in all these cases the optimiser eliminated the very feature that was supposed to be tested. In the EH test no 'exceptions' got thrown, the optimiser reduced them to a single machine instruction, a goto. Same in the threading test .. it just happened to use a chain of threads, and in Felix every procedure is a thread.. but the chain was tail-rec and Felix turned it into a loop. Last time I looked, Java was one of the fastest languages. Yet you say: > That is why the languages that we > already know are slowest are at the bottom of the list > and the fastest at the top. but we know Java is woefully slow. > Besides, one of the great strengths of the shootout is > that it is active. Search for benchmarks on database > systems, for example, and you cannot get to the end of > one-off tests comparing MySQL latest to PostgreSQL 7 > (older version). The shootout is a fair fight No it isn't. > and > does establish an accurate general speed mark for the > languages. Not even slightly. > Does anyone look at the list and think, > "there's no way OCaml is that fast"? Yes, absolutely. There is no way Java is anywhere near as fast as the tests make out. The test methodology is seriously flawed. The rules about which ways you have to do things are arbitrary. Can C use 'x' library or not? Is C++ allowed to use templates to implement Ackermann? [In which case it would cream absolutely everything, since the whole of the computation would be done at compile time, which is not counted in the results] Again, last I looked, the ranking of the languages was just rubbish. Haskell does badly, and it is indeed slow at some kinds of things, but it is very VERY good at doing others. Similarly, if you read the Ocaml programming lists, you'll see minor changes in code structure can have significant performance implications. A small number of micro benchmarks simply cannot average out all these factors, and tests based on implementing a particular algorithm are necessarily rubbish. I argued strenuously -- along with Jon Harrop -- the tests should be functional and independent of how the result was calculated, but Isaac didn't like that so he kicked both of us out. He removed Felix, after I worked for months on the compiler and tests, and he revoked Jon's CVS access to prevent him contributing the kinds of tests he thought made more sense. Look at the results, you'll see Erlang seems slow .. yet it is used heavily in Telco applications which demand ultra high real time performance. -- John Skaller <skaller at users dot sf dot net> Felix, successor to C++: http://felix.sf.net -- Neko : One VM to run them all (http://nekovm.org)
