> On Fri, 2006-05-19 at 09:48 -0700, therandthem wrote: > > I doubt the shootout is without merit. In the very > > least, it shows the general speed of a language > > implementation. > > I doubt it. I was heavily involved with the Shootout > for a while (until Isaac Gouy took over and tried to > be more Fascist than Adolf Hitler).
You know, it annoys me that you always seem so negative; not just in this discussion, but also in other discussions that have been held on this list. I'm not sure if it has the same effect on other people, but your negativity discourages me from taking part in the discussion. Could you try to be a bit negative, for my sake and the sake of any others who are affected by it? > First, many of the tests give wildly different > results run elsewhere. Could you give some examples of that? And could you state why you think the results differ, and which result (if any) you believe to be the correct one? > Second, many of the requirements are rubbish: > implementing an algorithm which does array manipulations > is nonsense in a functional programming language, > ditto for loops (FPLs use recursion). I wouldn't call that nonsense. If a language does not provide certain constructs (e.g. mutable arrays or loops), I think it's useful to represent that fact. Similarly, I think it's useful to include programs using constructs like first class functions and first class continuations, also to illustrate their presence, performance, and usage. > Thirdly, as a more sane kind of measurement such as > those done by Jon Harrop show, a language like Ocaml > may be slower for a fixed algorithm, but when you take > the expressiveness of the language into account, > and fix the amount of code, Ocaml, for example, > can be much faster than C, simply because you can > include more optimisations. I completely agree. On the other hand, note that the Shootout does measure lines of code and memory consumption. They _are_ doing more than measuring just one characteristic. Eventually, you do have to draw the line somewhere. It would perhaps be interesting to have data on how long it takes to develop such a program in a certain language, or perhaps include different entries optimized for run time, memory usage, program size, or program cleanness. > Fourthly, Micky Mouse benchmarks don't give any > serious indication of real world performance. True. But, when done right, they do give an indication of the run time cost of various constructs in various languages. This is valuable information in its own right. As for Real World performance, that tends to be determined by a whole lot of things that have nothing to do with the language. For example, it doesn't matter if a program is written in language foo or bar if it's performance is limited by the rate at which packets come in, the rate at which the operating system can copy memory pages, the time the disk head takes to find the right track, or the rate at which the video card can push out polygons. I think the rise of high level interpreted languages is largely due to the realization that, often, the performance offered by the language implementation is not the most important thing to consider. That doesn't negate the fact that, in some cases, it is important. In such cases, what do you base your decissions on? Traditional wisdom? Hearsay? The Shootout? Your own benchmarks? The only true way to know is to implement your application in all languages you consider, and then measure which implementation is fastest. Failing that, I think the Shootout isn't such a bad source to base your decissions on. As has already been said; it's results feel mostly right. > I can say that for sure, because on many tests my > own programming languague Felix utterly creamed all the > others. So much so said Fascist Isaac Gouy banned it. To be quite honest, when you make a statement like the above, I'm more inclined to take Isaac's side than yours. If you try to make the world believe that your pet language beat all the others (which is hard to believe), and resort to calling names to make your case, I seriously start to wonder if you have any good arguments at all. > On the EH test, Felix ran in negative time, orders > of magnitude faster than the nearest competitor! Ok...so maybe you're kidding and this is meant to show that the Shootout doesn't measure right. > It trashes every available translator on Ackermann test, > including Ocaml and even gcc -- despite generating C++ > code processed by gcc :) On the 'threading' test it > again wiped everyone else out of the sky. Hmm, so either Felix generates pretty good C++ code, or the C++ entry for the Ackermann test was really lousy. > In all these cases, as the author of the compiler, > I KNOW the reason was because the benchmarks were > not reasonable. Felix is good, but not that good: > in all these cases the optimiser eliminated the very > feature that was supposed to be tested. In the EH test > no 'exceptions' got thrown, the optimiser reduced > them to a single machine instruction, a goto. > Same in the threading test .. it just happened to > use a chain of threads, and in Felix every procedure > is a thread.. but the chain was tail-rec and Felix > turned it into a loop. Ok, beneath the surface of what you say is a good point. We need to carefully consider what the programs in the benchmarks actually measure. In the examples you mention, apparently the test doesn't measure the performance of throwing exceptions, but rather the performance of language implementations when no exceptions are thrown. I say good job to your optimizer, and congratulations on wiping the floor with the other languages in the test! (I'll keep my earlier remark a few paragraphs back.) > Last time I looked, Java was one of the fastest > languages. Yet you say: > > > That is why the languages that we > > already know are slowest are at the bottom of the list > > and the fastest at the top. > > but we know Java is woefully slow. Hmmm. I really wouldn't say that with so much certainty. As far as I know, there are basically two camps. One camp claims that Java is blazing fast, the other that it's dog slow. Obviously, there's something fishy going on. Personally, but I admit my judgment is colored by the Shootout, I believe that _some_ Java implementations, including old ones by Sun, are dog slow, but the latest ones are pretty fast for the stuff they have been optimized for. A similar story goes for Lua: the pure interpreter isn't the fastest language implementation around, but the JIT implementation puts Lua along the faster languages in the Shootout (and yes, that's only on the things the benchmark measures). I think it's foolish to deny that Java could have gotten a lot faster over the years, although I must say that it still feels sluggy to me more often than not. But then, I do things the Unix way, with lots of little programs, which obviously isn't what Java is optimized for. > > Besides, one of the great strengths of the shootout is > > that it is active. Search for benchmarks on database > > systems, for example, and you cannot get to the end of > > one-off tests comparing MySQL latest to PostgreSQL 7 > > (older version). The shootout is a fair fight > > No it isn't. You're right, it isn't. But it's better than the example. > > and > > does establish an accurate general speed mark for the > > languages. > > Not even slightly. The results from the Shootout, at best, fall in the last category of "lies, damn lies, and statistics". I.e., at best, the results represent which language implementations did best at the tests in the Shootout. You seem to claim that they don't even do that, because they are wrongly executed. Fair enough. Do you have any better source for measurements? > > Does anyone look at the list and think, > > "there's no way OCaml is that fast"? > > Yes, absolutely. There is no way Java is anywhere > near as fast as the tests make out. So you say. There are others who would dispute that. Without any better data to go on, I'm inclined to give the Shootout the benefit of the doubt. > The test methodology is seriously flawed. > The rules about which ways you have to do things > are arbitrary. Can C use 'x' library or not? > Is C++ allowed to use templates to implement Ackermann? > [In which case it would cream absolutely everything, > since the whole of the computation would be done at > compile time, which is not counted in the results] I agree. Still, I think perhaps the way to approach the issue is by asking "How can we improve the Shootout?", rather than just bashing it. For all I know, the Shootout is the best we have. > Again, last I looked, the ranking of the languages > was just rubbish. Haskell does badly, and it is indeed > slow at some kinds of things, but it is very VERY good > at doing others. I know from someone close to the Haskell community that Haskell is now near the top in the Shootout results, because the Haskell developers optimized ghc to score well on the Shootout. Some may find that a Bad Thing. I say that this is one of the good things that has come out of the Shootout. It showed the developers areas in which their implementation was lacking, and they improved it. Now all we need to do is add more benchmarks, and other areas of the implementation will be improved. If the same happens for other implementations, everybody wins. > Similarly, if you read the Ocaml programming lists, > you'll see minor changes in code structure can have > significant performance implications. Of course. But everybody gets a chance to submit a better program to the Shootout. One can at least hope that this will result in near-optimal entries being used for calculating the results. Again, I say it's not perfect, but it may be the best we have. > A small number of micro benchmarks simply cannot > average out all these factors, and tests based > on implementing a particular algorithm are necessarily > rubbish. I argued strenuously -- along with Jon Harrop -- > the tests should be functional and independent of how > the result was calculated, but Isaac didn't like that > so he kicked both of us out. He removed Felix, after > I worked for months on the compiler and tests, and > he revoked Jon's CVS access to prevent him contributing > the kinds of tests he thought made more sense. I think both sides have a point. I would say functional tests are what you usually should be interested in, but it's also good to know the performance (and availability) of specific language features. Maybe I really _do_ want to know how long an array update takes, instead of knowing the performance of the fastest contributed in-place sorting algorithm. > Look at the results, you'll see Erlang seems slow .. > yet it is used heavily in Telco applications which > demand ultra high real time performance. Again, you make very good points, and indeed the Shootout is not perfect, but I wouldn't say it's completely without merit, and it may be the best we have. Secretly, of course, the purpose of the Shootout is not to measure language performance, but to ignite flame wars. Regards, Bob --- When Marriage is Outlawed, Only Outlaws will have Inlaws.
pgp9ImxnXgox4.pgp
Description: PGP signature
-- Neko : One VM to run them all (http://nekovm.org)
