> From: wren ng thornton <w...@freegeek.org> > Sent: Saturday, May 19, 2012 12:49 AM > Subject: Re: [Haskell-cafe] Can Haskell outperform C++?
-snip- > "Fair" in what sense? That is, what _exactly_ are you hoping to > compare? > > If the goal is to benchmark the implementation of the runtime, VM, or > built-in > types, then requiring the same algorithm makes sense--- because the algorithm > is > irrelevant other than to provide a bunch of calls to the runtime/vm/etc. > However, benchmarking a language's implementation in this way is rarely that > helpful. It's great for comparing CPython to PyPy (or any other in-language > cross-compiler comparison), but what would it tell you about Haskell vs C++? The PyPy crowd won't like it if you take programs written for CPython and measure how they run with PyPy - that's "not fair". But it might take a couple of years before they contribute programs optimised for PyPy :-( (You already said what it would tell you, but questioned how helpful that would be.) > If the goal is to compare, say, production costs for a given level of > performance, then fixing the algorithm is not at all fair. The fact of the > matter is that different languages make different algorithms easier to (a) > implement, and (b) discover/identify/generalize. Thus, when it comes to > real-world software, the language that makes it easy to implement good > algorithms has a major advantage--- an advantage which is being specifically > ignored by fixing the algorithm aforehand. Let's just say that's true - Is it useful? What would we need to do to make the comparison? We could do something like - "Plat_Forms: Is there a single best web development technology? A professional programming contest" http://page.mi.fu-berlin.de/prechelt/Biblio/platforms07-cacm-2010.pdf But that was just once, with very few teams, and only one problem -- seems like it would need to be repeated more often than is affordable, and with more teams than can be persuaded to donate their time. Maybe your point is true but practically useless? :-( > In order for "fair" to have any meaning whatsoever, we must first > specify what is being compared, so that we know what it is that things are > supposed to be fair with regard to. 'What does "not fair" mean? (A fable)' http://stackoverflow.com/a/6380299 _______________________________________________ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
