Re: benchmarking against other languages
On 02/13/2011 11:16 PM, Guy Hulbert wrote: I am only interested in perl5, perl6 and C to start. I havent got the C or java languages to work yet. snip but re-organize the documentation a bit as I'm more interested in the framework than the specific results -- except for how perl6 compares with perl and C (for now). Tracking history and keeping results for different releases is not yet included. [snip] So can we conclude from this that perl6 is *really* slow right now ? Not a surprise. Early days yet. Richard
Re: benchmarking against other languages
Thanks for the encouragement. On 02/13/2011 11:02 PM, Will Coleda wrote: On Sun, Feb 13, 2011 at 12:27 PM, Richard Hainsworth rich...@rusrating.ru wrote: Before doing any more work on this benchmarking project, I was wondering if any one else thinks the effort is worth it. Assuming that perl6 here == rakudo on parrot, I and others on the parrot dev team would definitely find these helpful. (Be sure to use a parrot built with --optimize) It will be interesting to benchmark with both! The alioth repository contains parrot versions of the benchmarks. However, it appears they no longer run with current versions of parrot. Is there any possibility of someone revising them to get them working? Although it is early days, it seemed reasonable to start setting up a framework for benchmarking perl6 algorithms against implementations/releases and against other languages. Probably also want to include niecza, assuming the same program can run on both rakudo niecza. Definitely want to try different implementations of perl6. I havent got these set up yet. Current version of benchmarker not set up for tracking history.
Re: benchmarking against other languages
Guy Hulbert gwhulb...@eol.ca writes: I am interested first in developing a generic framework around the work already done for 'the benchmark game' (TBG*). I will pretend that I am starting from scratch and define a protocol for adding algorithms and exchanging information. I have been convinced that everything following has been done for TBG but some of it is obscure. The details are hidden in CVS. I'd like to set things up so everything is fully automated. Perl6 developers (and users :-) should be able to just run the benchmarks in a reasonable way (one which halts :-) after installing the latest rakudo release. Just a hint from my Perl5 benchmarking effort: I had to take quite some care to make benchmarking numbers stable. With every newly added library the numbers changed which made it difficult to have useful numbers over the lifetime of the benchmark suite itself. Feel free to look into http://cpansearch.perl.org/src/SCHWIGON/Benchmark-Perl-Formance-0.22/lib/Benchmark/Perl/Formance.pm and http://cpansearch.perl.org/src/SCHWIGON/Benchmark-Perl-Formance-0.22/ChangeLog especially around v0.20 to get some inspiration of why, how and when I forked plugins away, what I set to make a typical Linux system stable, etc., etc. It's all Perl5, but the collected ideas flew in from more general sources. If on the other side, you already have some more experiences on this topic I would love to hear comments and ideas to apply them to my Perl5 benchmarking. Kind regards, Steffen -- Steffen Schwigon s...@renormalist.net Dresden Perl Mongers http://dresden-pm.org/
Re: benchmarking against other languages
On 02/14/11 16:47, Guy Hulbert wrote: Some comments on what I plan to do after my 2 week hibernation. On Sun, 2011-13-02 at 20:27 +0300, Richard Hainsworth wrote: see http://shootout.alioth.debian.org/ for more info on the algorithms. There are many factors that can be considered when benchmarking, including IO and OS. It seemed to me it would be a good idea to fix on some elegant form of the perl6 implementation. By keeping the program the same, it will be possible to track how developments implementations affect speed/memory size. I am interested first in developing a generic framework around the work already done for 'the benchmark game' (TBG*). I will pretend that I am starting from scratch and define a protocol for adding algorithms and exchanging information. TBG was designed to test languages and assumes a stable implementation of the language. There is a possibility of different language implementations, but in a sense they are treated as different languages. Also the TBG tool allows for a single program to be altered and remeasured against a statically evolving language, rather than a change in the language and then measuring against a static program. This may be relevant for what Steffen wrote, as in a sense the modules are a part of the programing environment that changes. I have been convinced that everything following has been done for TBG but some of it is obscure. The details are hidden in CVS. Actually, all of the intelligence is hidden in the bencher/makefiles/xx.ini file Explanation is in the .ini and in the bencher/readme file. However, it can be very obscure. I am trying to get the c and cc working, as these require a compilation step prior to running the program. I'd like to set things up so everything is fully automated. Perl6 developers (and users :-) should be able to just run the benchmarks in a reasonable way (one which halts :-) after installing the latest rakudo release. Absolutely (A) Protocol to specify algorithm. 1. Define an algorithm and provide a reference for it. 2. Define standard inputs and implement algorithm in 2 languages. 3. Generate and verify outputs corresponding to inputs. 4. Make code, input and output available. TBG has directories for each benchmark, since the idea is to make it easy to add another language to a benchmark. Also a directory for input files and output files to be diffed against I think that we should have directories for each language, since the number of languages is small, and if a new implementation of a benchmark is added, we want to see when it appears in the historical record. Details can be posted on github and descriptions on the perl6 wiki. (B) Benchmark protocol per language. 1. Define hardware, operating system, language version. 2. Execute for a reasonable subset of inputs (some may be too slow or fast to be interesting). 3. Generate standard metrics (see alioth). Accumulate historical values Summaries can be posted on the perl6 wiki. It should be possible to extend the standard metrics. It should also be possible to filter them in standard ways to make results clearer. Collecting data should be separated from analysing the data. Given the above, I would just define a protocol to exchange results. One need only specify md5 sums to verify/identify input/output -- some of the algorithms in TBG have input and output so large that they are truncated in the results pages. In such cases publishing checksums (md5 is sufficient) of the results will be useful. I'm interested in autoamating B1 for other purposes. [*] Personally, I have nothing against 'shootout' but it does no harm to respect the wishes of the current maintainer of TBG on alioth.
Re: benchmarking against other languages
On Mon, 2011-14-02 at 15:56 +0100, Steffen Schwigon wrote: If on the other side, you already have some more experiences on this topic I would love to hear comments and ideas to apply them to my Perl5 benchmarking. Thanks Steffen. I will look at this when I get started in two weeks. If you have not done so already, it may be worth your time to read through the documentation on shootout.alioth.debian.org. I've spent the last 10 years using perl5 to learn OO. The most useful thing I've learned is that there are very few general rules and I think the same applies to benchmarking. It all depends on what you are trying to measure and why you are trying to measure it. -- --gh
Re: benchmarking against other languages
On Mon, 2011-14-02 at 18:16 +0300, Richard Hainsworth wrote: On 02/14/11 16:47, Guy Hulbert wrote: Some comments on what I plan to do after my 2 week hibernation. [snip] I am interested first in developing a generic framework around the work already done for 'the benchmark game' (TBG*). I will pretend that I am starting from scratch and define a protocol for adding algorithms and exchanging information. TBG was designed to test languages and assumes a stable implementation of the language. There is a possibility of different language implementations, but in a sense they are treated as different languages. Yes. I'd like to make things a bit more abstract. I want to remove their goals but keep the useful parts of their framework. [snip] Actually, all of the intelligence is hidden in the bencher/makefiles/xx.ini file Explanation is in the .ini and in the bencher/readme file. However, it can be very obscure. That's very useful to know. [snip] (A) Protocol to specify algorithm. 1. Define an algorithm and provide a reference for it. 2. Define standard inputs and implement algorithm in 2 languages. 3. Generate and verify outputs corresponding to inputs. 4. Make code, input and output available. TBG has directories for each benchmark, since the idea is to make it easy to add another language to a benchmark. I would change their structure but that should be independent of the protocol. Also a directory for input files and output files to be diffed against I think that we should have directories for each language, since the number of languages is small, and if a new implementation of a benchmark is added, we want to see when it appears in the historical record. Keeping a historical record is useful for perl6. I want to address the following use cases: 1. perl6 development 2. TBG 3. idle curiosity 4. adding algorithms By (3), I mean it should be as easy as: $ apt-get install BM $ apt-get install C C++ perl6 $ run-bm lang: .all algo: nbody algo: pidigits algo: .done ... run ... Algorithm C C++ perl6 nbody 1 1.2 1.5 pidigits 2 1.9 3.7 The program would download examples on request and automatically find installed languages. There would be a filtered command-line presentation and the user could build his own database of results. If you can do that then you can do pretty much anything you want. [snip] -- --gh
Re: benchmarking against other languages
On Sun, Feb 13, 2011 at 12:27 PM, Richard Hainsworth
rich...@rusrating.ru wrote:
Before doing any more work on this benchmarking project, I was wondering if
any one else thinks the effort is worth it.
Assuming that perl6 here == rakudo on parrot, I and others on the
parrot dev team would definitely find these helpful. (Be sure to use a
parrot built with --optimize)
Although it is early days, it seemed reasonable to start setting up a
framework for benchmarking perl6 algorithms against implementations/releases
and against other languages.
Probably also want to include niecza, assuming the same program can
run on both rakudo niecza.
The Alioth shootout is one benchmarking system, with thirteen benchmark
programs. I have converted one benchmark to perl6 (two programs, slightly
different). Gradually working on the remainder.
The results below compare perl, perl6, ruby, php, and python programs. (I
haven't got java or gcc to work yet). The input data are very small, since
perl6 is still weak.
Hopefully the table formatting below will come out properly.
name lang id n size(B) cpu(s) mem(KB)
status load elapsed(s)
nbody python 1 30 1329 0.01 0 0 % 0
nbody php 2 30 1333 0.01 0 0 % 0
nbody perl 1 30 1556 0.01 0 0 % 0
nbody ruby 2 30 1326 0 0 0 % 0
nbody php 3 30 1240 0.01 0 0 100% 100%
0.02
nbody python 4 30 1238 0.02 0 0 0% 100%
0.02
nbody php 1 30 1383 0.01 0 0 50% 100%
0.02
nbody perl6 2 30 1320 5.41 206776 0 100% 13%
5.42
nbody perl6 1 30 1394 5.64 208820 0 32% 89%
5.68
see http://shootout.alioth.debian.org/ for more info on the algorithms.
There are many factors that can be considered when benchmarking, including
IO and OS.
It seemed to me it would be a good idea to fix on some elegant form of the
perl6 implementation. By keeping the program the same, it will be possible
to track how developments implementations affect speed/memory size.
The perl6 program is given below. It still retains the form of the original
perl, the python version is quite different.
Richard
# The Computer Language Shootout
# http://shootout.alioth.debian.org/
#
# contributed by Christoph Bauer
# converted into Perl by Márton Papp
# converted to perl6 by Richard Hainsworth
my $solar-mass = (4 * pi * pi);
my $days-per-year = 365.24;
# Globals for arrays... Oh well.
# Globals introduced to speed up perl.
# @ns = ( sun, jupiter, saturn, uranus, neptune )
my @xs = (0, 4.84143144246472090e+00, 8.34336671824457987e+00,
1.28943695621391310e+01, 1.53796971148509165e+01);
my @ys = (0, -1.16032004402742839e+00, 4.12479856412430479e+00,
-1.5514016986312e+01, -2.59193146099879641e+01);
my @zs = (0, -1.03622044471123109e-01, -4.03523417114321381e-01,
-2.23307578892655734e-01, 1.79258772950371181e-01);
my @vxs = $days-per-year *
(0, 1.66007664274403694e-03, -2.76742510726862411e-03,
2.96460137564761618e-03, 2.68067772490389322e-03);
my @vys = $days-per-year *
(0, 7.69901118419740425e-03, 4.99852801234917238e-03,
2.37847173959480950e-03, 1.62824170038242295e-03);
my @vzs = $days-per-year *
(0, -6.90460016972063023e-05, 2.30417297573763929e-05,
-2.96589568540237556e-05, -9.51592254519715870e-05);
my @mass = $solar-mass *
(1, 9.54791938424326609e-04, 2.85885980666130812e-04,
4.36624404335156298e-05, 5.15138902046611451e-05);
# Almost every iteration is a range, so keep the last index rather than a
count.
my $last = +@xs;
@vxs[0] = - ( [+] @vxs Z* @mass ) / $solar-mass;
@vys[0] = - ( [+] @vys Z* @mass ) / $solar-mass;
@vzs[0] = - ( [+] @vzs Z* @mass ) / $solar-mass;
sub advance($dt)
{
my ($mm, $mm2, $j, $dx, $dy, $dz, $distance, $mag);
for ^$last {
for $_ ^..^ $last - $j {
$dx = @xs[$_] - @xs[$j];
$dy = @ys[$_] - @ys[$j];
$dz = @zs[$_] - @zs[$j];
$distance = sqrt($dx * $dx + $dy * $dy + $dz * $dz);
$mag = $dt / ($distance * $distance * $distance);
$mm = @mass[$_] * $mag;
$mm2 = @mass[$j] * $mag;
@vxs[$_] -= $dx * $mm2;
@vxs[$j] += $dx * $mm;
@vys[$_] -= $dy * $mm2;
@vys[$j] += $dy * $mm;
@vzs[$_] -= $dz * $mm2;
@vzs[$j] += $dz * $mm;
}
# We're done with planet $_ at this point
# This could be done in a seperate loop, but it's slower
@xs[$_] += $dt * @vxs[$_];
@ys[$_] += $dt * @vys[$_];
@zs[$_] += $dt * @vzs[$_];
}
}
sub energy
{
my ($dx, $dy, $dz, $distance);
my $e = 0.0;
for ^$last - $i {
$e += 0.5 * @mass[$i] *
(@vxs[$i] * @vxs[$i] + @vys[$i] * @vys[$i] + @vzs[$i] * @vzs[$i]);
for $i ^..^ $last {
$dx = @xs[$i] - @xs[$_];
$dy = @ys[$i] -
Re: benchmarking against other languages
On Sun, 2011-13-02 at 20:27 +0300, Richard Hainsworth wrote: Before doing any more work on this benchmarking project, I was wondering if any one else thinks the effort is worth it. Me. I hope to get started in a couple of weeks once my accounting is done. Although it is early days, it seemed reasonable to start setting up a framework for benchmarking perl6 algorithms against implementations/releases and against other languages. I am only interested in perl5, perl6 and C to start. The Alioth shootout is one benchmarking system, with thirteen benchmark The Alioth benchmarking game (current maintainer deprecates shootout) is a very well-defined system. I would like to start there but re-organize the documentation a bit as I'm more interested in the framework than the specific results -- except for how perl6 compares with perl and C (for now). One thing I like about it is that most of the algorithms are specified and referenced. So what's being compared are implementations in specific languages. programs. I have converted one benchmark to perl6 (two programs, slightly different). Gradually working on the remainder. The results below compare perl, perl6, ruby, php, and python programs. (I haven't got java or gcc to work yet). The input data are very small, since perl6 is still weak. Hopefully the table formatting below will come out properly. Unfortunately not too well. name langid n size(B) cpu(s) mem(KB) status loadelapsed(s) nbody python 1 30 13290.010 0 %0 nbody php 2 30 13330.010 0 %0 nbody perl1 30 15560.010 0 %0 [snip] nbody perl6 2 30 13205.41206776 0 100% 13%5.42 nbody perl6 1 30 13945.64208820 0 32% 89% 5.68 [snip] So can we conclude from this that perl6 is *really* slow right now ? -- --gh
