On Mon, 2011-14-02 at 18:16 +0300, Richard Hainsworth wrote:
> On 02/14/11 16:47, Guy Hulbert wrote:
> > Some comments on what I plan to do after my 2 week hibernation.
> > I am interested first in developing a generic framework around the work
> > already done for 'the benchmark game' (TBG*).  I will pretend that I am
> > starting from scratch and define a protocol for adding algorithms and
> > exchanging information.
> TBG was designed to test languages and assumes a stable implementation 
> of the language. There is a possibility of different language 
> implementations, but in a sense they are treated as different languages.

Yes.  I'd like to make things a bit more abstract.  I want to remove
their goals but keep the useful parts of their framework.

> Actually, all of the intelligence is hidden in the 
> bencher/makefiles/xxxxxx.ini file
> Explanation is in the .ini and in the bencher/readme file. However, it 
> can be very obscure.

That's very useful to know.

> > (A) Protocol to specify algorithm.
> >
> > 1. Define an algorithm and provide a reference for it.
> > 2. Define standard inputs and implement algorithm in 2 languages.
> > 3. Generate and verify outputs corresponding to inputs.
> > 4. Make code, input and output available.
> TBG has directories for each benchmark, since the idea is to make it 
> easy to add another language to a benchmark.

I would change their structure but that should be independent of the

> Also a directory for input files and output files to be diffed against
> I think that we should have directories for each language, since the 
> number of languages is small, and if a new implementation of a benchmark 
> is added, we want to see when it appears in the historical record.

Keeping a historical record is useful for perl6.  I want to address the
following use cases:

 1. perl6 development
 2. TBG
 3. idle curiosity
 4. adding algorithms

By (3), I mean it should be as easy as:

        $ apt-get install BM
        $ apt-get install C C++ perl6
        $ run-bm
        lang: .all
        algo: nbody
        algo: pidigits
        algo: .done
        ... run ...
        Algorithm  C  C++  perl6
        nbody      1  1.2  1.5
        pidigits   2  1.9  3.7

The program would download examples on request and automatically find
installed languages.  There would be a filtered command-line
presentation and the user could build his own database of results.

If you can do that then you can do pretty much anything you want.


Reply via email to