On Mon, 2011-14-02 at 18:16 +0300, Richard Hainsworth wrote: > On 02/14/11 16:47, Guy Hulbert wrote: > > Some comments on what I plan to do after my 2 week hibernation. [snip] > > I am interested first in developing a generic framework around the work > > already done for 'the benchmark game' (TBG*). I will pretend that I am > > starting from scratch and define a protocol for adding algorithms and > > exchanging information. > TBG was designed to test languages and assumes a stable implementation > of the language. There is a possibility of different language > implementations, but in a sense they are treated as different languages.
Yes. I'd like to make things a bit more abstract. I want to remove their goals but keep the useful parts of their framework. [snip] > Actually, all of the intelligence is hidden in the > bencher/makefiles/xxxxxx.ini file > Explanation is in the .ini and in the bencher/readme file. However, it > can be very obscure. That's very useful to know. [snip] > > (A) Protocol to specify algorithm. > > > > 1. Define an algorithm and provide a reference for it. > > 2. Define standard inputs and implement algorithm in 2 languages. > > 3. Generate and verify outputs corresponding to inputs. > > 4. Make code, input and output available. > TBG has directories for each benchmark, since the idea is to make it > easy to add another language to a benchmark. I would change their structure but that should be independent of the protocol. > > Also a directory for input files and output files to be diffed against > > I think that we should have directories for each language, since the > number of languages is small, and if a new implementation of a benchmark > is added, we want to see when it appears in the historical record. Keeping a historical record is useful for perl6. I want to address the following use cases: 1. perl6 development 2. TBG 3. idle curiosity 4. adding algorithms By (3), I mean it should be as easy as: $ apt-get install BM $ apt-get install C C++ perl6 $ run-bm lang: .all algo: nbody algo: pidigits algo: .done ... run ... Algorithm C C++ perl6 nbody 1 1.2 1.5 pidigits 2 1.9 3.7 The program would download examples on request and automatically find installed languages. There would be a filtered command-line presentation and the user could build his own database of results. If you can do that then you can do pretty much anything you want. [snip] -- --gh