#5793: make nofib not suck
------------------------------+---------------------------------------------
Reporter: dterei | Owner: dterei
Type: task | Status: new
Priority: normal | Component: NoFib benchmark suite
Version: | Keywords:
Os: Unknown/Multiple | Architecture: Unknown/Multiple
Failure: None/Unknown | Testcase:
Blockedby: | Blocking: 5794
Related: |
------------------------------+---------------------------------------------
Description changed by dterei:
Old description:
> fibon is the standard tool GHC developers use to benchmark changes to the
> compiler. Its a known fact though that fibon sucks. Its overall design is
> OK but it's had no love and care for many years and has bittrotted into
> near uselessness in a lot of situations.
>
> This task isn't to implement an awesome, really useful benchmark suite
> for Haskell. That would be great but is a huge undertaking. This task is
> just about making fibon useful again.
>
> The breakdown for this is something like:
>
> 1. Think and maybe fix fibon framework design. It has 'ways' which I
> think correspond to compilation method but more in the sense of 'dynamic'
> vs 'static', seems it may not suite being able to use ways for 'fasm' vs
> 'fllvm'. There is also the concept of 'modes' which corresponds to
> different benchmark input. So 'normal' and 'slow' for getting different
> run-times. At moment no easy way to select which benchmark groups to run,
> so may want to change that. I guess we should just decide, what knobs do
> we want to be able to easily tweak, and see how well the current design
> allows that.
>
> 2. Fixup the runtimes for benchmarks to be significant. (A lot of them
> run in < 1 or < 0.1 seconds). I think trying to get all of them to run in
> around 10s is a good target.
>
> 3. Above task is to fix normal but we may want to fixup slow as well and
> perhaps add a 'fast' mode where benchmarks run in around 1 second.
>
> 4. Maybe add more benchmarks to the suite (text, bytestring, performance
> regressions from ghc testsuite, vector....)
New description:
Nofib is the standard tool GHC developers use to benchmark changes to the
compiler. Its a known fact though that nofib sucks. Its overall design is
OK but it's had no love and care for many years and has bittrotted into
near uselessness in a lot of situations.
This task isn't to implement an awesome, really useful benchmark suite for
Haskell. That would be great but is a huge undertaking. This task is just
about making nofib useful again.
The breakdown for this is something like:
1. Think and maybe fix nofib framework design. It has 'ways' which I think
correspond to compilation method but more in the sense of 'dynamic' vs
'static', seems it may not suite being able to use ways for 'fasm' vs
'fllvm'. There is also the concept of 'modes' which corresponds to
different benchmark input. So 'normal' and 'slow' for getting different
run-times. At moment no easy way to select which benchmark groups to run,
so may want to change that. I guess we should just decide, what knobs do
we want to be able to easily tweak, and see how well the current design
allows that.
2. Fixup the runtimes for benchmarks to be significant. (A lot of them run
in < 1 or < 0.1 seconds). I think trying to get all of them to run in
around 10s is a good target.
3. Above task is to fix normal but we may want to fixup slow as well and
perhaps add a 'fast' mode where benchmarks run in around 1 second.
4. Maybe add more benchmarks to the suite (text, bytestring, performance
regressions from ghc testsuite, vector....)
--
--
Ticket URL: <http://hackage.haskell.org/trac/ghc/ticket/5793#comment:5>
GHC <http://www.haskell.org/ghc/>
The Glasgow Haskell Compiler
_______________________________________________
Glasgow-haskell-bugs mailing list
[email protected]
http://www.haskell.org/mailman/listinfo/glasgow-haskell-bugs
