Hello!
Olivier Dion <[email protected]> skribis:
> For example, I often find myself modifying the compiler. I pay close
> attention to the size of the emitted code on disk like so:
>
> $ git checkout main
> $ make
> $ find stage2 -iname "*.go" -printf "%f %s\n" > then
> $ git cherry-pick ...
> $ make
> $ find stage2 -iname "*.go" -printf "%f %s\n" > now
> $ sdiff -s then now | awk '{ printf("%s\t%+d\n", $1, $5 - $2) }' | column -t
>
> This gives me something like:
>
> ice-9/boot-9.go +272
> ice-9/i18n.go -72
> ice-9/local-eval.go +160
> ...
Nice trick!
> There are other kinds of metrics which are interesting to have. For
> example:
>
> $ hyperfine --export-csv load-time.csv "guile -c '(use-modules (rnrs
> base))'" > dev/null
> Benchmark 1: guile -c '(use-modules (rnrs base))'
> Time (mean ± σ): 9.6 ms ± 0.3 ms [User: 5.9 ms, System: 4.3 ms]
> Range (min … max): 8.6 ms … 11.5 ms 241 runs
>
> $ cat load-tim.csv
> command,mean,stddev,median,user,system,min,max
> guile -c '(use-modules (rnrs
> base))',0.009666700473662545,0.00034979357393589316,0.009594730700000002,0.005276396707818929,0.004960339588477369,0.0088011947,0.0111349187
>
> I think it would be nice to track these kind of metrics and expose them
> through a web UI. I know that the Zig language does that [0]. What
> would also be nice is for the maintainers of Guile to be notified about
> potential performance regressions.
I agree it would be nice to have.
In terms of existing infrastructure, Cuirass is not customizable right
now: you could not have it draw plots of individual .go file sizes. It
could keep track of package sizes (Hydra, the inspiration for Cuirass,
does this), but that’s probably too coarse-grain for your needs.
As for run-time performance, I think it needs a dedicated machine (you
always want to run your benchmarks on the same machine, without anything
else running on it at the same time), and appropriate tools. In HPC,
ReFrame and JUBE are quite widespread for this, but I’m sure there are
many others.
I would suggest asking Guix Foundation for a dedicated machine to do
that. It doesn’t need to be powerful or anything; it just needs to be
stable and kept around long enough (a few years at least). There are
even laptops that were donated to Guix Foundation that would be good
enough for the job.
Thoughts?
Ludo’.
