2016-09-01 19:53 GMT+02:00 Kevin Modzelewski <k...@dropbox.com>: > Just my two cents -- having a benchmark change underneath the benchmark > runner is quite confusing to debug, because it looks indistinguishable from > a non-reproducible regression that happens in the performance itself.
I agree. That's why I proposed to use semantic versionning. I'm not sure that old results must be removed. We should just be explicit about versions. The main issue is when you *compare* two results produced by two different performance versions. I have an item in my TODO list to emit a warning if the exact version (minor version) is different, and display an error if the major version is different. About reproductability: I made another change in the development version, indirect dependencies are now pinned as well: https://github.com/python/performance/blob/master/performance/requirements.txt#L15 It should help to have a more reproductible benchmark ;-) The last known issue about reproductability is that I dropped the code to remove environment variables. I should fix this in the perf module directly. Interesting link: https://reproducible-builds.org/ > Another thing to consider is that there will be other people using this > benchmark set than just the codespeed setup: there will be long-lived > benchmark results in the form of blogs and academic papers. I think it's > important to get some good wording about having to include the version of > the benchmarks when publishing results, and then it would be good to follow > that advice internally as well. The perf module has a feature for that: it supports storing metadata in JSON files. I modified many benchmarks to store dependency version (like Django verison), but now also store performance version (since performance 0.2). The perf module stores its own version by default ;-) I suggest to store JSON file rather than only the compact text output (which contains much less information). Victor _______________________________________________ Speed mailing list Speed@python.org https://mail.python.org/mailman/listinfo/speed
