On Wed, 14 Oct 2020 at 19:12, Ivan Pozdeev via Python-Dev <python-dev@python.org> wrote: > > > On 14.10.2020 17:04, M.-A. Lemburg wrote: > > On 14.10.2020 16:00, Pablo Galindo Salgado wrote: > >>> Would it be possible to get the data for older runs back, so that > >> it's easier to find the changes which caused the slowdown ? > >> > >> Unfortunately no. The reasons are that that data was misleading because > >> different points were computed with a different version of pyperformance > >> and > >> therefore with different packages (and therefore different code). So the > >> points > >> could not be compared among themselves. > >> > >> Also, past data didn't include 3.9 commits because the data gathering was > >> not > >> automated and it didn't run in a long time :( > > Make sense. > > > > Would it be possible rerun the tests with the current > > setup for say the last 1000 revisions or perhaps a subset of these > > (e.g. every 10th revision) to try to binary search for the revision which > > introduced the change ? > > Git has a facility for this called "bisection". It can be run in automated > mode, exactly for a scenario like this.
As Victor mentioned earlier, asv (airspeed velocity) has this feature as "asv find" and it can run a benchmark and has some heuristics to bisect when a "slowdown" occurred. In some situations that is very useful for identifying a commit that introduced a performance regression but in other cases there can be a number of changes that have different effects and the timings can be noisy. This can mean that the bisection doesn't converge properly to a relevant commit. Often there's really no substitute for running over all commits and being able to see how the benchmark changes over time. You can do this with asv and then the web interface gives a timeseries where the data-points link directly to the commits on github. I'm currently preparing the SymPy 1.7 release so Aaron Meurer recently ran asv over all commits from 1.6 to master and you can see the output here: https://www.asmeurer.com/sympy_benchmarks/ -- Oscar _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YU5KOX7SHMSGUO4B3PIOZFMKA6S5GJ22/ Code of Conduct: http://python.org/psf/codeofconduct/
