On Sun, Jul 19, 2015 at 5:16 AM, Björn Dahlgren <bjo...@gmail.com> wrote: > Hi all, > > On Tuesday, 14 July 2015 02:49:57 UTC+2, Aaron Meurer wrote: >> >> - Get a benchmark machine and run airspeed velocity on it. We need to >> catch performance regressions. The benchmark suite can be anything, although >> obviously well-made benchmarks are better. > > > I found a few hours and threw this together (motivated by having noticed > some things I have been doing with sympy has been getting slower): > > https://github.com/bjodah/sympy_benchmarks_bjodah > > then, on a machine with asv 0.1.1, I ran: > > $ asv run ALL -s 100 > > which essensitally takes 100 equally spaced commits from the whle history > and run the benchmarks one those. > You can view the results here: > > http://hera.physchem.kth.se/~sympy_asv > > some commits failed, maybe I should be targetting the merge commits but I > haven't > figured out how to do that yet. > > Furthermore: the comparison isn't quite fair since I belive the later > versions return more correct results (Piecewise solutions), > I'm not sure how to deal with this fairly, maybe some kind of assertion > should be made (like a tests) at the end of the individual benchmark, > a passing assertion means it should be included, a failing one that it > should be excluded. But then this > assertion shouldn't be included in the timing. There could be something in > asv for this but I haven't figured it out yet. > > I am stil a bit new to using asv so I haven't quite figured out best > practives here. > It would be nice to have a dedicated machine (or maybe several), maybe a > Raspberry Pi 2 which > would make it easy for people to get the exact same hardware for > comparisons? > > Anyways, thought I would share. Feel free to fork/send pull requests, I'll > happily run them on that machine if there is an > interest.
Björn, thanks a lot for doing this! I need to setup something similar for SymEngine, I am currently benchmarking every commit that I think might slow things down by hand. I agree with everything that Aaron said. SymEngine is only fixing the "the speed as a side-effect of being in pure Python", i.e. with C++ we are much closer to the metal, so for low level representations of the symbolic expressions we have more freedom to nail the speed, but algorithmically it's the same issue as with SymPy and everything that Aaron said applies to SymEngine as well. We can buy a dedicated machine via NumFOCUS, SymPy should have enough money there, but I don't have time personally to make it happen. If anybody is interested, let me or Aaron now. Ondrej -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBBSj_BRVzPGubknQYmAvEaGLYhH%2BK4B7dXR90szteXNg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
