FYI, this database is building up: http://www.moorepants.info/misc/sympy-asv/ (I'm running this with the sympy cache set to 10000).
For the integration benchmarks it has every single functioning commit and maybe half the commits for the other benchmarks. The repository to submit benchmarks to is now at: https://github.com/sympy/sympy_benchmarks Please submit benchmarks for core parts of SymPy that you work on. Jason moorepants.info +01 530-601-9791 On Tue, Jul 21, 2015 at 12:16 PM, Ondřej Čertík <ondrej.cer...@gmail.com> wrote: > On Mon, Jul 20, 2015 at 11:29 PM, Ondřej Čertík <ondrej.cer...@gmail.com> > wrote: > > It's because I didn't have fastcache installed.... After installing > > it, by default I got: > > > > certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py > > Setup > > Converting to SymEngine... > > SymPy Jacobian: > > Total time: 0.123499155045 s > > SymEngine Jacobian: > > Total time: 0.00305485725403 s > > > > Speedup: 40.43x > > > > > > (I am running some longer calculation at the same time, so the timing > > is a bit different, but it got faster.) > > But bigger cache doesn't seem to affect it, as you noted: > > > > certik@redhawk:~/repos/symengine/benchmarks(py)$ > > SYMPY_CACHE_SIZE=50000 python kane2.py > > Setup > > Converting to SymEngine... > > SymPy Jacobian: > > Total time: 0.121557950974 s > > SymEngine Jacobian: > > Total time: 0.00302696228027 s > > > > Speedup: 40.16x > > > > > > > > More interestingly, I can now differentiate the actual bicycle > > expression. So far I just took one element on the right hand side > > vector, and differentiate with respect to q5. I substitute for symbols > > first (in both SymPy and SymEngine). This is what I got: > > > > certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane3.py > > Converting to SymPy... > > Converting to SymEngine... > > Done. > > Subs > > Eval initial expression > > -14960.1119361 > > -14960.111936127382563475771853345057230700491739739146860248076 + > > 0.00000000000000000000000000000000000000000000000000000000000000*I > > SymPy diff: > > Total time: 3.90587186813 s > > SymEngine diff: > > Total time: 0.0883350372314 s > > > > Speedup: 44.22x > > > > Subs > > Eval derivative > > -31789.936485 > > -31789.936484889832248148585748003998439133612431284784993412159 + > > 0.00000000000000000000000000000000000000000000000000000000000000*I > > Converting SymPy derivative to SymEngine... > > Subs > > Eval derivative > > -31789.936485 > > -31789.936484889832248148585748003998439133612431284784993279022 + > > 0.00000000000000000000000000000000000000000000000000000000000000*I > > > > > > The floating point numbers are calculated by substituting integers for > > the symbols and evaluating. The shorter number is using machine double > > precision, the longer number is using MPC (evaluates in complex plane, > > to make sure we don't get any negative values under square roots). > > They both agree as you can see. The initial expression number I > > checked against sympy (substituting the numbers using sympy and > > evaluating using sympy). For the derivatives, currently I just the > > sympy derivative to symengine to evaluate as a floating point, since > > sympy is just too slow (it's been running for 30 minutes -- it's stuck > > at the .n() method in sympy). The SymEngine evaluation is immediate > > It eventually finished, and the floating point number calculated using > pure SymPy was exactly the same as with SymEngine. So I trust the > derivatives and conversion back and forth by now for this particular > expression. > > I don't know how long it took, probably a few hours (compared to less > than 1s in symengine, so clearly the .n() algorithm is not the most > efficient, it probably has to do with the fact that it automatically > checks the accuracy --- showing that a better way to do that is just > keep evaluating at increasing precision and see if it converges, or > use something like Arb). > > Overall I am very happy about this progress, we have a robust > foundation to build upon. I'll try to get this very benchmark working > with Sage to get an idea how long it takes there. The branch with > symengine+sympy code is here: > https://github.com/sympy/symengine/compare/master...certik:py, it's > not very polished, that I will do later. > > Ondrej > > > for double precision and a fraction of a second for the MPC. But the > > conversion between symengine and sympy works well and I tested the > > initial expression in both, so I think this part works. And the fact > > that sympy derivative calculated using sympy and converted to > > symengine and symengine derivative agrees I think proves with pretty > > high confidence that things work. > > > > Now the next step is to calculate the whole jacobian and again test > > that we get the same expression. It's very important to spend time and > > check the numerical evaluation of the expression, since there could be > > bugs in symengine. I just fixed one here: > > https://github.com/sympy/symengine/pull/556 (it was giving me > > different numbers for the initial expression, but now they agree). > > This was a bug in subs, but there could be a bug in the derivative as > > well. > > > > After we get the same expressions, we can start benchmarking. I can > > start profiling the derivative code in C++ and see if it is possible > > to speed it up. I can also implement cache in symengine, though I > > don't know if it would speed up anything here. Still, even if we only > > get 45x speedup, then instead of 4 minutes, the whole bicycle should > > take 5s. So I think that's a big deal. I will also try to benchmark > > this against other software like Sage to get an idea how we are doing. > > > > Ondrej > > > > On Mon, Jul 20, 2015 at 11:03 PM, Jason Moore <moorepa...@gmail.com> > wrote: > >> Ondrej, > >> > >> I'm not sure why you don't see performance increase with increased > cache. > >> The following shows that the benchmarks do run faster with a large > cache. > >> Interestingly the memory doesn't seem to change (but I'm not sure I > >> understand how they measure mem usage). Notice that the jacobian wrt to > >> symbols is not affected by the cache size but the jacobians wrt to the > >> functions is, as is substitution. It seems that .diff() calls subs a lot > >> (see https://github.com/sympy/sympy/issues/9701), so maybe the slow > down is > >> all due to the caching used for subs. > >> > >> moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ asv > >> --config asv.conf.conda.json run > >> · Fetching recent changes. > >> · Creating environments > >> · Discovering benchmarks > >> ·· Uninstalling from py2.7-fastcache-mpmath. > >> ·· Building for py2.7-fastcache-mpmath > >> ·· Installing into py2.7-fastcache-mpmath.. > >> · Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks) > >> [ 0.00%] · For sympy commit hash 488f3c20: > >> [ 0.00%] ·· Building for py2.7-fastcache-mpmath... > >> [ 0.00%] ·· Benchmarking py2.7-fastcache-mpmath > >> [ 11.11%] ··· Running integrate.TimeIntegration01.time_doit > >> 365.73ms > >> [ 22.22%] ··· Running integrate.TimeIntegration01.time_doit_meijerg > >> 107.41ms > >> [ 33.33%] ··· Running > >> large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions > >> 37M > >> [ 44.44%] ··· Running > >> large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols > >> 37M > >> [ 55.56%] ··· Running > large_exprs.TimeLargeExpressionOperations.peakmem_subs > >> 37M > >> [ 66.67%] ··· Running > >> large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions > >> 1.75s > >> [ 77.78%] ··· Running > >> large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols > >> 35.49ms > >> [ 88.89%] ··· Running > >> > large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions > >> 1.79s > >> [100.00%] ··· Running > large_exprs.TimeLargeExpressionOperations.time_subs > >> 1.66s > >> moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ > >> SYMPY_CACHE_SIZE=10000 asv --config asv.conf.conda.json run > >> · Fetching recent changes. > >> · Creating environments > >> · Discovering benchmarks > >> ·· Uninstalling from py2.7-fastcache-mpmath. > >> ·· Building for py2.7-fastcache-mpmath > >> ·· Installing into py2.7-fastcache-mpmath.. > >> · Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks) > >> [ 0.00%] · For sympy commit hash 488f3c20: > >> [ 0.00%] ·· Building for py2.7-fastcache-mpmath... > >> [ 0.00%] ·· Benchmarking py2.7-fastcache-mpmath > >> [ 11.11%] ··· Running integrate.TimeIntegration01.time_doit > >> 361.81ms > >> [ 22.22%] ··· Running integrate.TimeIntegration01.time_doit_meijerg > >> 104.40ms > >> [ 33.33%] ··· Running > >> large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions > >> 37M > >> [ 44.44%] ··· Running > >> large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols > >> 37M > >> [ 55.56%] ··· Running > large_exprs.TimeLargeExpressionOperations.peakmem_subs > >> 37M > >> [ 66.67%] ··· Running > >> large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions > >> 56.62ms > >> [ 77.78%] ··· Running > >> large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols > >> 33.04ms > >> [ 88.89%] ··· Running > >> > large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions > >> 55.25ms > >> [100.00%] ··· Running > large_exprs.TimeLargeExpressionOperations.time_subs > >> 13.89ms > >> > >> > >> > >> Jason > >> moorepants.info > >> +01 530-601-9791 > >> > >> On Mon, Jul 20, 2015 at 6:57 PM, Ondřej Čertík <ondrej.cer...@gmail.com > > > >> wrote: > >>> > >>> On Mon, Jul 20, 2015 at 7:48 PM, Ondřej Čertík < > ondrej.cer...@gmail.com> > >>> wrote: > >>> > On Sun, Jul 19, 2015 at 4:57 PM, Jason Moore <moorepa...@gmail.com> > >>> > wrote: > >>> >> I just tried this out with jacobian() and subs() over the commits > since > >>> >> 0.7.3 to master. It's showing me that the new caching is the killer > >>> >> slowdown: > >>> >> > >>> >> https://github.com/sympy/sympy/commit/a63005e4 > >>> >> > >>> >> I've submitted a PR to Björn's repo: > >>> >> https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files > >>> >> > >>> >> I'm assuming it uses fastcache because I have it installed, but > maybe > >>> >> not. > >>> >> I'm not sure how to control for dependencies yet. > >>> > > >>> > I modified your benchmark to first do the substitution for symbols, > >>> > only then do the Jacobian: > >>> > > >>> > > >>> > > https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c > >>> > > >>> > Here are the results: > >>> > > >>> > certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py > >>> > Setup > >>> > Converting to SymEngine... > >>> > SymPy Jacobian: > >>> > Total time: 0.155253887177 s > >>> > SymEngine Jacobian: > >>> > Total time: 0.00277400016785 s > >>> > > >>> > Speedup: 55.97x > >>> > > >>> > > >>> > I played with the cache size, it makes no difference. Your original > >>> > benchmark ran about 1s on my machine, with SYMPY_CACHE_SIZE=5000 it > >>> > took about 0.9s. So you can speedup SymPy itself just by using > symbols > >>> > instead of functions by a factor of 5.8x. Btw, I tested that the > >>> > result that I get from symengine is exactly the same as in sympy > >>> > > >>> > ( > https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c#diff-e423f8edd8641c702513b2ea9c10eaecR32 > ). > >>> > > >>> > > >>> > How did you generate the benchmark? Can you generate a bit larger > >>> > matrix? Once the symengine benchmark runs in about 1s, then I can try > >>> > to speed it up (because I would see the result in the timing more > >>> > easily). > >>> > >>> I actually found a much larger benchmark, where just a simple > >>> differentiation of a single entry (after substituting functionsf or > >>> symbols) takes 60s for sympy. > >>> > >>> > >>> > https://github.com/sympy/symengine/commit/31378da77e5463870b9a22fda7a17deb68068fa8 > >>> > >>> Here is where I am saving the matrix: > >>> > >>> > >>> > https://github.com/sympy/symengine/commit/31378da77e5463870b9a22fda7a17deb68068fa8#diff-39829995ea91876984e13e3b828d740aR252 > >>> > >>> Does this make sense? Is that a good benchmark? I love the size, I > >>> just want to make sure I am saving the right thing. Is that the full > >>> bicycle? > >>> > >>> Ondrej > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups > >>> "sympy" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an > >>> email to sympy+unsubscr...@googlegroups.com. > >>> To post to this group, send email to sympy@googlegroups.com. > >>> Visit this group at http://groups.google.com/group/sympy. > >>> To view this discussion on the web visit > >>> > https://groups.google.com/d/msgid/sympy/CADDwiVDZ_JSgC-C_j5J0zmSOXS9TAQsGFv_XcmbBvx3ijPhPrg%40mail.gmail.com > . > >>> For more options, visit https://groups.google.com/d/optout. > >> > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "sympy" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to sympy+unsubscr...@googlegroups.com. > >> To post to this group, send email to sympy@googlegroups.com. > >> Visit this group at http://groups.google.com/group/sympy. > >> To view this discussion on the web visit > >> > https://groups.google.com/d/msgid/sympy/CAP7f1AhavPNhXzdhUDUVH1ihgDgbc4FbdmRPBHrMW9HQ4ijZDA%40mail.gmail.com > . > >> > >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to sympy+unsubscr...@googlegroups.com. > To post to this group, send email to sympy@googlegroups.com. > Visit this group at http://groups.google.com/group/sympy. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CADDwiVCB9%2B8EmZjEO%3DTH9ihU3eD-aGdKP7fymKnsXh3Y5bjsPA%40mail.gmail.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AjVG-myP9ebB-fv7vzgxeudwFyA4z4sD5FzEfrKJT_Qog%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
