Nice work!
Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 10:29 PM, Ondřej Čertík <[email protected]> wrote: > It's because I didn't have fastcache installed.... After installing > it, by default I got: > > certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py > Setup > Converting to SymEngine... > SymPy Jacobian: > Total time: 0.123499155045 s > SymEngine Jacobian: > Total time: 0.00305485725403 s > > Speedup: 40.43x > > > (I am running some longer calculation at the same time, so the timing > is a bit different, but it got faster.) > But bigger cache doesn't seem to affect it, as you noted: > > certik@redhawk:~/repos/symengine/benchmarks(py)$ > SYMPY_CACHE_SIZE=50000 python kane2.py > Setup > Converting to SymEngine... > SymPy Jacobian: > Total time: 0.121557950974 s > SymEngine Jacobian: > Total time: 0.00302696228027 s > > Speedup: 40.16x > > > > More interestingly, I can now differentiate the actual bicycle > expression. So far I just took one element on the right hand side > vector, and differentiate with respect to q5. I substitute for symbols > first (in both SymPy and SymEngine). This is what I got: > > certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane3.py > Converting to SymPy... > Converting to SymEngine... > Done. > Subs > Eval initial expression > -14960.1119361 > -14960.111936127382563475771853345057230700491739739146860248076 + > 0.00000000000000000000000000000000000000000000000000000000000000*I > SymPy diff: > Total time: 3.90587186813 s > SymEngine diff: > Total time: 0.0883350372314 s > > Speedup: 44.22x > > Subs > Eval derivative > -31789.936485 > -31789.936484889832248148585748003998439133612431284784993412159 + > 0.00000000000000000000000000000000000000000000000000000000000000*I > Converting SymPy derivative to SymEngine... > Subs > Eval derivative > -31789.936485 > -31789.936484889832248148585748003998439133612431284784993279022 + > 0.00000000000000000000000000000000000000000000000000000000000000*I > > > The floating point numbers are calculated by substituting integers for > the symbols and evaluating. The shorter number is using machine double > precision, the longer number is using MPC (evaluates in complex plane, > to make sure we don't get any negative values under square roots). > They both agree as you can see. The initial expression number I > checked against sympy (substituting the numbers using sympy and > evaluating using sympy). For the derivatives, currently I just the > sympy derivative to symengine to evaluate as a floating point, since > sympy is just too slow (it's been running for 30 minutes -- it's stuck > at the .n() method in sympy). The SymEngine evaluation is immediate > for double precision and a fraction of a second for the MPC. But the > conversion between symengine and sympy works well and I tested the > initial expression in both, so I think this part works. And the fact > that sympy derivative calculated using sympy and converted to > symengine and symengine derivative agrees I think proves with pretty > high confidence that things work. > > Now the next step is to calculate the whole jacobian and again test > that we get the same expression. It's very important to spend time and > check the numerical evaluation of the expression, since there could be > bugs in symengine. I just fixed one here: > https://github.com/sympy/symengine/pull/556 (it was giving me > different numbers for the initial expression, but now they agree). > This was a bug in subs, but there could be a bug in the derivative as > well. > > After we get the same expressions, we can start benchmarking. I can > start profiling the derivative code in C++ and see if it is possible > to speed it up. I can also implement cache in symengine, though I > don't know if it would speed up anything here. Still, even if we only > get 45x speedup, then instead of 4 minutes, the whole bicycle should > take 5s. So I think that's a big deal. I will also try to benchmark > this against other software like Sage to get an idea how we are doing. > > Ondrej > > On Mon, Jul 20, 2015 at 11:03 PM, Jason Moore <[email protected]> > wrote: > > Ondrej, > > > > I'm not sure why you don't see performance increase with increased cache. > > The following shows that the benchmarks do run faster with a large cache. > > Interestingly the memory doesn't seem to change (but I'm not sure I > > understand how they measure mem usage). Notice that the jacobian wrt to > > symbols is not affected by the cache size but the jacobians wrt to the > > functions is, as is substitution. It seems that .diff() calls subs a lot > > (see https://github.com/sympy/sympy/issues/9701), so maybe the slow > down is > > all due to the caching used for subs. > > > > moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ asv > > --config asv.conf.conda.json run > > · Fetching recent changes. > > · Creating environments > > · Discovering benchmarks > > ·· Uninstalling from py2.7-fastcache-mpmath. > > ·· Building for py2.7-fastcache-mpmath > > ·· Installing into py2.7-fastcache-mpmath.. > > · Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks) > > [ 0.00%] · For sympy commit hash 488f3c20: > > [ 0.00%] ·· Building for py2.7-fastcache-mpmath... > > [ 0.00%] ·· Benchmarking py2.7-fastcache-mpmath > > [ 11.11%] ··· Running integrate.TimeIntegration01.time_doit > > 365.73ms > > [ 22.22%] ··· Running integrate.TimeIntegration01.time_doit_meijerg > > 107.41ms > > [ 33.33%] ··· Running > > large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions > > 37M > > [ 44.44%] ··· Running > > large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols > > 37M > > [ 55.56%] ··· Running > large_exprs.TimeLargeExpressionOperations.peakmem_subs > > 37M > > [ 66.67%] ··· Running > > large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions > > 1.75s > > [ 77.78%] ··· Running > > large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols > > 35.49ms > > [ 88.89%] ··· Running > > > large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions > > 1.79s > > [100.00%] ··· Running large_exprs.TimeLargeExpressionOperations.time_subs > > 1.66s > > moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ > > SYMPY_CACHE_SIZE=10000 asv --config asv.conf.conda.json run > > · Fetching recent changes. > > · Creating environments > > · Discovering benchmarks > > ·· Uninstalling from py2.7-fastcache-mpmath. > > ·· Building for py2.7-fastcache-mpmath > > ·· Installing into py2.7-fastcache-mpmath.. > > · Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks) > > [ 0.00%] · For sympy commit hash 488f3c20: > > [ 0.00%] ·· Building for py2.7-fastcache-mpmath... > > [ 0.00%] ·· Benchmarking py2.7-fastcache-mpmath > > [ 11.11%] ··· Running integrate.TimeIntegration01.time_doit > > 361.81ms > > [ 22.22%] ··· Running integrate.TimeIntegration01.time_doit_meijerg > > 104.40ms > > [ 33.33%] ··· Running > > large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions > > 37M > > [ 44.44%] ··· Running > > large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols > > 37M > > [ 55.56%] ··· Running > large_exprs.TimeLargeExpressionOperations.peakmem_subs > > 37M > > [ 66.67%] ··· Running > > large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions > > 56.62ms > > [ 77.78%] ··· Running > > large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols > > 33.04ms > > [ 88.89%] ··· Running > > > large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions > > 55.25ms > > [100.00%] ··· Running large_exprs.TimeLargeExpressionOperations.time_subs > > 13.89ms > > > > > > > > Jason > > moorepants.info > > +01 530-601-9791 > > > > On Mon, Jul 20, 2015 at 6:57 PM, Ondřej Čertík <[email protected]> > > wrote: > >> > >> On Mon, Jul 20, 2015 at 7:48 PM, Ondřej Čertík <[email protected] > > > >> wrote: > >> > On Sun, Jul 19, 2015 at 4:57 PM, Jason Moore <[email protected]> > >> > wrote: > >> >> I just tried this out with jacobian() and subs() over the commits > since > >> >> 0.7.3 to master. It's showing me that the new caching is the killer > >> >> slowdown: > >> >> > >> >> https://github.com/sympy/sympy/commit/a63005e4 > >> >> > >> >> I've submitted a PR to Björn's repo: > >> >> https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files > >> >> > >> >> I'm assuming it uses fastcache because I have it installed, but maybe > >> >> not. > >> >> I'm not sure how to control for dependencies yet. > >> > > >> > I modified your benchmark to first do the substitution for symbols, > >> > only then do the Jacobian: > >> > > >> > > >> > > https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c > >> > > >> > Here are the results: > >> > > >> > certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py > >> > Setup > >> > Converting to SymEngine... > >> > SymPy Jacobian: > >> > Total time: 0.155253887177 s > >> > SymEngine Jacobian: > >> > Total time: 0.00277400016785 s > >> > > >> > Speedup: 55.97x > >> > > >> > > >> > I played with the cache size, it makes no difference. Your original > >> > benchmark ran about 1s on my machine, with SYMPY_CACHE_SIZE=5000 it > >> > took about 0.9s. So you can speedup SymPy itself just by using symbols > >> > instead of functions by a factor of 5.8x. Btw, I tested that the > >> > result that I get from symengine is exactly the same as in sympy > >> > > >> > ( > https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c#diff-e423f8edd8641c702513b2ea9c10eaecR32 > ). > >> > > >> > > >> > How did you generate the benchmark? Can you generate a bit larger > >> > matrix? Once the symengine benchmark runs in about 1s, then I can try > >> > to speed it up (because I would see the result in the timing more > >> > easily). > >> > >> I actually found a much larger benchmark, where just a simple > >> differentiation of a single entry (after substituting functionsf or > >> symbols) takes 60s for sympy. > >> > >> > >> > https://github.com/sympy/symengine/commit/31378da77e5463870b9a22fda7a17deb68068fa8 > >> > >> Here is where I am saving the matrix: > >> > >> > >> > https://github.com/sympy/symengine/commit/31378da77e5463870b9a22fda7a17deb68068fa8#diff-39829995ea91876984e13e3b828d740aR252 > >> > >> Does this make sense? Is that a good benchmark? I love the size, I > >> just want to make sure I am saving the right thing. Is that the full > >> bicycle? > >> > >> Ondrej > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups > >> "sympy" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to [email protected]. > >> To post to this group, send email to [email protected]. > >> Visit this group at http://groups.google.com/group/sympy. > >> To view this discussion on the web visit > >> > https://groups.google.com/d/msgid/sympy/CADDwiVDZ_JSgC-C_j5J0zmSOXS9TAQsGFv_XcmbBvx3ijPhPrg%40mail.gmail.com > . > >> For more options, visit https://groups.google.com/d/optout. > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "sympy" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To post to this group, send email to [email protected]. > > Visit this group at http://groups.google.com/group/sympy. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/sympy/CAP7f1AhavPNhXzdhUDUVH1ihgDgbc4FbdmRPBHrMW9HQ4ijZDA%40mail.gmail.com > . > > > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/sympy. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CADDwiVCNOioRCvXJ5SrdTn41m7R9RMpp_c9ASTY-oO5nVkR8LQ%40mail.gmail.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1Ai%3DtNzfxrnpad5_aQSt7qyyEtQ8esBo_-w3096Q57mNrA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
