Re: [sympy] Re: Being more conscious of performance
FYI, this database is building up: http://www.moorepants.info/misc/sympy-asv/ (I'm running this with the sympy cache set to 1). For the integration benchmarks it has every single functioning commit and maybe half the commits for the other benchmarks. The repository to submit benchmarks to is now at: https://github.com/sympy/sympy_benchmarks Please submit benchmarks for core parts of SymPy that you work on. Jason moorepants.info +01 530-601-9791 On Tue, Jul 21, 2015 at 12:16 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Mon, Jul 20, 2015 at 11:29 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: It's because I didn't have fastcache installed After installing it, by default I got: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.123499155045 s SymEngine Jacobian: Total time: 0.00305485725403 s Speedup: 40.43x (I am running some longer calculation at the same time, so the timing is a bit different, but it got faster.) But bigger cache doesn't seem to affect it, as you noted: certik@redhawk:~/repos/symengine/benchmarks(py)$ SYMPY_CACHE_SIZE=5 python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.121557950974 s SymEngine Jacobian: Total time: 0.00302696228027 s Speedup: 40.16x More interestingly, I can now differentiate the actual bicycle expression. So far I just took one element on the right hand side vector, and differentiate with respect to q5. I substitute for symbols first (in both SymPy and SymEngine). This is what I got: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane3.py Converting to SymPy... Converting to SymEngine... Done. Subs Eval initial expression -14960.1119361 -14960.111936127382563475771853345057230700491739739146860248076 + 0.00*I SymPy diff: Total time: 3.90587186813 s SymEngine diff: Total time: 0.0883350372314 s Speedup: 44.22x Subs Eval derivative -31789.936485 -31789.936484889832248148585748003998439133612431284784993412159 + 0.00*I Converting SymPy derivative to SymEngine... Subs Eval derivative -31789.936485 -31789.936484889832248148585748003998439133612431284784993279022 + 0.00*I The floating point numbers are calculated by substituting integers for the symbols and evaluating. The shorter number is using machine double precision, the longer number is using MPC (evaluates in complex plane, to make sure we don't get any negative values under square roots). They both agree as you can see. The initial expression number I checked against sympy (substituting the numbers using sympy and evaluating using sympy). For the derivatives, currently I just the sympy derivative to symengine to evaluate as a floating point, since sympy is just too slow (it's been running for 30 minutes -- it's stuck at the .n() method in sympy). The SymEngine evaluation is immediate It eventually finished, and the floating point number calculated using pure SymPy was exactly the same as with SymEngine. So I trust the derivatives and conversion back and forth by now for this particular expression. I don't know how long it took, probably a few hours (compared to less than 1s in symengine, so clearly the .n() algorithm is not the most efficient, it probably has to do with the fact that it automatically checks the accuracy --- showing that a better way to do that is just keep evaluating at increasing precision and see if it converges, or use something like Arb). Overall I am very happy about this progress, we have a robust foundation to build upon. I'll try to get this very benchmark working with Sage to get an idea how long it takes there. The branch with symengine+sympy code is here: https://github.com/sympy/symengine/compare/master...certik:py, it's not very polished, that I will do later. Ondrej for double precision and a fraction of a second for the MPC. But the conversion between symengine and sympy works well and I tested the initial expression in both, so I think this part works. And the fact that sympy derivative calculated using sympy and converted to symengine and symengine derivative agrees I think proves with pretty high confidence that things work. Now the next step is to calculate the whole jacobian and again test that we get the same expression. It's very important to spend time and check the numerical evaluation of the expression, since there could be bugs in symengine. I just fixed one here: https://github.com/sympy/symengine/pull/556 (it was giving me different numbers for the initial expression, but now they agree). This was a bug in subs, but there could be a bug in the derivative
Re: [sympy] Re: Being more conscious of performance
We will work on the documentation. But essentially I'm a running: SYMPY_CACHE_SIZE=1 asv run ALL --parallel -k -e You can use -s num to do a sparser mesh and then replace ALL with a git commit range to narrow down to commits in specific areas. Note that `asv find` is useful for bisecting to an offending commit. To view the results do: asv publish asv preview Bjorn and I discussed the ideal time. I'd say keep them under a second or two in general. There may be cases that you need to use longer benchmarks but I think you should profile the code and then write a benchmark that tests the slow portions (which will likely be short benchmarks). But it can run any length benchmark, but remember that they will ultimately be run on every single commit (~25000) so it can take a really long time. I'll start running asv run NEW once we have things going and benchmark designs stabilize. Jason moorepants.info +01 530-601-9791 On Wed, Jul 22, 2015 at 4:50 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Wed, Jul 22, 2015 at 4:16 PM, Jason Moore moorepa...@gmail.com wrote: FYI, this database is building up: http://www.moorepants.info/misc/sympy-asv/ (I'm running this with the sympy cache set to 1). For the integration benchmarks it has every single functioning commit and maybe half the commits for the other benchmarks. The repository to submit benchmarks to is now at: https://github.com/sympy/sympy_benchmarks Please submit benchmarks for core parts of SymPy that you work on. That is awesome. Essentially we can rerun it any time in the future as well if we want to figure out why is some code slow and if it was always that slow. Can you post instructions how to run it on my own machine? (I'll be happy to write it up into the README.) What is the ideal time for a benchmark? Can in benchmark a quick running code by running it several times? Does it work well if the benchmark takes 10s? Or is that too long. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVCv6zYH5crbtEG76O_D27e-e7uvOj-FQv%2BYF8z8xtbx1A%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AiqsyHOQo%3Duy5pMNiqgriOhRrT%3DS18JO-uFTr%2BJUXC8-A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Wed, Jul 22, 2015 at 6:26 PM, Jason Moore moorepa...@gmail.com wrote: We will work on the documentation. But essentially I'm a running: SYMPY_CACHE_SIZE=1 asv run ALL --parallel -k -e You can use -s num to do a sparser mesh and then replace ALL with a git commit range to narrow down to commits in specific areas. Note that `asv find` is useful for bisecting to an offending commit. To view the results do: asv publish asv preview Bjorn and I discussed the ideal time. I'd say keep them under a second or two in general. There may be cases that you need to use longer benchmarks but I think you should profile the code and then write a benchmark that tests the slow portions (which will likely be short benchmarks). But it can run any length benchmark, but remember that they will ultimately be run on every single commit (~25000) so it can take a really long time. Thanks for the info. Just to clarify, if I run 1 benchmark that takes 1s for the whole history, it would take less than 7 hours to finish: In [2]: 25000 * 1. / 3600 Out[2]: 6.945 Is that right? I'll start running asv run NEW once we have things going and benchmark designs stabilize. Jason moorepants.info +01 530-601-9791 On Wed, Jul 22, 2015 at 4:50 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Wed, Jul 22, 2015 at 4:16 PM, Jason Moore moorepa...@gmail.com wrote: FYI, this database is building up: http://www.moorepants.info/misc/sympy-asv/ (I'm running this with the sympy cache set to 1). For the integration benchmarks it has every single functioning commit and maybe half the commits for the other benchmarks. The repository to submit benchmarks to is now at: https://github.com/sympy/sympy_benchmarks Please submit benchmarks for core parts of SymPy that you work on. That is awesome. Essentially we can rerun it any time in the future as well if we want to figure out why is some code slow and if it was always that slow. Can you post instructions how to run it on my own machine? (I'll be happy to write it up into the README.) What is the ideal time for a benchmark? Can in benchmark a quick running code by running it several times? Does it work well if the benchmark takes 10s? Or is that too long. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVCv6zYH5crbtEG76O_D27e-e7uvOj-FQv%2BYF8z8xtbx1A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AiqsyHOQo%3Duy5pMNiqgriOhRrT%3DS18JO-uFTr%2BJUXC8-A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDjgpvUJAREPZVWw1BTgPxhe9sYrfSt0zVsew%3DrhSJr1A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Wed, Jul 22, 2015 at 4:16 PM, Jason Moore moorepa...@gmail.com wrote: FYI, this database is building up: http://www.moorepants.info/misc/sympy-asv/ (I'm running this with the sympy cache set to 1). For the integration benchmarks it has every single functioning commit and maybe half the commits for the other benchmarks. The repository to submit benchmarks to is now at: https://github.com/sympy/sympy_benchmarks Please submit benchmarks for core parts of SymPy that you work on. That is awesome. Essentially we can rerun it any time in the future as well if we want to figure out why is some code slow and if it was always that slow. Can you post instructions how to run it on my own machine? (I'll be happy to write it up into the README.) What is the ideal time for a benchmark? Can in benchmark a quick running code by running it several times? Does it work well if the benchmark takes 10s? Or is that too long. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVCv6zYH5crbtEG76O_D27e-e7uvOj-FQv%2BYF8z8xtbx1A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
Yes, but it is likely to fail on many of the commits (probably more than half depending on the functionality being tested and how backwards compatible the benchmark is written). There is also setup overhead time for each commit, so it will take longer than your calc. Jason moorepants.info +01 530-601-9791 On Wed, Jul 22, 2015 at 5:39 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Wed, Jul 22, 2015 at 6:26 PM, Jason Moore moorepa...@gmail.com wrote: We will work on the documentation. But essentially I'm a running: SYMPY_CACHE_SIZE=1 asv run ALL --parallel -k -e You can use -s num to do a sparser mesh and then replace ALL with a git commit range to narrow down to commits in specific areas. Note that `asv find` is useful for bisecting to an offending commit. To view the results do: asv publish asv preview Bjorn and I discussed the ideal time. I'd say keep them under a second or two in general. There may be cases that you need to use longer benchmarks but I think you should profile the code and then write a benchmark that tests the slow portions (which will likely be short benchmarks). But it can run any length benchmark, but remember that they will ultimately be run on every single commit (~25000) so it can take a really long time. Thanks for the info. Just to clarify, if I run 1 benchmark that takes 1s for the whole history, it would take less than 7 hours to finish: In [2]: 25000 * 1. / 3600 Out[2]: 6.945 Is that right? I'll start running asv run NEW once we have things going and benchmark designs stabilize. Jason moorepants.info +01 530-601-9791 On Wed, Jul 22, 2015 at 4:50 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Wed, Jul 22, 2015 at 4:16 PM, Jason Moore moorepa...@gmail.com wrote: FYI, this database is building up: http://www.moorepants.info/misc/sympy-asv/ (I'm running this with the sympy cache set to 1). For the integration benchmarks it has every single functioning commit and maybe half the commits for the other benchmarks. The repository to submit benchmarks to is now at: https://github.com/sympy/sympy_benchmarks Please submit benchmarks for core parts of SymPy that you work on. That is awesome. Essentially we can rerun it any time in the future as well if we want to figure out why is some code slow and if it was always that slow. Can you post instructions how to run it on my own machine? (I'll be happy to write it up into the README.) What is the ideal time for a benchmark? Can in benchmark a quick running code by running it several times? Does it work well if the benchmark takes 10s? Or is that too long. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVCv6zYH5crbtEG76O_D27e-e7uvOj-FQv%2BYF8z8xtbx1A%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AiqsyHOQo%3Duy5pMNiqgriOhRrT%3DS18JO-uFTr%2BJUXC8-A%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDjgpvUJAREPZVWw1BTgPxhe9sYrfSt0zVsew%3DrhSJr1A%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AgUj07xbpibduc2Hi6KHutkbTEPcR4MzgVuXd1CTAJVdQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Mon, Jul 20, 2015 at 11:29 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: It's because I didn't have fastcache installed After installing it, by default I got: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.123499155045 s SymEngine Jacobian: Total time: 0.00305485725403 s Speedup: 40.43x (I am running some longer calculation at the same time, so the timing is a bit different, but it got faster.) But bigger cache doesn't seem to affect it, as you noted: certik@redhawk:~/repos/symengine/benchmarks(py)$ SYMPY_CACHE_SIZE=5 python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.121557950974 s SymEngine Jacobian: Total time: 0.00302696228027 s Speedup: 40.16x More interestingly, I can now differentiate the actual bicycle expression. So far I just took one element on the right hand side vector, and differentiate with respect to q5. I substitute for symbols first (in both SymPy and SymEngine). This is what I got: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane3.py Converting to SymPy... Converting to SymEngine... Done. Subs Eval initial expression -14960.1119361 -14960.111936127382563475771853345057230700491739739146860248076 + 0.00*I SymPy diff: Total time: 3.90587186813 s SymEngine diff: Total time: 0.0883350372314 s Speedup: 44.22x Subs Eval derivative -31789.936485 -31789.936484889832248148585748003998439133612431284784993412159 + 0.00*I Converting SymPy derivative to SymEngine... Subs Eval derivative -31789.936485 -31789.936484889832248148585748003998439133612431284784993279022 + 0.00*I The floating point numbers are calculated by substituting integers for the symbols and evaluating. The shorter number is using machine double precision, the longer number is using MPC (evaluates in complex plane, to make sure we don't get any negative values under square roots). They both agree as you can see. The initial expression number I checked against sympy (substituting the numbers using sympy and evaluating using sympy). For the derivatives, currently I just the sympy derivative to symengine to evaluate as a floating point, since sympy is just too slow (it's been running for 30 minutes -- it's stuck at the .n() method in sympy). The SymEngine evaluation is immediate It eventually finished, and the floating point number calculated using pure SymPy was exactly the same as with SymEngine. So I trust the derivatives and conversion back and forth by now for this particular expression. I don't know how long it took, probably a few hours (compared to less than 1s in symengine, so clearly the .n() algorithm is not the most efficient, it probably has to do with the fact that it automatically checks the accuracy --- showing that a better way to do that is just keep evaluating at increasing precision and see if it converges, or use something like Arb). Overall I am very happy about this progress, we have a robust foundation to build upon. I'll try to get this very benchmark working with Sage to get an idea how long it takes there. The branch with symengine+sympy code is here: https://github.com/sympy/symengine/compare/master...certik:py, it's not very polished, that I will do later. Ondrej for double precision and a fraction of a second for the MPC. But the conversion between symengine and sympy works well and I tested the initial expression in both, so I think this part works. And the fact that sympy derivative calculated using sympy and converted to symengine and symengine derivative agrees I think proves with pretty high confidence that things work. Now the next step is to calculate the whole jacobian and again test that we get the same expression. It's very important to spend time and check the numerical evaluation of the expression, since there could be bugs in symengine. I just fixed one here: https://github.com/sympy/symengine/pull/556 (it was giving me different numbers for the initial expression, but now they agree). This was a bug in subs, but there could be a bug in the derivative as well. After we get the same expressions, we can start benchmarking. I can start profiling the derivative code in C++ and see if it is possible to speed it up. I can also implement cache in symengine, though I don't know if it would speed up anything here. Still, even if we only get 45x speedup, then instead of 4 minutes, the whole bicycle should take 5s. So I think that's a big deal. I will also try to benchmark this against other software like Sage to get an idea how we are doing. Ondrej On Mon, Jul 20, 2015 at 11:03 PM, Jason Moore moorepa...@gmail.com wrote: Ondrej, I'm not sure why you don't see performance increase
Re: [sympy] Re: Being more conscious of performance
Yes, it seems that the new cache commit is the slow down in these tests. Jason moorepants.info +01 530-601-9791 On Sun, Jul 19, 2015 at 11:54 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Mon, Jul 20, 2015 at 12:33 AM, Jason Moore moorepa...@gmail.com wrote: Here is the last run I made: http://www.moorepants.info/misc/sympy-asv/ from: asv run sympy-0.7.3..master -s 200 Is caching causing the massive (10x) slowdown? If so, I know you can turn it off. We should investigate this. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVA2QwER52pHrCTWvhQ%2B2fTSGDE9R0hF74aqmV%3DTM%2B3pxQ%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AhMRBKXuQS6m7X2m8qF_h7tHNLO92b2-1MTbv6AXHD9VA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
Here is the last run I made: http://www.moorepants.info/misc/sympy-asv/ from: asv run sympy-0.7.3..master -s 200 Jason moorepants.info +01 530-601-9791 On Sun, Jul 19, 2015 at 9:38 PM, Aaron Meurer asmeu...@gmail.com wrote: Cool. For this benchmark, you're likely seeing the evolution in the integration algorithms as well. The risch algorithm doesn't handle this integral (too many symbolic parameters), but meijerg does, and so does heurisch. On Sun, Jul 19, 2015 at 6:16 AM, Björn Dahlgren bjo...@gmail.com wrote: Hi all, On Tuesday, 14 July 2015 02:49:57 UTC+2, Aaron Meurer wrote: - Get a benchmark machine and run airspeed velocity on it. We need to catch performance regressions. The benchmark suite can be anything, although obviously well-made benchmarks are better. I found a few hours and threw this together (motivated by having noticed some things I have been doing with sympy has been getting slower): https://github.com/bjodah/sympy_benchmarks_bjodah then, on a machine with asv 0.1.1, I ran: $ asv run ALL -s 100 which essensitally takes 100 equally spaced commits from the whle history and run the benchmarks one those. You can view the results here: http://hera.physchem.kth.se/~sympy_asv some commits failed, maybe I should be targetting the merge commits but I haven't figured out how to do that yet. Furthermore: the comparison isn't quite fair since I belive the later versions return more correct results (Piecewise solutions), I'm not sure how to deal with this fairly, maybe some kind of assertion should be made (like a tests) at the end of the individual benchmark, a passing assertion means it should be included, a failing one that it should be excluded. But then this assertion shouldn't be included in the timing. There could be something in asv for this but I haven't figured it out yet. I am stil a bit new to using asv so I haven't quite figured out best practives here. It would be nice to have a dedicated machine (or maybe several), maybe a Raspberry Pi 2 which would make it easy for people to get the exact same hardware for comparisons? That's a great idea. Aaron Meurer Anyways, thought I would share. Feel free to fork/send pull requests, I'll happily run them on that machine if there is an interest. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/5a7e4b53-6daa-4fc2-a8ac-a3288a0270c7%40googlegroups.com https://groups.google.com/d/msgid/sympy/5a7e4b53-6daa-4fc2-a8ac-a3288a0270c7%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2Bi13eMuPvm3bK_ME9kxpBjW9MpwWvJ5XyNJiWPYqAYMA%40mail.gmail.com https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2Bi13eMuPvm3bK_ME9kxpBjW9MpwWvJ5XyNJiWPYqAYMA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AgvPvQeMNshMg3F6vjH_Q3CBD_50dksMBsW1qGPy8yZfw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Mon, Jul 20, 2015 at 12:33 AM, Jason Moore moorepa...@gmail.com wrote: Here is the last run I made: http://www.moorepants.info/misc/sympy-asv/ from: asv run sympy-0.7.3..master -s 200 Is caching causing the massive (10x) slowdown? If so, I know you can turn it off. We should investigate this. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVA2QwER52pHrCTWvhQ%2B2fTSGDE9R0hF74aqmV%3DTM%2B3pxQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Mon, Jul 20, 2015 at 1:02 AM, Jason Moore moorepa...@gmail.com wrote: Yes, it seems that the new cache commit is the slow down in these tests. If this is the case, then I know that Peter Brady who wrote it will be interested in this. We should get to the bottom of the issue. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVC5x6UcC60G9RZdB%3DQah8gC1EQ5B2PZug7%2BeM9ixmBYJg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Monday, 20 July 2015 19:09:54 UTC+2, Aaron Meurer wrote: So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). asv has support for tracking memory usage: http://asv.readthedocs.org/en/latest/writing_benchmarks.html#memory It's just a matter of adding memory benchmarks.. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
Regarding the Raspberry Pi 2, as far as I can tell it is a good option, but I opened https://github.com/spacetelescope/asv/issues/292 to see if anyone else has any suggestions. Aaron Meurer On Mon, Jul 20, 2015 at 1:20 PM, Björn Dahlgren bjo...@gmail.com wrote: On Monday, 20 July 2015 19:09:54 UTC+2, Aaron Meurer wrote: So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). asv has support for tracking memory usage: http://asv.readthedocs.org/en/latest/writing_benchmarks.html#memory It's just a matter of adding memory benchmarks.. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2B0Oha6UMz4O6KK3kcVdLXEOaUdXWWODF6B0PfAZ%2B08JQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
Elaborating a bit, the 2 main additional costs of a bounded cache compared to an unbounded one are: 1. The extra cost of managing the LRU machinery (fastcache alleviates this by doing all the necessary management at the C level) 2. The cost of repeated cache misses because the cache size is not appropriate for a particular problem. Problem 2 is much harder to quantify and solve efficiently and could easily be the cause of the significant slowdown on some tests. We could just make the cache bigger... On Monday, July 20, 2015 at 10:30:11 AM UTC-6, Peter Brady wrote: We visited the jacobian issue a while ago and I think the takeaway was that a larger cache size (about 2-3000) sped things up considerably. Not sure if this is the same issue though. On Monday, July 20, 2015 at 10:25:24 AM UTC-6, Ondřej Čertík wrote: On Mon, Jul 20, 2015 at 1:02 AM, Jason Moore moore...@gmail.com wrote: Yes, it seems that the new cache commit is the slow down in these tests. If this is the case, then I know that Peter Brady who wrote it will be interested in this. We should get to the bottom of the issue. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/a2484c2c-a031-46f9-8dc9-827efbc48927%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
We visited the jacobian issue a while ago and I think the takeaway was that a larger cache size (about 2-3000) sped things up considerably. Not sure if this is the same issue though. On Monday, July 20, 2015 at 10:25:24 AM UTC-6, Ondřej Čertík wrote: On Mon, Jul 20, 2015 at 1:02 AM, Jason Moore moore...@gmail.com javascript: wrote: Yes, it seems that the new cache commit is the slow down in these tests. If this is the case, then I know that Peter Brady who wrote it will be interested in this. We should get to the bottom of the issue. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/d7948933-9732-4b9e-8300-c9413f09be5e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
Awesome. This is exactly the sort of thing I've wanted to see for a long time. So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). Aaron Meurer On Mon, Jul 20, 2015 at 12:05 PM, Jason Moore moorepa...@gmail.com wrote: This now has every commit from 0.7.3 on: http://www.moorepants.info/misc/sympy-asv/#diff.TimeJacobian.time_subs first major slow down: new caching added slight speedup: fastcache optional dep added another speedup: cache increased from 500 to 1000 slight slow down: c removed from core This goes for subs and jacobian. Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 9:38 AM, Peter Brady petertbr...@gmail.com wrote: Elaborating a bit, the 2 main additional costs of a bounded cache compared to an unbounded one are: 1. The extra cost of managing the LRU machinery (fastcache alleviates this by doing all the necessary management at the C level) 2. The cost of repeated cache misses because the cache size is not appropriate for a particular problem. Problem 2 is much harder to quantify and solve efficiently and could easily be the cause of the significant slowdown on some tests. We could just make the cache bigger... On Monday, July 20, 2015 at 10:30:11 AM UTC-6, Peter Brady wrote: We visited the jacobian issue a while ago and I think the takeaway was that a larger cache size (about 2-3000) sped things up considerably. Not sure if this is the same issue though. On Monday, July 20, 2015 at 10:25:24 AM UTC-6, Ondřej Čertík wrote: On Mon, Jul 20, 2015 at 1:02 AM, Jason Moore moore...@gmail.com wrote: Yes, it seems that the new cache commit is the slow down in these tests. If this is the case, then I know that Peter Brady who wrote it will be interested in this. We should get to the bottom of the issue. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/a2484c2c-a031-46f9-8dc9-827efbc48927%40googlegroups.com https://groups.google.com/d/msgid/sympy/a2484c2c-a031-46f9-8dc9-827efbc48927%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AhWxQBUQY9fkJVmxOcAPuz%2BJefBOdmTDMNO5f1cSoHfOA%40mail.gmail.com https://groups.google.com/d/msgid/sympy/CAP7f1AhWxQBUQY9fkJVmxOcAPuz%2BJefBOdmTDMNO5f1cSoHfOA%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6KaQZKJipbaqj-4dTtsgYHivXZDnYBBsaK8gtRG%3DN34cQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
This now has every commit from 0.7.3 on: http://www.moorepants.info/misc/sympy-asv/#diff.TimeJacobian.time_subs first major slow down: new caching added slight speedup: fastcache optional dep added another speedup: cache increased from 500 to 1000 slight slow down: c removed from core This goes for subs and jacobian. Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 9:38 AM, Peter Brady petertbr...@gmail.com wrote: Elaborating a bit, the 2 main additional costs of a bounded cache compared to an unbounded one are: 1. The extra cost of managing the LRU machinery (fastcache alleviates this by doing all the necessary management at the C level) 2. The cost of repeated cache misses because the cache size is not appropriate for a particular problem. Problem 2 is much harder to quantify and solve efficiently and could easily be the cause of the significant slowdown on some tests. We could just make the cache bigger... On Monday, July 20, 2015 at 10:30:11 AM UTC-6, Peter Brady wrote: We visited the jacobian issue a while ago and I think the takeaway was that a larger cache size (about 2-3000) sped things up considerably. Not sure if this is the same issue though. On Monday, July 20, 2015 at 10:25:24 AM UTC-6, Ondřej Čertík wrote: On Mon, Jul 20, 2015 at 1:02 AM, Jason Moore moore...@gmail.com wrote: Yes, it seems that the new cache commit is the slow down in these tests. If this is the case, then I know that Peter Brady who wrote it will be interested in this. We should get to the bottom of the issue. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/a2484c2c-a031-46f9-8dc9-827efbc48927%40googlegroups.com https://groups.google.com/d/msgid/sympy/a2484c2c-a031-46f9-8dc9-827efbc48927%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AhWxQBUQY9fkJVmxOcAPuz%2BJefBOdmTDMNO5f1cSoHfOA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
How is the memory usage? We should try to find a good balance. Can you create plots of memory usage and performance vs. cache size (in master)? Aaron Meurer On Mon, Jul 20, 2015 at 2:14 PM, Jason Moore moorepa...@gmail.com wrote: FYI, if I increased the cache size I can push the timings, post new cache, down to the normal speeds. e.g. SYMPY_CACHE_SIZE=5000 asv run 051850f2..880f5fa6 Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 12:07 PM, Aaron Meurer asmeu...@gmail.com wrote: Regarding the Raspberry Pi 2, as far as I can tell it is a good option, but I opened https://github.com/spacetelescope/asv/issues/292 to see if anyone else has any suggestions. Aaron Meurer On Mon, Jul 20, 2015 at 1:20 PM, Björn Dahlgren bjo...@gmail.com wrote: On Monday, 20 July 2015 19:09:54 UTC+2, Aaron Meurer wrote: So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). asv has support for tracking memory usage: http://asv.readthedocs.org/en/latest/writing_benchmarks.html#memory It's just a matter of adding memory benchmarks.. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2B0Oha6UMz4O6KK3kcVdLXEOaUdXWWODF6B0PfAZ%2B08JQ%40mail.gmail.com https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2B0Oha6UMz4O6KK3kcVdLXEOaUdXWWODF6B0PfAZ%2B08JQ%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AgfYAA7y87vAczE%3D8e%2BbKGX7E6p9-MfBsWT%2B9f627D3Ug%40mail.gmail.com https://groups.google.com/d/msgid/sympy/CAP7f1AgfYAA7y87vAczE%3D8e%2BbKGX7E6p9-MfBsWT%2B9f627D3Ug%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BFmBOr4O85ZD8%2BYrE-Hn9NRK04P8%2BuusEG1H4S%3DN%3DF6g%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
FYI, if I increased the cache size I can push the timings, post new cache, down to the normal speeds. e.g. SYMPY_CACHE_SIZE=5000 asv run 051850f2..880f5fa6 Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 12:07 PM, Aaron Meurer asmeu...@gmail.com wrote: Regarding the Raspberry Pi 2, as far as I can tell it is a good option, but I opened https://github.com/spacetelescope/asv/issues/292 to see if anyone else has any suggestions. Aaron Meurer On Mon, Jul 20, 2015 at 1:20 PM, Björn Dahlgren bjo...@gmail.com wrote: On Monday, 20 July 2015 19:09:54 UTC+2, Aaron Meurer wrote: So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). asv has support for tracking memory usage: http://asv.readthedocs.org/en/latest/writing_benchmarks.html#memory It's just a matter of adding memory benchmarks.. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2B0Oha6UMz4O6KK3kcVdLXEOaUdXWWODF6B0PfAZ%2B08JQ%40mail.gmail.com https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2B0Oha6UMz4O6KK3kcVdLXEOaUdXWWODF6B0PfAZ%2B08JQ%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AgfYAA7y87vAczE%3D8e%2BbKGX7E6p9-MfBsWT%2B9f627D3Ug%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Mon, Jul 20, 2015 at 11:09 AM, Aaron Meurer asmeu...@gmail.com wrote: Awesome. This is exactly the sort of thing I've wanted to see for a long time. So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). Indeed, thanks everybody for pushing this. I need to setup something similar for SymEngine --- we can just run the tests using Python wrappers, so ASV can be used for it as well. I think this is useful even without a dedicated machine --- you just run it on some machine that you have for few days, first every couple hundreds of commits to get a quick graph back and then you zoom in. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDukTrcuPZz38gWqDvTKs6Ot5D88O-vGhW3XnS6jkG16w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
I can try to come up with something...but I need to get back to the day job at the moment :( Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 12:17 PM, Aaron Meurer asmeu...@gmail.com wrote: How is the memory usage? We should try to find a good balance. Can you create plots of memory usage and performance vs. cache size (in master)? Aaron Meurer On Mon, Jul 20, 2015 at 2:14 PM, Jason Moore moorepa...@gmail.com wrote: FYI, if I increased the cache size I can push the timings, post new cache, down to the normal speeds. e.g. SYMPY_CACHE_SIZE=5000 asv run 051850f2..880f5fa6 Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 12:07 PM, Aaron Meurer asmeu...@gmail.com wrote: Regarding the Raspberry Pi 2, as far as I can tell it is a good option, but I opened https://github.com/spacetelescope/asv/issues/292 to see if anyone else has any suggestions. Aaron Meurer On Mon, Jul 20, 2015 at 1:20 PM, Björn Dahlgren bjo...@gmail.com wrote: On Monday, 20 July 2015 19:09:54 UTC+2, Aaron Meurer wrote: So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). asv has support for tracking memory usage: http://asv.readthedocs.org/en/latest/writing_benchmarks.html#memory It's just a matter of adding memory benchmarks.. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com https://groups.google.com/d/msgid/sympy/09b68809-bd21-4806-8f96-2b0a7cff722b%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2B0Oha6UMz4O6KK3kcVdLXEOaUdXWWODF6B0PfAZ%2B08JQ%40mail.gmail.com https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2B0Oha6UMz4O6KK3kcVdLXEOaUdXWWODF6B0PfAZ%2B08JQ%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AgfYAA7y87vAczE%3D8e%2BbKGX7E6p9-MfBsWT%2B9f627D3Ug%40mail.gmail.com https://groups.google.com/d/msgid/sympy/CAP7f1AgfYAA7y87vAczE%3D8e%2BbKGX7E6p9-MfBsWT%2B9f627D3Ug%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BFmBOr4O85ZD8%2BYrE-Hn9NRK04P8%2BuusEG1H4S%3DN%3DF6g%40mail.gmail.com https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2BFmBOr4O85ZD8%2BYrE-Hn9NRK04P8%2BuusEG1H4S%3DN%3DF6g%40mail.gmail.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AhH5rqwemnJCKL68SAsmRbvZ1z32bbj4WQnrkfez5hAFw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Monday, 20 July 2015 09:02:23 UTC+2, Jason Moore wrote: Yes, it seems that the new cache commit is the slow down in these tests. Running with fastcache installed seems to make a minor difference (~10-30%) http://hera.physchem.kth.se/~sympy_asv/ I haven't yet tried running tests with SYMPY_USE_CACHE=no (it is my understanding that the environment variable needs to be set before importing sympy, which means I need to hardcode it to be set in the *.py files in benchmarks/) -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/4a9190ca-447b-43c1-af5d-7202845f17e3%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Sun, Jul 19, 2015 at 4:57 PM, Jason Moore moorepa...@gmail.com wrote: I just tried this out with jacobian() and subs() over the commits since 0.7.3 to master. It's showing me that the new caching is the killer slowdown: https://github.com/sympy/sympy/commit/a63005e4 I've submitted a PR to Björn's repo: https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files I'm assuming it uses fastcache because I have it installed, but maybe not. I'm not sure how to control for dependencies yet. I modified your benchmark to first do the substitution for symbols, only then do the Jacobian: https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c Here are the results: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.155253887177 s SymEngine Jacobian: Total time: 0.00277400016785 s Speedup: 55.97x I played with the cache size, it makes no difference. Your original benchmark ran about 1s on my machine, with SYMPY_CACHE_SIZE=5000 it took about 0.9s. So you can speedup SymPy itself just by using symbols instead of functions by a factor of 5.8x. Btw, I tested that the result that I get from symengine is exactly the same as in sympy (https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c#diff-e423f8edd8641c702513b2ea9c10eaecR32). How did you generate the benchmark? Can you generate a bit larger matrix? Once the symengine benchmark runs in about 1s, then I can try to speed it up (because I would see the result in the timing more easily). Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDf9OLX1H3dS2Ugnr1bH%2BX-Q_dOziwWGB7asUWDCvUJ4Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Mon, Jul 20, 2015 at 7:48 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Sun, Jul 19, 2015 at 4:57 PM, Jason Moore moorepa...@gmail.com wrote: I just tried this out with jacobian() and subs() over the commits since 0.7.3 to master. It's showing me that the new caching is the killer slowdown: https://github.com/sympy/sympy/commit/a63005e4 I've submitted a PR to Björn's repo: https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files I'm assuming it uses fastcache because I have it installed, but maybe not. I'm not sure how to control for dependencies yet. I modified your benchmark to first do the substitution for symbols, only then do the Jacobian: https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c Here are the results: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.155253887177 s SymEngine Jacobian: Total time: 0.00277400016785 s Speedup: 55.97x I played with the cache size, it makes no difference. Your original benchmark ran about 1s on my machine, with SYMPY_CACHE_SIZE=5000 it took about 0.9s. So you can speedup SymPy itself just by using symbols instead of functions by a factor of 5.8x. Btw, I tested that the result that I get from symengine is exactly the same as in sympy (https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c#diff-e423f8edd8641c702513b2ea9c10eaecR32). How did you generate the benchmark? Can you generate a bit larger matrix? Once the symengine benchmark runs in about 1s, then I can try to speed it up (because I would see the result in the timing more easily). I actually found a much larger benchmark, where just a simple differentiation of a single entry (after substituting functionsf or symbols) takes 60s for sympy. https://github.com/sympy/symengine/commit/31378da77e5463870b9a22fda7a17deb68068fa8 Here is where I am saving the matrix: https://github.com/sympy/symengine/commit/31378da77e5463870b9a22fda7a17deb68068fa8#diff-39829995ea91876984e13e3b828d740aR252 Does this make sense? Is that a good benchmark? I love the size, I just want to make sure I am saving the right thing. Is that the full bicycle? Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDZ_JSgC-C_j5J0zmSOXS9TAQsGFv_XcmbBvx3ijPhPrg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
I've got a machine that I don't use, so I'm going periodically run the benchmarks from Bjorn's repo and automatically publish them to: moorepants.info/misc/sympy-asv I'll make an initial pass and the results will likely be up by tomorrow sometime. Once I get the base database up and running I will eventually have it pull down the benchmark repo from master so it will pick up new benchmarks and I'll have it skip existing successfully benchmarks/commits so that we only have to run the latest commits on a daily basis or so. I'll see how this goes for a while and we can think about having a different machine run these and pushing to gh-pages or something. Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 12:51 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Mon, Jul 20, 2015 at 11:09 AM, Aaron Meurer asmeu...@gmail.com wrote: Awesome. This is exactly the sort of thing I've wanted to see for a long time. So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). Indeed, thanks everybody for pushing this. I need to setup something similar for SymEngine --- we can just run the tests using Python wrappers, so ASV can be used for it as well. I think this is useful even without a dedicated machine --- you just run it on some machine that you have for few days, first every couple hundreds of commits to get a quick graph back and then you zoom in. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDukTrcuPZz38gWqDvTKs6Ot5D88O-vGhW3XnS6jkG16w%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1AjaaiR4UrCpYNEbGgdaGg4s6kv%2BOgpTL%3DmoitxHpzsNxw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
If we increase the default cache size and it speeds things up it might be worthwhile to revisit the travis test splits again. On Monday, July 20, 2015 at 4:36:45 PM UTC-6, Jason Moore wrote: I've got a machine that I don't use, so I'm going periodically run the benchmarks from Bjorn's repo and automatically publish them to: moorepants.info/misc/sympy-asv I'll make an initial pass and the results will likely be up by tomorrow sometime. Once I get the base database up and running I will eventually have it pull down the benchmark repo from master so it will pick up new benchmarks and I'll have it skip existing successfully benchmarks/commits so that we only have to run the latest commits on a daily basis or so. I'll see how this goes for a while and we can think about having a different machine run these and pushing to gh-pages or something. Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 12:51 PM, Ondřej Čertík ondrej...@gmail.com javascript: wrote: On Mon, Jul 20, 2015 at 11:09 AM, Aaron Meurer asme...@gmail.com javascript: wrote: Awesome. This is exactly the sort of thing I've wanted to see for a long time. So apparently the new cache is way too slow. Can the size be increased to a point that makes the performance comparable to the old cache? One obviously has to balance the cache size against memory usage (which won't show up in the performance tests). Indeed, thanks everybody for pushing this. I need to setup something similar for SymEngine --- we can just run the tests using Python wrappers, so ASV can be used for it as well. I think this is useful even without a dedicated machine --- you just run it on some machine that you have for few days, first every couple hundreds of commits to get a quick graph back and then you zoom in. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+un...@googlegroups.com javascript:. To post to this group, send email to sy...@googlegroups.com javascript: . Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVDukTrcuPZz38gWqDvTKs6Ot5D88O-vGhW3XnS6jkG16w%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/3d7de67c-3e02-417e-8e07-f345ae38ef96%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
Ondrej, I'm not sure why you don't see performance increase with increased cache. The following shows that the benchmarks do run faster with a large cache. Interestingly the memory doesn't seem to change (but I'm not sure I understand how they measure mem usage). Notice that the jacobian wrt to symbols is not affected by the cache size but the jacobians wrt to the functions is, as is substitution. It seems that .diff() calls subs a lot (see https://github.com/sympy/sympy/issues/9701), so maybe the slow down is all due to the caching used for subs. moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ asv --config asv.conf.conda.json run · Fetching recent changes. · Creating environments · Discovering benchmarks ·· Uninstalling from py2.7-fastcache-mpmath. ·· Building for py2.7-fastcache-mpmath ·· Installing into py2.7-fastcache-mpmath.. · Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks) [ 0.00%] · For sympy commit hash 488f3c20: [ 0.00%] ·· Building for py2.7-fastcache-mpmath... [ 0.00%] ·· Benchmarking py2.7-fastcache-mpmath [ 11.11%] ··· Running integrate.TimeIntegration01.time_doit 365.73ms [ 22.22%] ··· Running integrate.TimeIntegration01.time_doit_meijerg 107.41ms [ 33.33%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions 37M [ 44.44%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols 37M [ 55.56%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_subs 37M [ 66.67%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions 1.75s [ 77.78%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols 35.49ms [ 88.89%] ··· Running large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions 1.79s [100.00%] ··· Running large_exprs.TimeLargeExpressionOperations.time_subs 1.66s moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ SYMPY_CACHE_SIZE=1 asv --config asv.conf.conda.json run · Fetching recent changes. · Creating environments · Discovering benchmarks ·· Uninstalling from py2.7-fastcache-mpmath. ·· Building for py2.7-fastcache-mpmath ·· Installing into py2.7-fastcache-mpmath.. · Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks) [ 0.00%] · For sympy commit hash 488f3c20: [ 0.00%] ·· Building for py2.7-fastcache-mpmath... [ 0.00%] ·· Benchmarking py2.7-fastcache-mpmath [ 11.11%] ··· Running integrate.TimeIntegration01.time_doit 361.81ms [ 22.22%] ··· Running integrate.TimeIntegration01.time_doit_meijerg 104.40ms [ 33.33%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_functions 37M [ 44.44%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_jacobian_wrt_symbols 37M [ 55.56%] ··· Running large_exprs.TimeLargeExpressionOperations.peakmem_subs 37M [ 66.67%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_functions 56.62ms [ 77.78%] ··· Running large_exprs.TimeLargeExpressionOperations.time_jacobian_wrt_symbols 33.04ms [ 88.89%] ··· Running large_exprs.TimeLargeExpressionOperations.time_manual_jacobian_wrt_functions 55.25ms [100.00%] ··· Running large_exprs.TimeLargeExpressionOperations.time_subs 13.89ms Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 6:57 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Mon, Jul 20, 2015 at 7:48 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Sun, Jul 19, 2015 at 4:57 PM, Jason Moore moorepa...@gmail.com wrote: I just tried this out with jacobian() and subs() over the commits since 0.7.3 to master. It's showing me that the new caching is the killer slowdown: https://github.com/sympy/sympy/commit/a63005e4 I've submitted a PR to Björn's repo: https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files I'm assuming it uses fastcache because I have it installed, but maybe not. I'm not sure how to control for dependencies yet. I modified your benchmark to first do the substitution for symbols, only then do the Jacobian: https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c Here are the results: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.155253887177 s SymEngine Jacobian: Total time: 0.00277400016785 s Speedup: 55.97x I played with the cache size, it makes no difference. Your original benchmark ran about 1s on my machine, with SYMPY_CACHE_SIZE=5000 it took about 0.9s. So you can speedup SymPy itself just by using symbols instead of functions by a factor of 5.8x. Btw, I tested that the result that I get from symengine is exactly the same as in sympy ( https://github.com/sympy/symengine/commit/2ababdb89e6c6db04d4a5df3e30185b824de9c0c#diff-e423f8edd8641c702513b2ea9c10eaecR32 ). How did you generate the benchmark? Can you generate a bit larger matrix? Once the symengine
Re: [sympy] Re: Being more conscious of performance
Nice work! Jason moorepants.info +01 530-601-9791 On Mon, Jul 20, 2015 at 10:29 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: It's because I didn't have fastcache installed After installing it, by default I got: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.123499155045 s SymEngine Jacobian: Total time: 0.00305485725403 s Speedup: 40.43x (I am running some longer calculation at the same time, so the timing is a bit different, but it got faster.) But bigger cache doesn't seem to affect it, as you noted: certik@redhawk:~/repos/symengine/benchmarks(py)$ SYMPY_CACHE_SIZE=5 python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.121557950974 s SymEngine Jacobian: Total time: 0.00302696228027 s Speedup: 40.16x More interestingly, I can now differentiate the actual bicycle expression. So far I just took one element on the right hand side vector, and differentiate with respect to q5. I substitute for symbols first (in both SymPy and SymEngine). This is what I got: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane3.py Converting to SymPy... Converting to SymEngine... Done. Subs Eval initial expression -14960.1119361 -14960.111936127382563475771853345057230700491739739146860248076 + 0.00*I SymPy diff: Total time: 3.90587186813 s SymEngine diff: Total time: 0.0883350372314 s Speedup: 44.22x Subs Eval derivative -31789.936485 -31789.936484889832248148585748003998439133612431284784993412159 + 0.00*I Converting SymPy derivative to SymEngine... Subs Eval derivative -31789.936485 -31789.936484889832248148585748003998439133612431284784993279022 + 0.00*I The floating point numbers are calculated by substituting integers for the symbols and evaluating. The shorter number is using machine double precision, the longer number is using MPC (evaluates in complex plane, to make sure we don't get any negative values under square roots). They both agree as you can see. The initial expression number I checked against sympy (substituting the numbers using sympy and evaluating using sympy). For the derivatives, currently I just the sympy derivative to symengine to evaluate as a floating point, since sympy is just too slow (it's been running for 30 minutes -- it's stuck at the .n() method in sympy). The SymEngine evaluation is immediate for double precision and a fraction of a second for the MPC. But the conversion between symengine and sympy works well and I tested the initial expression in both, so I think this part works. And the fact that sympy derivative calculated using sympy and converted to symengine and symengine derivative agrees I think proves with pretty high confidence that things work. Now the next step is to calculate the whole jacobian and again test that we get the same expression. It's very important to spend time and check the numerical evaluation of the expression, since there could be bugs in symengine. I just fixed one here: https://github.com/sympy/symengine/pull/556 (it was giving me different numbers for the initial expression, but now they agree). This was a bug in subs, but there could be a bug in the derivative as well. After we get the same expressions, we can start benchmarking. I can start profiling the derivative code in C++ and see if it is possible to speed it up. I can also implement cache in symengine, though I don't know if it would speed up anything here. Still, even if we only get 45x speedup, then instead of 4 minutes, the whole bicycle should take 5s. So I think that's a big deal. I will also try to benchmark this against other software like Sage to get an idea how we are doing. Ondrej On Mon, Jul 20, 2015 at 11:03 PM, Jason Moore moorepa...@gmail.com wrote: Ondrej, I'm not sure why you don't see performance increase with increased cache. The following shows that the benchmarks do run faster with a large cache. Interestingly the memory doesn't seem to change (but I'm not sure I understand how they measure mem usage). Notice that the jacobian wrt to symbols is not affected by the cache size but the jacobians wrt to the functions is, as is substitution. It seems that .diff() calls subs a lot (see https://github.com/sympy/sympy/issues/9701), so maybe the slow down is all due to the caching used for subs. moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ asv --config asv.conf.conda.json run · Fetching recent changes. · Creating environments · Discovering benchmarks ·· Uninstalling from py2.7-fastcache-mpmath. ·· Building for py2.7-fastcache-mpmath ·· Installing into py2.7-fastcache-mpmath.. · Running 9 total benchmarks (1 commits * 1
Re: [sympy] Re: Being more conscious of performance
It's because I didn't have fastcache installed After installing it, by default I got: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.123499155045 s SymEngine Jacobian: Total time: 0.00305485725403 s Speedup: 40.43x (I am running some longer calculation at the same time, so the timing is a bit different, but it got faster.) But bigger cache doesn't seem to affect it, as you noted: certik@redhawk:~/repos/symengine/benchmarks(py)$ SYMPY_CACHE_SIZE=5 python kane2.py Setup Converting to SymEngine... SymPy Jacobian: Total time: 0.121557950974 s SymEngine Jacobian: Total time: 0.00302696228027 s Speedup: 40.16x More interestingly, I can now differentiate the actual bicycle expression. So far I just took one element on the right hand side vector, and differentiate with respect to q5. I substitute for symbols first (in both SymPy and SymEngine). This is what I got: certik@redhawk:~/repos/symengine/benchmarks(py)$ python kane3.py Converting to SymPy... Converting to SymEngine... Done. Subs Eval initial expression -14960.1119361 -14960.111936127382563475771853345057230700491739739146860248076 + 0.00*I SymPy diff: Total time: 3.90587186813 s SymEngine diff: Total time: 0.0883350372314 s Speedup: 44.22x Subs Eval derivative -31789.936485 -31789.936484889832248148585748003998439133612431284784993412159 + 0.00*I Converting SymPy derivative to SymEngine... Subs Eval derivative -31789.936485 -31789.936484889832248148585748003998439133612431284784993279022 + 0.00*I The floating point numbers are calculated by substituting integers for the symbols and evaluating. The shorter number is using machine double precision, the longer number is using MPC (evaluates in complex plane, to make sure we don't get any negative values under square roots). They both agree as you can see. The initial expression number I checked against sympy (substituting the numbers using sympy and evaluating using sympy). For the derivatives, currently I just the sympy derivative to symengine to evaluate as a floating point, since sympy is just too slow (it's been running for 30 minutes -- it's stuck at the .n() method in sympy). The SymEngine evaluation is immediate for double precision and a fraction of a second for the MPC. But the conversion between symengine and sympy works well and I tested the initial expression in both, so I think this part works. And the fact that sympy derivative calculated using sympy and converted to symengine and symengine derivative agrees I think proves with pretty high confidence that things work. Now the next step is to calculate the whole jacobian and again test that we get the same expression. It's very important to spend time and check the numerical evaluation of the expression, since there could be bugs in symengine. I just fixed one here: https://github.com/sympy/symengine/pull/556 (it was giving me different numbers for the initial expression, but now they agree). This was a bug in subs, but there could be a bug in the derivative as well. After we get the same expressions, we can start benchmarking. I can start profiling the derivative code in C++ and see if it is possible to speed it up. I can also implement cache in symengine, though I don't know if it would speed up anything here. Still, even if we only get 45x speedup, then instead of 4 minutes, the whole bicycle should take 5s. So I think that's a big deal. I will also try to benchmark this against other software like Sage to get an idea how we are doing. Ondrej On Mon, Jul 20, 2015 at 11:03 PM, Jason Moore moorepa...@gmail.com wrote: Ondrej, I'm not sure why you don't see performance increase with increased cache. The following shows that the benchmarks do run faster with a large cache. Interestingly the memory doesn't seem to change (but I'm not sure I understand how they measure mem usage). Notice that the jacobian wrt to symbols is not affected by the cache size but the jacobians wrt to the functions is, as is substitution. It seems that .diff() calls subs a lot (see https://github.com/sympy/sympy/issues/9701), so maybe the slow down is all due to the caching used for subs. moorepants@moorepants-2170p:sympy_benchmarks_bjodah(larger-expr)$ asv --config asv.conf.conda.json run · Fetching recent changes. · Creating environments · Discovering benchmarks ·· Uninstalling from py2.7-fastcache-mpmath. ·· Building for py2.7-fastcache-mpmath ·· Installing into py2.7-fastcache-mpmath.. · Running 9 total benchmarks (1 commits * 1 environments * 9 benchmarks) [ 0.00%] · For sympy commit hash 488f3c20: [ 0.00%] ·· Building for py2.7-fastcache-mpmath... [ 0.00%] ·· Benchmarking py2.7-fastcache-mpmath [ 11.11%] ··· Running integrate.TimeIntegration01.time_doit
Re: [sympy] Re: Being more conscious of performance
I'm rerunning with the dependencies installed (I think). I'll post results once it is done. Jason moorepants.info +01 530-601-9791 On Sun, Jul 19, 2015 at 4:16 PM, Ondřej Čertík ondrej.cer...@gmail.com wrote: Can you post your timings results? I am doing your benchmark with SymEngine and Sage for comparison. On Sun, Jul 19, 2015 at 4:57 PM, Jason Moore moorepa...@gmail.com wrote: I just tried this out with jacobian() and subs() over the commits since 0.7.3 to master. It's showing me that the new caching is the killer slowdown: https://github.com/sympy/sympy/commit/a63005e4 I've submitted a PR to Björn's repo: https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files I'm assuming it uses fastcache because I have it installed, but maybe not. I'm not sure how to control for dependencies yet. Jason moorepants.info +01 530-601-9791 On Sun, Jul 19, 2015 at 10:39 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Sun, Jul 19, 2015 at 5:16 AM, Björn Dahlgren bjo...@gmail.com wrote: Hi all, On Tuesday, 14 July 2015 02:49:57 UTC+2, Aaron Meurer wrote: - Get a benchmark machine and run airspeed velocity on it. We need to catch performance regressions. The benchmark suite can be anything, although obviously well-made benchmarks are better. I found a few hours and threw this together (motivated by having noticed some things I have been doing with sympy has been getting slower): https://github.com/bjodah/sympy_benchmarks_bjodah then, on a machine with asv 0.1.1, I ran: $ asv run ALL -s 100 which essensitally takes 100 equally spaced commits from the whle history and run the benchmarks one those. You can view the results here: http://hera.physchem.kth.se/~sympy_asv some commits failed, maybe I should be targetting the merge commits but I haven't figured out how to do that yet. Furthermore: the comparison isn't quite fair since I belive the later versions return more correct results (Piecewise solutions), I'm not sure how to deal with this fairly, maybe some kind of assertion should be made (like a tests) at the end of the individual benchmark, a passing assertion means it should be included, a failing one that it should be excluded. But then this assertion shouldn't be included in the timing. There could be something in asv for this but I haven't figured it out yet. I am stil a bit new to using asv so I haven't quite figured out best practives here. It would be nice to have a dedicated machine (or maybe several), maybe a Raspberry Pi 2 which would make it easy for people to get the exact same hardware for comparisons? Anyways, thought I would share. Feel free to fork/send pull requests, I'll happily run them on that machine if there is an interest. Björn, thanks a lot for doing this! I need to setup something similar for SymEngine, I am currently benchmarking every commit that I think might slow things down by hand. I agree with everything that Aaron said. SymEngine is only fixing the the speed as a side-effect of being in pure Python, i.e. with C++ we are much closer to the metal, so for low level representations of the symbolic expressions we have more freedom to nail the speed, but algorithmically it's the same issue as with SymPy and everything that Aaron said applies to SymEngine as well. We can buy a dedicated machine via NumFOCUS, SymPy should have enough money there, but I don't have time personally to make it happen. If anybody is interested, let me or Aaron now. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBBSj_BRVzPGubknQYmAvEaGLYhH%2BK4B7dXR90szteXNg%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1Aggf6HP_ka_3b73QwVGv67ShwRqtqgoRnCULVDmtbxTMg%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email
Re: [sympy] Re: Being more conscious of performance
I just tried this out with jacobian() and subs() over the commits since 0.7.3 to master. It's showing me that the new caching is the killer slowdown: https://github.com/sympy/sympy/commit/a63005e4 I've submitted a PR to Björn's repo: https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files I'm assuming it uses fastcache because I have it installed, but maybe not. I'm not sure how to control for dependencies yet. Jason moorepants.info +01 530-601-9791 On Sun, Jul 19, 2015 at 10:39 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Sun, Jul 19, 2015 at 5:16 AM, Björn Dahlgren bjo...@gmail.com wrote: Hi all, On Tuesday, 14 July 2015 02:49:57 UTC+2, Aaron Meurer wrote: - Get a benchmark machine and run airspeed velocity on it. We need to catch performance regressions. The benchmark suite can be anything, although obviously well-made benchmarks are better. I found a few hours and threw this together (motivated by having noticed some things I have been doing with sympy has been getting slower): https://github.com/bjodah/sympy_benchmarks_bjodah then, on a machine with asv 0.1.1, I ran: $ asv run ALL -s 100 which essensitally takes 100 equally spaced commits from the whle history and run the benchmarks one those. You can view the results here: http://hera.physchem.kth.se/~sympy_asv some commits failed, maybe I should be targetting the merge commits but I haven't figured out how to do that yet. Furthermore: the comparison isn't quite fair since I belive the later versions return more correct results (Piecewise solutions), I'm not sure how to deal with this fairly, maybe some kind of assertion should be made (like a tests) at the end of the individual benchmark, a passing assertion means it should be included, a failing one that it should be excluded. But then this assertion shouldn't be included in the timing. There could be something in asv for this but I haven't figured it out yet. I am stil a bit new to using asv so I haven't quite figured out best practives here. It would be nice to have a dedicated machine (or maybe several), maybe a Raspberry Pi 2 which would make it easy for people to get the exact same hardware for comparisons? Anyways, thought I would share. Feel free to fork/send pull requests, I'll happily run them on that machine if there is an interest. Björn, thanks a lot for doing this! I need to setup something similar for SymEngine, I am currently benchmarking every commit that I think might slow things down by hand. I agree with everything that Aaron said. SymEngine is only fixing the the speed as a side-effect of being in pure Python, i.e. with C++ we are much closer to the metal, so for low level representations of the symbolic expressions we have more freedom to nail the speed, but algorithmically it's the same issue as with SymPy and everything that Aaron said applies to SymEngine as well. We can buy a dedicated machine via NumFOCUS, SymPy should have enough money there, but I don't have time personally to make it happen. If anybody is interested, let me or Aaron now. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBBSj_BRVzPGubknQYmAvEaGLYhH%2BK4B7dXR90szteXNg%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1Aggf6HP_ka_3b73QwVGv67ShwRqtqgoRnCULVDmtbxTMg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
Can you post your timings results? I am doing your benchmark with SymEngine and Sage for comparison. On Sun, Jul 19, 2015 at 4:57 PM, Jason Moore moorepa...@gmail.com wrote: I just tried this out with jacobian() and subs() over the commits since 0.7.3 to master. It's showing me that the new caching is the killer slowdown: https://github.com/sympy/sympy/commit/a63005e4 I've submitted a PR to Björn's repo: https://github.com/bjodah/sympy_benchmarks_bjodah/pull/1/files I'm assuming it uses fastcache because I have it installed, but maybe not. I'm not sure how to control for dependencies yet. Jason moorepants.info +01 530-601-9791 On Sun, Jul 19, 2015 at 10:39 AM, Ondřej Čertík ondrej.cer...@gmail.com wrote: On Sun, Jul 19, 2015 at 5:16 AM, Björn Dahlgren bjo...@gmail.com wrote: Hi all, On Tuesday, 14 July 2015 02:49:57 UTC+2, Aaron Meurer wrote: - Get a benchmark machine and run airspeed velocity on it. We need to catch performance regressions. The benchmark suite can be anything, although obviously well-made benchmarks are better. I found a few hours and threw this together (motivated by having noticed some things I have been doing with sympy has been getting slower): https://github.com/bjodah/sympy_benchmarks_bjodah then, on a machine with asv 0.1.1, I ran: $ asv run ALL -s 100 which essensitally takes 100 equally spaced commits from the whle history and run the benchmarks one those. You can view the results here: http://hera.physchem.kth.se/~sympy_asv some commits failed, maybe I should be targetting the merge commits but I haven't figured out how to do that yet. Furthermore: the comparison isn't quite fair since I belive the later versions return more correct results (Piecewise solutions), I'm not sure how to deal with this fairly, maybe some kind of assertion should be made (like a tests) at the end of the individual benchmark, a passing assertion means it should be included, a failing one that it should be excluded. But then this assertion shouldn't be included in the timing. There could be something in asv for this but I haven't figured it out yet. I am stil a bit new to using asv so I haven't quite figured out best practives here. It would be nice to have a dedicated machine (or maybe several), maybe a Raspberry Pi 2 which would make it easy for people to get the exact same hardware for comparisons? Anyways, thought I would share. Feel free to fork/send pull requests, I'll happily run them on that machine if there is an interest. Björn, thanks a lot for doing this! I need to setup something similar for SymEngine, I am currently benchmarking every commit that I think might slow things down by hand. I agree with everything that Aaron said. SymEngine is only fixing the the speed as a side-effect of being in pure Python, i.e. with C++ we are much closer to the metal, so for low level representations of the symbolic expressions we have more freedom to nail the speed, but algorithmically it's the same issue as with SymPy and everything that Aaron said applies to SymEngine as well. We can buy a dedicated machine via NumFOCUS, SymPy should have enough money there, but I don't have time personally to make it happen. If anybody is interested, let me or Aaron now. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBBSj_BRVzPGubknQYmAvEaGLYhH%2BK4B7dXR90szteXNg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAP7f1Aggf6HP_ka_3b73QwVGv67ShwRqtqgoRnCULVDmtbxTMg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBECZQHgY9KfpGCmymq%3DbGFWCr4LNxq6HY8UbakL6qWZg%40mail.gmail.com. For more options, visit
Re: [sympy] Re: Being more conscious of performance
Cool. For this benchmark, you're likely seeing the evolution in the integration algorithms as well. The risch algorithm doesn't handle this integral (too many symbolic parameters), but meijerg does, and so does heurisch. On Sun, Jul 19, 2015 at 6:16 AM, Björn Dahlgren bjo...@gmail.com wrote: Hi all, On Tuesday, 14 July 2015 02:49:57 UTC+2, Aaron Meurer wrote: - Get a benchmark machine and run airspeed velocity on it. We need to catch performance regressions. The benchmark suite can be anything, although obviously well-made benchmarks are better. I found a few hours and threw this together (motivated by having noticed some things I have been doing with sympy has been getting slower): https://github.com/bjodah/sympy_benchmarks_bjodah then, on a machine with asv 0.1.1, I ran: $ asv run ALL -s 100 which essensitally takes 100 equally spaced commits from the whle history and run the benchmarks one those. You can view the results here: http://hera.physchem.kth.se/~sympy_asv some commits failed, maybe I should be targetting the merge commits but I haven't figured out how to do that yet. Furthermore: the comparison isn't quite fair since I belive the later versions return more correct results (Piecewise solutions), I'm not sure how to deal with this fairly, maybe some kind of assertion should be made (like a tests) at the end of the individual benchmark, a passing assertion means it should be included, a failing one that it should be excluded. But then this assertion shouldn't be included in the timing. There could be something in asv for this but I haven't figured it out yet. I am stil a bit new to using asv so I haven't quite figured out best practives here. It would be nice to have a dedicated machine (or maybe several), maybe a Raspberry Pi 2 which would make it easy for people to get the exact same hardware for comparisons? That's a great idea. Aaron Meurer Anyways, thought I would share. Feel free to fork/send pull requests, I'll happily run them on that machine if there is an interest. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/5a7e4b53-6daa-4fc2-a8ac-a3288a0270c7%40googlegroups.com https://groups.google.com/d/msgid/sympy/5a7e4b53-6daa-4fc2-a8ac-a3288a0270c7%40googlegroups.com?utm_medium=emailutm_source=footer . For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6%2Bi13eMuPvm3bK_ME9kxpBjW9MpwWvJ5XyNJiWPYqAYMA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [sympy] Re: Being more conscious of performance
On Sun, Jul 19, 2015 at 5:16 AM, Björn Dahlgren bjo...@gmail.com wrote: Hi all, On Tuesday, 14 July 2015 02:49:57 UTC+2, Aaron Meurer wrote: - Get a benchmark machine and run airspeed velocity on it. We need to catch performance regressions. The benchmark suite can be anything, although obviously well-made benchmarks are better. I found a few hours and threw this together (motivated by having noticed some things I have been doing with sympy has been getting slower): https://github.com/bjodah/sympy_benchmarks_bjodah then, on a machine with asv 0.1.1, I ran: $ asv run ALL -s 100 which essensitally takes 100 equally spaced commits from the whle history and run the benchmarks one those. You can view the results here: http://hera.physchem.kth.se/~sympy_asv some commits failed, maybe I should be targetting the merge commits but I haven't figured out how to do that yet. Furthermore: the comparison isn't quite fair since I belive the later versions return more correct results (Piecewise solutions), I'm not sure how to deal with this fairly, maybe some kind of assertion should be made (like a tests) at the end of the individual benchmark, a passing assertion means it should be included, a failing one that it should be excluded. But then this assertion shouldn't be included in the timing. There could be something in asv for this but I haven't figured it out yet. I am stil a bit new to using asv so I haven't quite figured out best practives here. It would be nice to have a dedicated machine (or maybe several), maybe a Raspberry Pi 2 which would make it easy for people to get the exact same hardware for comparisons? Anyways, thought I would share. Feel free to fork/send pull requests, I'll happily run them on that machine if there is an interest. Björn, thanks a lot for doing this! I need to setup something similar for SymEngine, I am currently benchmarking every commit that I think might slow things down by hand. I agree with everything that Aaron said. SymEngine is only fixing the the speed as a side-effect of being in pure Python, i.e. with C++ we are much closer to the metal, so for low level representations of the symbolic expressions we have more freedom to nail the speed, but algorithmically it's the same issue as with SymPy and everything that Aaron said applies to SymEngine as well. We can buy a dedicated machine via NumFOCUS, SymPy should have enough money there, but I don't have time personally to make it happen. If anybody is interested, let me or Aaron now. Ondrej -- You received this message because you are subscribed to the Google Groups sympy group. To unsubscribe from this group and stop receiving emails from it, send an email to sympy+unsubscr...@googlegroups.com. To post to this group, send email to sympy@googlegroups.com. Visit this group at http://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CADDwiVBBSj_BRVzPGubknQYmAvEaGLYhH%2BK4B7dXR90szteXNg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
