On Fri, 11 Mar 2022 at 14:53, praneeth ratna <[email protected]> wrote: > > Hi all, > > As mentioned in the project description here , > https://github.com/sympy/sympy/wiki/GSoC-Ideas#benchmarks-and-performance , > the sympy_benchmarks misses some benchmarks which are needed to be shifted > fro the sympy main repo. Also the sympy_benchmarks misses benchmarks for > combinatorics, series , stats and many other modules which can be > added.(Please correct me here) > Also I did not have a clear understanding of what the new implementation of > benchmarks do and how it should be done.
I think that with some of the project ideas the intention is not necessarily that anyone has a clear idea of exactly what should be done but that someone wanting to do a project would do some research themselves to come up with an idea. In relation to benchmarks the immediate issues I see are: 1. Some old benchmarking code is in the main repo and should be moved over to the benchmarks repo. 2. The benchmarks repo is currently broken because some test fails so no PR can be merged there until that is fixed. 3. The benchmarks are affected by caching which means that many of the timings reported are completely unrepresentative. Bigger issues: 1. The benchmarks are based on ASV which is awkward to use. It's unnecessarily difficult even just to run a single benchmark. A better framework should be found. 2. The benchmarks mostly only measure things that are already quite fast because otherwise they would take too long to run. To guide future improvements we really need benchmarks for things that are currently slow. 3. The benchmarks are too tied up with sympy itself. I would rather have an independent benchmark suite that collects good examples and can be used by other projects as well. For this you need benchmarks that are language agnostic and not simply written in sympy code. (I also think it would be better if a lot of the test suite was like this as well.) 4. The things that are tested in the benchmarks are a bit random. It would be better to focus on core features like differentiation, integration, linear algebra, polynomials, solvers, numerical evaluation. A big part of the difficulty in making good benchmarks is that it really requires having some understanding of what are the core operations that should or could be made faster. -- Oscar -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAHVvXxRPFjmZBrkJpusB-WER3V%3DMd%2B%2BP_X8HKAuHHJr3PPXe7Q%40mail.gmail.com.
