Hi All, I'd like to begin adding some additional benchmarks to SymPy to help inform the code generation work that I'm doing as part of the CZI grant.
I'm aware of the benchmarks in the benchmarks repository <https://github.com/sympy/sympy_benchmarks>. My understanding is that these are run using airspeed velocity as part of the CI, and track how the performance of a particular benchmark has changed relative to the most recent SymPy release and the master branch. There are two other types of benchmark that I think might be useful: *1. Comparison of multiple ways to do equivalent computations* Below is a contrived example in which there are two functions, option_1 and option_2, that produce the same result but have different implementations. >>> from sympy import Matrix, symbols >>> >>> def option_1(a, b): ... return Matrix([a+b, a*b]).jacobian(Matrix([a, b])) ... >>> def option_2(a, b): ... Matrix([[(a+b).diff(a), (a+b).diff(b)], ... [(a*b).diff(a), (a*b).diff(b)]]) ... >>> a, b = symbols(“a, b”) >>> option_1(a, b) == option_2(a, b) True A benchmark in this case would time the execution of both option_1 and option_2 (for a range of inputs), compare the relative speeds, and report the differences. As this type of benchmark is not comparing the same benchmark across different SymPy versions, I believe that airspeed velocity may not be the best tool to use here. I see this type of benchmark as being useful for: (1) determining which algorithm to use when implementing a new function or refactoring an existing function; and (2) ensuring that an implementation remains superior to alternatives as changes are made elsewhere in SymPy. I have had success in the past implementing these sorts of benchmarks using pytest-benchmark <https://pytest-benchmark.readthedocs.io/en/latest/>. Is there currently anything similar anywhere is SymPy? Would the sympy/sympy_benchmarks repository be the best place to contribute PRs for these sorts of benchmarks? Does anyone have any differing opinions about how and where these should be implemented, or the value of this type of benchmark? *2. Measurement of non-time metrics* Below is another contrived example in which common subexpression elimination is used on an expression, y, and it is shown that the result of cse(y) involves fewer operations that the original expression. >>> from sympy import count_ops, cse, exp, sin, symbols >>> >>> a, b = symbols(“a, b”) >>> y = (sin(a/b) + (a/b) - exp(b)) * ((a/b) - exp(b)) >>> >>> count_ops(y) 10 >>> count_ops(cse(y)) 6 A benchmark in this case would count the number of operations in the return value from cse(y) and compare this to 6. Assuming that the implementation of the cse function has been changed, if the number of operations is six then we know that its performance hasn’t been changed by the refactor. If the count is greater than six a regression has taken place. If the count is less than six the performance of the function has been improved. Benchmarking for a range of inputs would obviously be required. I see this type of benchmark as being useful for: (1) measuring SymPy’s performance in instances where timing code snippets isn’t necessarily the best, or only valuable, indicator of performance; and (2) ensuring regressions haven’t occurred during refactoring. I believe this type of benchmark can be implemented using airspeed velocity’s track prefix. Or perhaps this type of benchmark would be best implemented as regression tests in the sympy/sympy repository’s test suite, comparing the non-time metrics to hard-coded values. As before, is there currently anything similar anywhere is SymPy? Should PRs for these sorts of benchmarks be contributed to the sympy/sympy_benchmarks repository using airspeed velocity's track or as regression tests in the sympy/sympy repository? Does anyone have any differing opinions about how and where these should be implemented, or the value of this type of benchmark? Sam -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/36e40796-3caa-4aa1-9753-1606773e9288n%40googlegroups.com.
