Thanks. My biggest question has been how we can do it in the cloud. Most CI services run multiple concurrent jobs on the same machine, making the performance inconsistent. Does drone.io let you have a dedicated machine?
If we can't do it in the cloud, another alternative would be to buy a cheap dedicated machine that we can provision ourselves. We have funds in our NumFOCUS account to do either of these, so the question is which will be easier for us to set up and maintain. What do the sumpy benchmarks run on? Aaron Meurer On Mon, May 20, 2019 at 1:59 PM Isuru Fernando <[email protected]> wrote: > > Hi, > > I setup asv in Gitlab CI for https://gitlab.tiker.net/inducer/sumpy. It runs > the PR head and master (cached) and errors if there is a test with 10% > regression. > Script is at > https://gitlab.tiker.net/inducer/ci-support/blob/master/build-and-benchmark-py-project.sh. > It runs on a dedicated Gitlab runner. > > For sympy, you'll have to have a dedicated machine for benchmarking and some > CI service that integrates with github installed. (Maybe Drone CI which is > free for 15000 builds per year. > https://drone.io/enterprise/starter/#features). You'll have to limit to 1 > concurrent job to avoid multiple PRs affecting each other. > > Isuru > > On Mon, May 20, 2019 at 1:20 PM Aaron Meurer <[email protected]> wrote: >> >> Hello everyone. >> >> I would like to start looking into setting up some kind of >> benchmarking CI for SymPy. >> >> Ideally the CI would work just like Travis or codecov. It would run >> the benchmark suite on every pull request, and report if there are any >> major performance regressions. >> >> If anyone has any thoughts on how to go about doing this, let me know. >> If anyone knows of any projects that already do this, let me know that >> as well. >> >> Finally, if anyone is interested in helping out with this, please let me >> know. >> >> Aaron Meurer >> >> -- >> You received this message because you are subscribed to the Google Groups >> "sympy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/sympy. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/sympy/CAKgW%3D6Lks6y8CKbiBRauwD1DQ5%3DcnMzQNbXuhhD9MefbN%3DQQrQ%40mail.gmail.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/sympy. > To view this discussion on the web visit > https://groups.google.com/d/msgid/sympy/CA%2B01voNuyrPRemoJ-3Q0HgucebO4%3DB1g0JScFqs0WwMj3-5odg%40mail.gmail.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "sympy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/sympy. To view this discussion on the web visit https://groups.google.com/d/msgid/sympy/CAKgW%3D6L%3Ddx%3DXpxEH%2BBG%2B9nW%3DaLWceAsyRSv2xD6Zm-iQCHmDLw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
