I think the answer to 1 and 3 depend on what the server limitations are. If the server can handle running the tests after each commit, then I say go for it. Depending on what the setup is, this might also include cost limitations. So I would defer to Ondrej to any decisions on that.
Regarding 2, I guess posting to IRC if a regression occurs in master would be alright, as long as it isn't too spammy (you can make it only report each regression once, right?) It definitely should report results back to pull requests if it is just testing an unmerged pull request. We might consider doing some smart stuff like reopening a pull request and/or issue if it can determine that that pull request/issue caused a regression in master. By the way, as a recent thread here has pointed out, we should also test SymPy with the dependencies numpy and scipy, because some parts of the code use that. Perhaps we should just add numpy and scipy to the gmpy tests if it would make it too many combinations (I think it's unlikely that something would break with numpy or scipy installed but only when gmpy is not installed, as there is presently no part of sympy to my knowledge that uses both of those). Aaron Meurer On Sun, Jun 19, 2011 at 4:57 AM, Vladimir Perić <[email protected]> wrote: > Hello, > > As you may or may not be aware, part of my GSoC work is to set up a Jenkins > continuous integration server (see my blog post on the subject for some more > info[1]). As I announced on IRC a few days ago, there is a "test" server up > now on: > > http://72.14.182.119:8080/ > > Anonymous users can see the build history, including the output for each > test run. Registered users can run testa and generally manage Jenkins. If > you'd like an account, please send me a mail or contact me on IRC. The > current installation is a "test" one so we can play around it; I'll set up > the "live" Jenkins installation later this week (Ondřej and me will > reinstall the whole server to the latest Ubuntu). Currently things work as > it should be expected, though there are some small issues to iron out. Once > we have the basic functionality set up, the idea is to somehow have Jenkins > run all tests on every pull request (a sympy-bot upgrade, if you will). > Before we do that, we should decide on some settings of Jenkins: > > 1) How often should Jenkins run the tests? Currently, we only do it > manually, but it's possible to schedule Jenkins to run at specific times > (daily, twice a day..), or after every commit? I think running after every > commit is a bit excessive (as we need to run the test suite at least 6 > times, py25-py27 x gmpy, and that's already about one hour). I think running > twice a day would be more than enough for the amount of activity we have > currently, just daily is probably fine. On the other hand, the Jenkins > developers recommend running after every change (as that's the point of > continuous integration). > > 2) How should we report the results? The Jenkins interface is confusing at > best and it isn't realistic to expect all developers to check it all the > time. A common solution to this is to have it announce failures on IRC > automatically (we could report successes too, but that's a lot spam for no > practical gain). I could also set it up to send mails somewhere, but that > too might be excessive. If we run after each commit, though, it ought to be > possible to mail the author of the offending commit automatically. For the > moment, though, I think reporting failures on IRC is enough. > > 3) More of a longer-term consideration, but it would be useful to have at > least one additional Windows and Mac client. If anyone has suggestions... > > 4) Anything else to talk about? > > > Finally, I'd just like to mention that we've already found two test > failures/bugs with Jenkins, so there is definitely value to using it. > > > [1] http://vperic.blogspot.com/2011/05/continuous-integration-and-sympy.html > > -- > Vladimir Perić > > -- > You received this message because you are subscribed to the Google Groups > "sympy" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/sympy?hl=en. > -- You received this message because you are subscribed to the Google Groups "sympy" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/sympy?hl=en.
