Re: [matplotlib-devel] Usefulness of Travis-CI
Hi all, Recently, it has become more and more common that the travis builds fail. They fail not because the test suite fails but because of some inherent issues with travis itself. Usually, either the git clone fails or some other simple shell command fails. I consider these 'failures' as being false negatives. While a false negative once or twice a month would not be anything to worry about, I feel like this is happening often enough to render the pull request status extremely unhelpful. I realise the above is merely my opinion, but I feel that I would be surprised if others did not find the Travis' false negative hit rate tiresome. Agreed. If practical, I would like to move away from travis and have our test suite run on a more reliable service. How do others feel about this? Good--but what is the superior alternative? There's shiningpanda: this is what scikit-learn now uses (with travis-ci): https://jenkins.shiningpanda.com/scikit-learn/ It's quite flexible, but I think you are limited to only a couple of builds per day on master. It doesn't really replace travis, as travis is supposed to run against PR but it is better than nothing. Previously, scikit-learn used buildbot, hosted on one of the french python association. I've got access to the server, and I can ask the board members if we can set up a buildbot there. I'm sure that's not a problem. But, once again, this will run the tests on master, not on the PR. Last but not least, maybe we can see what numfocus has to offer. I know nothing about who offers such services, why they do so, or the quality of the service. Eric -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Usefulness of Travis-CI
On Wed, Jan 16, 2013 at 12:10 AM, Nelle Varoquaux nelle.varoqu...@gmail.com wrote: Last but not least, maybe we can see what numfocus has to offer. Absolutely! I'll be offline for two weeks, but others on the list can certainly propose this to numfocus on the list and we can look into what can be done, esp. in a way that could also help other projects as well. Also, there's snakebite: http://www.snakebite.net. The project seemed very dormant for a long time, but there's been some activity since: http://www.snakebite.net/network. I'd ping Titus Brown on Twitter (@ctitusbrown) for info... Cheers, f -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Usefulness of Travis-CI
On 16 Jan 2013 09:30, Fernando Perez fperez@gmail.com wrote: On Wed, Jan 16, 2013 at 12:10 AM, Nelle Varoquaux nelle.varoqu...@gmail.com wrote: Last but not least, maybe we can see what numfocus has to offer. Absolutely! I'll be offline for two weeks, but others on the list can certainly propose this to numfocus on the list and we can look into what can be done, esp. in a way that could also help other projects as well. When this has come up on the numfocus list before the status was, sure, they can provide resources/funding if you can find someone to do the actual work :-). If anyone is interested in putting in the time to solve this problem properly then a lot of projects would be grateful I think... -n -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Usefulness of Travis-CI
On 16 January 2013 08:35, Nathaniel Smith n...@pobox.com wrote: When this has come up on the numfocus list before the status was, sure, they can provide resources/funding if you can find someone to do the actual work :-). If anyone is interested in putting in the time to solve this problem properly then a lot of projects would be grateful I think... We'd certainly be interested in a better solution for IPython - I think we see more Travis builds failing because of Travis than because of the actual code. Before Travis started testing pull requests, we wrote our own test_pr.py script [1], which fetches a PR, merges it into master, tests on several versions of Python, and (optionally) posts a comment to Github with the results. This should be pretty easy to adapt to other projects - NetworkX already decide to use it [2]. With a little more code, I imagine we could configure a server to regularly scan pull requests for changes and test them. Sympy also have a system for automatically testing pull requests [3]. I think this is somewhat more advanced, with a queue on Google Appengine, and workers which pull jobs from the queue. Finally, we're keen users of ShiningPanda [4]. It doesn't test pull requests, but it's reliable, and can handle things like coverage metrics nicely. [1] https://github.com/ipython/ipython/blob/master/tools/test_pr.py [2] https://github.com/networkx/networkx/pull/752 [3] http://reviews.sympy.org/ [4] https://jenkins.shiningpanda.com/ipython/ Best wishes, Thomas -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Usefulness of Travis-CI
On 01/16/2013 07:15 AM, Thomas Kluyver wrote: On 16 January 2013 08:35, Nathaniel Smith n...@pobox.com mailto:n...@pobox.com wrote: When this has come up on the numfocus list before the status was, sure, they can provide resources/funding if you can find someone to do the actual work :-). If anyone is interested in putting in the time to solve this problem properly then a lot of projects would be grateful I think... We'd certainly be interested in a better solution for IPython - I think we see more Travis builds failing because of Travis than because of the actual code. Before Travis started testing pull requests, we wrote our own test_pr.py script [1], which fetches a PR, merges it into master, tests on several versions of Python, and (optionally) posts a comment to Github with the results. This should be pretty easy to adapt to other projects - NetworkX already decide to use it [2]. With a little more code, I imagine we could configure a server to regularly scan pull requests for changes and test them. This definitely would be worth investigating. At least in my institution, finding CPU time isn't terribly difficult. Hosting our own network services is -- so something that does builds locally and then publishes the results somewhere else (i.e. Github) may make a lot of sense. Sympy also have a system for automatically testing pull requests [3]. I think this is somewhat more advanced, with a queue on Google Appengine, and workers which pull jobs from the queue. Finally, we're keen users of ShiningPanda [4]. It doesn't test pull requests, but it's reliable, and can handle things like coverage metrics nicely. The matplotlib test suite is really CPU-intensive, and the last time I did the math, the ShiningPanda free plan was too tight -- it's about an hour of CPU time (on ShiningPanda's machines) per test for 4 Numpy/Python combinations. It's about $0.50/hr, and you get $6 free per month. I really like ShiningPanda, mostly for how flexible the setup of prerequisites are, but it will cost some money to use. Cheers, Mike [1] https://github.com/ipython/ipython/blob/master/tools/test_pr.py [2] https://github.com/networkx/networkx/pull/752 [3] http://reviews.sympy.org/ [4] https://jenkins.shiningpanda.com/ipython/ Best wishes, Thomas -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Usefulness of Travis-CI
On Tue, Jan 15, 2013 at 6:20 PM, Damon McDougall damon.mcdoug...@gmail.comwrote: Hi all, Recently, it has become more and more common that the travis builds fail. They fail not because the test suite fails but because of some inherent issues with travis itself. Usually, either the git clone fails or some other simple shell command fails. I consider these 'failures' as being false negatives. While a false negative once or twice a month would not be anything to worry about, I feel like this is happening often enough to render the pull request status extremely unhelpful. I haven't been active enough lately to have a good feel overall on the false negatives. What I can say right now is that it would help if we would disable python 3.1 (and optionally replace it with 3.3). I believe (based on numpy commit I saw the other day) that Travis no longer supports 3.1. That would solve the failures I've seen with my PR's. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Usefulness of Travis-CI
On Jan 16, 2013 9:30 AM, Fernando Perez fperez@gmail.com wrote: On Wed, Jan 16, 2013 at 12:10 AM, Nelle Varoquaux nelle.varoqu...@gmail.com wrote: Last but not least, maybe we can see what numfocus has to offer. Absolutely! I'll be offline for two weeks, but others on the list can certainly propose this to numfocus on the list and we can look into what can be done, esp. in a way that could also help other projects as well. Also, there's snakebite: http://www.snakebite.net. The project seemed very dormant for a long time, but there's been some activity since: http://www.snakebite.net/network. I'd ping Titus Brown on Twitter (@ctitusbrown) for info... Cheers, There is also the open build service, which is more of a build farm but can be set up pull, build, test, and publish git snapshots for most common Linux distributions to your own online software repository simultaneously with one click on a website or one commandline call. https://build.opensuse.org/ They provide hosting, building, and distribution for free. You can probably set up a script to automatically rebuild master on a change, or daily. However, setting it up to test individual commits would be overly difficult and probably be seen as an abuse of the system. Using it to always build, test, and release offer the latest master to most linux distros, on the other hand, would be fine. If someone contacts them they can probably set up a repository just for you, or if this sort of thing is useful a more general repository you can share with others (there is already devel:languages:python, maybe devel:languages:python:unstable). You can also use it to build release rpms and debs for various distros. It is already being used to build the packages discussed so far for openSUSE, but if someone is willing to maintain them they can be built for other distros as well. -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Usefulness of Travis-CI
On Wed, Jan 16, 2013 at 10:54 AM, Todd toddr...@gmail.com wrote: On Jan 16, 2013 9:30 AM, Fernando Perez fperez@gmail.com wrote: On Wed, Jan 16, 2013 at 12:10 AM, Nelle Varoquaux nelle.varoqu...@gmail.com wrote: Last but not least, maybe we can see what numfocus has to offer. Absolutely! I'll be offline for two weeks, but others on the list can certainly propose this to numfocus on the list and we can look into what can be done, esp. in a way that could also help other projects as well. Also, there's snakebite: http://www.snakebite.net. The project seemed very dormant for a long time, but there's been some activity since: http://www.snakebite.net/network. I'd ping Titus Brown on Twitter (@ctitusbrown) for info... Cheers, There is also the open build service, which is more of a build farm but can be set up pull, build, test, and publish git snapshots for most common Linux distributions to your own online software repository simultaneously with one click on a website or one commandline call. https://build.opensuse.org/ They provide hosting, building, and distribution for free. You can probably set up a script to automatically rebuild master on a change, or daily. However, setting it up to test individual commits would be overly difficult and probably be seen as an abuse of the system. Using it to always build, test, and release offer the latest master to most linux distros, on the other hand, would be fine. If someone contacts them they can probably set up a repository just for you, or if this sort of thing is useful a more general repository you can share with others (there is already devel:languages:python, maybe devel:languages:python:unstable). You can also use it to build release rpms and debs for various distros. It is already being used to build the packages discussed so far for openSUSE, but if someone is willing to maintain them they can be built for other distros as well. GitHub allow for a custom service-hook. If, as Mike says, it's not too hard to garner compute cycles, it shouldn't be too hard to write a small script to execute the test suite when the github repo receives a push. -- Damon McDougall http://www.damon-is-a-geek.com Institute for Computational Engineering Sciences 201 E. 24th St. Stop C0200 The University of Texas at Austin Austin, TX 78712-1229 -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
Re: [matplotlib-devel] Usefulness of Travis-CI
On 2013/01/15 2:20 PM, Damon McDougall wrote: Hi all, Recently, it has become more and more common that the travis builds fail. They fail not because the test suite fails but because of some inherent issues with travis itself. Usually, either the git clone fails or some other simple shell command fails. I consider these 'failures' as being false negatives. While a false negative once or twice a month would not be anything to worry about, I feel like this is happening often enough to render the pull request status extremely unhelpful. I realise the above is merely my opinion, but I feel that I would be surprised if others did not find the Travis' false negative hit rate tiresome. Agreed. If practical, I would like to move away from travis and have our test suite run on a more reliable service. How do others feel about this? Good--but what is the superior alternative? I know nothing about who offers such services, why they do so, or the quality of the service. Eric -- Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery and much more. Keep your Java skills current with LearnJavaNow - 200+ hours of step-by-step video tutorials by Java experts. SALE $49.99 this month only -- learn more at: http://p.sf.net/sfu/learnmore_122612 ___ Matplotlib-devel mailing list Matplotlib-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-devel
