There's NumFOCUS money for CI -- also for the work used to configure and
running it etc.
It would not be only for Cython, but for open source scientific Python
(Cython, NumPy, SciPy etc.). So if anybody would like to work up to 50%
time on that (or perhaps just want to work for some weeks with setting
up a CI system that can be shared by said projects) then please get in
touch with Travis and NumFOCUS (see thread below).
Some context:
There's been heavy discussion on the NumFOCUS list on identifying a
"core subset" of packages (Cython+NumPy+SciPy+Pandas+...) that could be
tested and promoted together [1].
I started a discussion [2] to try to divert the question of CI-ing that
core subset in the usual way ("release managing a software
distribution") to instead be about giving those resources to the open
source projects involved, so that there's not duplicate work in doing
the CI-ing.
Of course, Cython is in better standing than most already on the CI
front thanks to Stefan and the Sage project, but the fact that the work
is paid for may perhaps make it more appealing.
Full threads:
[1] https://groups.google.com/forum/?fromgroups#!topic/numfocus/MnRzBhmqXqk
[2] https://groups.google.com/forum/?fromgroups#!topic/numfocus/I_kmL4FUGaY
Dag
-------- Original Message --------
Subject: Re: Continuous integration
Date: Mon, 28 May 2012 13:47:52 -0500
From: Travis Oliphant <teoliph...@gmail.com>
Reply-To: numfo...@googlegroups.com
To: numfo...@googlegroups.com
On May 28, 2012, at 1:27 PM, Dag Sverre Seljebotn wrote:
On 05/28/2012 05:55 PM, David wrote:
On Sunday, May 27, 2012 3:05:06 AM UTC+9, Dag Sverre Seljebotn wrote:
That Other Thread contained some references to CI. So I'm mainly
wondering what the current NumFOCUS plans for supporting CI efforts
are,
if any? Was there a mention on money being available for somebody to
work on that?
I think of Cython as one vertex in a CI graph:
a) Upstream, Cython depends on various versions of NumPy and Python as
part of its CI
b) Downstream, we rely on building and testing Sage as an extended
regression test suite. It's not just that our test suite doesn't have
100% coverage, but also sometimes that we intentionally break backwards
compatibility, and fixing up Sage gives a nice indication of the
consequences of a change. (I imagine NumPy is in a very similar
situations with lots of libraries depending on it and potential for
very
subtle breakage of backwards compatibility.)
Ideally, we'd have LOTS of libraries using Cython in our CI -- mpi4py,
Pandas, scikits-learn... BUT, we're running into problems as it is with
the CI server hardware keeping up (if there was infinite CI there'd
probably be a lot more tests set up).
Since most scientific-python libraries have both dependencies and
dependees, it seems like there should be some benefit to having the
same
CI system test all of them. That could conserve both hardware use and
administration overhead. And, which I believe is very important,
make it
easier for small projects like scikits-sparse to participate in
automatic CI by simply participating in an environment maintained by
the
larger projects like Cython and NumPy.
I think there's two ways of spinning this:
1) Build "Sciome" in a CI and approach it by "release managing" Sciome
2) Focus on providing "infrastructure for library developers", where
there's a relatively big CI graph where each project has a node. I.e.
something like "ShiningPanda for scientific Python", but with a
critical
difference being that each project can use the build artifacts of
others; Cython would flip a switch and have all projects depending on
Cython being rebuilt to try the new Cython version. This seems
complicated, but certainly Jenkins for one seems to support such setups
already so its mainly about hardware and administration...
I know I get a lot more excited by 2) than 1), even if it's perhaps
mainly the spin put on it rather than a technical difference.
I know I prefer 2 as well. I know of at least one attempt of doing
something like this (linux-only, though): the build service from open
suse (http://www.open-build-service.org/). 4 years ago, it could already
do useful bits that are not trivial in any CI I am aware of:
- dependencies between packages is known: if you update the build
service package numpy, and the build is successful, it will
automatically rebuild all the dependent packages
- it handles completely isolated build environments through vms
- it produces rpm, debian, etc.. that can be easily installed in the
supported distributions.
My ideal setup would be something like this, but working on windows and
mac (and not developed in perl), and with easier set up ala travis CI. I
don't know yet the best way to go there: is is based on existing
infrastructure (jenkins, travis, something else), building our own, a
hybrid between the two ?
What I was imagined was really something very simple (and I put it too
convoluted). Just have NumFOCUS cash out :-) for one or more servers, then give
everybody who wants shell access to a shared CI instance (running Jenkins or
whatever).
This sounds like a great plan. NumFOCUS can provide for this. We
will just need a budget and board approval (but I suspect the board will
be happy to approve this kind of plan).
So, who wants to spec out the machine? There is some money. The
problem is actually a people problem. Who is going to be available to
do the work. So, far, I have not been able to find anyone experienced
who is willing to work on this stuff at 50% time.
If you are willing, let me know in a private email or an email to
ad...@numfocus.org
Best,
-Travis
_______________________________________________
cython-devel mailing list
cython-devel@python.org
http://mail.python.org/mailman/listinfo/cython-devel
