Achieving such a change is a often more of a social problem than a technical
one...
From: Tom Roche tom_ro...@pobox.com Sent: Monday, April 23, 2012 1:06 AM
summary: background, motivation, and plan for a tree-structured
development
process is presented. My question is (roughly), should the code bucket
to
which code is committed at each level of the process be implemented by a
separate repository, a separate branch on a single repository, something
else, or does it matter?
details:
Apologies for the length of this post, but it seems there's a lot to
explain. In reverse order, I have a question about a plan, for which I
present the motivation and some background: feel free to skip over parts,
but I suspect it all ties together.
BACKGROUND
I recently began to work with a group that recently had an embarrassingly
extended release. The short story is, we kept throwing what we thought was
good code over the wall to the beta testers, who kept throwing it
back--for
5 months. The long story/etiology includes:
1 Our people are software engineers only by default. They're really
scientists who code, and who have learned a bit about software engineering
just by doing. But their focus has been on what they do with their code,
not their tools or development process (until now), which can seem pretty
crude (at least, to a coder who's starting to learn the science, like me).
Been there. Been that engineer.
2 We have a very centralized dev process. There is one CVS repo to which
everyone commits. (Technically, there are several: since they don't know
how
to branch or create read-only users, they just clone the filesystem
everytime they want to freeze something. But for commits there is only one
repo.) Everyone commits to HEAD, for the reasons why most CVS users don't
branch. Theoretically everyone runs a big bucket o' tests before
committing;
in practice, there's a small group (2 guys) who manage releases and
actually/reliably test.
3 We have a very long release cycle: several years, for which there are
apparently some legitimate reasons. But we don't do intermediate
integrations, or manage dependencies; ISTM, that's just slack, and means
that pre-release testing follows a painful pre-release integration of code
from our many contributors.
Related to this etiology are the following continuing constraints:
4 Resource: our funding is flat, and our group's headcount is actually
declining (retirees are not being replaced). We are supplied with
contractors who service our clusters (more below), but no other computing
support (other than desktop support for productivity apps like Lotus
Notes). We need more contributions from our community of users (which I
suspect many could/would give), but, for legal reasons (not related to
licensing--the code is open-source), it's hard for us to enable access to
code that has not been fully reviewed. (More on excessive security
below.)
These are longer-term problems :-(
5 Automated testing of large-scale scientific models seems inherently
hard.
(If there's anyone out there working on this problem, please ping me
offline--I'd like to learn more.) There are ways to attack this in which
I'm
definitely interested, but that's also a longer-term problem.
Some interesting papers:
http://http.icsi.berkeley.edu/ftp/pub/speech/papers/wikipapers/cox_harris_testing_numerical_software.pdf
http://www.cs.ua.edu/~SECSE09/Presentations/09_Hook.pdf
http://www.associationforsoftwaretesting.org/?dl_name=DianeKellyRebeccaSanders_TheChallengeOfTestingScientificSoftware_paper.pdf
The matmute.sourceforge.net described in 09_Hook looks interesting for
gaining some level of _statistical_ confidence in the quality of the
software. Also see Kelly's ref 8 about just how wrong scientific software
can be.
6 We are not mobile developers. We run and test our code on a couple of
clusters which are behind some exceedingly strict firewalls--so strict
that
few folks have the ability to VPN (aggravated by the resource constraint),
and it's painful for those that do. We can't ssh or https out of the
cluster, which complicates sharing of code (via, e.g., github) and data.
Hence folks work on code almost entirely from their desks (which are on
LANs
that have cluster access) and not from home or on the road. This is also
not
likely to change anytime soon.
MOTIVATION
My group intensively uses our tool for our scientific work (we majorly
eat
our own dogfood), but we also have a significant external community of
users. The 5-month delay of an announced release was therefore rather
embarrassing, and we also realize that it wasted lots of time/effort. Now
that we're planning for the next release, I'm proposing some process
upgrades to address those problems. Some proposals are no-brainers, or at
least are off-topic for this post:
* CVS - git: the following plan presupposes we do this. This is not quite
a
no-brainer, since we'll hafta train folks how to use git, but I can't see
disadvantages to migration that
