Robyn and Russell,
I believe there should be one master global repository that contains
every line of code anyone writes that needs to be tracked so we
can always reproduce the full system when someone wants to reproduce
some analysis.
It's usually possible to draw major dividing lines that can let you
check out a major component. I agree it is important to find a way for
individual pipeline developers to get new versions of their code into
the
overall production systems. Developers can develop! And maintain their
own local version control as they see fit, but hopefully when they
are ready to integrate a major change to a pipeline, some people can
examine the new version for possible impacts on other parts of the
pipeline. This should be a
relatively infrequent event. The purpose of having one global
repository is
we can maintain complete provenance over the entire software system.
Isn't
it the complete full software system, ie everything, the actual
product? It
seems to me that we need a global master version number which allows
one to
dive down to any level of code. This grand master versioning has to be
done. We have 2 choices: do it by hand or let some computer do it
for us.
As Robyn points out, this gives people the choice of when to try
someone else's
new piece of code that someone wrote. A global repository lets them
easily
find out when others have made changes, easily examine their changes,
and
decide when to incorporate them.
--Don
Message: 7
Date: Thu, 16 Nov 2006 15:24:28 -0800
From: Russell E Owen <[EMAIL PROTECTED]>
Subject: Re: [LSST-data] [DataChallenge] SVN organization
To: Robyn Allsman <[EMAIL PROTECTED]>, Russell E Owen <[EMAIL PROTECTED]>
Cc: LSST Data Management <[email protected]>
Message-ID: <[EMAIL PROTECTED]>
Content-Type: text/plain; charset="us-ascii" ; format="flowed"
At 10:45 AM -0700 2006-11-16, Robyn Allsman wrote:
Hi Russell,
A question for an SVN expert: how to configure the LSST DM source
repository so that major sub-components (each pipeline, the
framework,
the middleware, etc) maintain their own versioning tag independent
of other
DM sub-components.
The primary motivation is to enable a developer to create a new
version
of a package/module without affecting the rest of the source tree
in any way.
Later, other developers can integrate use of the the new release into
their own packages/modules when desired (or at a convenient
time). The
existing deployment tools (pacman, eups) are currently capable of
maintaining
dependency trees for each sub-component.
On the flip side, this also enables the developer working on a
package/module
to select specific versions of other modules on which the package
is dependent
and which need to remain stable whilst in development mode.
My initial inclination is to maintain multiple SVN repositories -
one for
each *major* subcomponent. This also maps to the current reality
of developers for a module/package being clustered in close
proximity.
To support the infrequent situation where changes across multiple
repositories
need to be simultaneously implemented, a wrapper code could be
developed to
orchestrate 'checkout' from multiple repositories and 'commit' across
multiple repositories. A post-commit step could update the version
tags
in the new dependency tree. But this wrapper code is really
superfluous
since it would be little more than a 'for' loop over the list of
sub-components.
What's your opinion / suggestion?
Robyn and I talked on the phone about this. Here are my notes:
We want to break the code up into packages that are independently
versioned and controlled by something like eups and/or pacman. But I
don't think we are completely sure which bits of code become
packages. I expect those packages to match particular directories in
the svn repository (at least for the most part), but I'm also
guessing they will not be all at the same level. For example these
packages may include each subdirectory of "apps", but probably some
other grouping of the code under "mw".
From the svn perspective...it is helpful to have only one or a few
repositories because one can more easily check out chunks of code.
Fortunately svn offers a fair amount of flexibility within a
repository. One can easily check out or check in any subdirectory of
a repository. Thus even if we keep a format like DC1, it is easy for
a developer to work on a particular package or sub-package in
isolation.
So I think we can happily keep all the code in one repository, if we
can find an organization that meets our needs. I think the main
drivers are:
1) The tools like eups and pacman may have particular needs.
Unfortunately I am not clear on what they are. Robyn is looking into
this further.
Would it help to have all versioned packages at the same level in the
repository? For example they could all go in trunk/ or perhaps one
one level down distributed between trunk/apps/, trunk/mw/, etc. But
does it matter. If so...what about the URL model?:
2) The DC1 layout was chosen to match the URL model, to make it
easier to go between code and model.
That is a good feature and if the eups/packman/? tools can handle it,
I'm in favor of keeping it. But if it's not going to work for those
tools, well...I'm not clear on how much flexibility we have here.
Let's worry about it once we know what the eups/pacman/? tools need.
In addition, please don't forget the issue of building python. It
will be much easier and cleaner (from the developer and installer's
perspective) to have individually versioned python packages have a
source tree that only has python source. For example: repository
root/.../python package dir/python/package source/sub-package1
source/sub-sub-package1 source...)
-- Russell
------------------------------
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
End of LSST-data Digest, Vol 27, Issue 9
****************************************
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
