We need a plan for organizing our python code for DC1. In particular:
- Organizing the installed python package structure (so developers
can import code).
- Organizing the python source code in the svn repository
- Managing installation (how many installers, what do they install?).
This is a preliminary standard put together by Russell, Ray and
Michelle. We tried to make this general enough that it could apply to
future data challenges and (presumably with some evolution) LSST
itself. I have also appended an email thread that is relevant.
* The python package structure will approximately reflect the svn
source code layout, but in turn reflects the UML model. However:
- Names are abbreviated and lower case.
- Branches will be pruned as needed (and empty branches are omitted)
Thus:
lsst
apps (contains sextractor, wcs, etc.)
mw
pipe (contains policy, log and queue)
daf (contains fitsio and dbingest)
* The python code will be divided into separately installable
packages based on common functionality and individually versioned
units. For DC1 we primarily see these as being:
- lsst.mw: middleware code such as logging, policy, queue and fitsio.
- lsst.apps: individual lsst applications/pipeline stages, such as
sextractor and wcs.
* Command-line applications (including scripts) need some naming
convention to avoid name collisions. Ray suggests 3-letter prefixes
-- nice and short, but somebody has to pick them. Meanwhile, for the
examples below I just prepend the package name. (Alternatives include
doing both, or only prepending the last two components of the package
name). We also need to decide whether to use the language suffix. I
personally prefer not to, but either standard is better than none.
* The source code is arranged as follows. Each installable python
package is contained in its own directory, such that:
- It is be put in the appropriate place in the svn hierarchy.
- Its name has a prefix of py.
- It includes all python code, documentation, an installer (setup.py
file), any python-specific C/C++ code (e.g. code required to
interface to some existing library), swig wrappers, command-line
scripts, etc. -- organized using the lsst-DC1-standard directories.
For example (I've combined all middleware python code into one python
package since discussing this with Ray and Michelle; if that proves
controversial it can easily be undone):
apps/
pysextractor/ (source for lsst.apps.sextractor)
...
middleware/
daf/
pyfitsio/
...
pydbingest/
...
pipe/
pymw/ (source code for the lsst.mw package)
setup.py
docs/ (all documentation goes here; flat unless hierarchy needed)
python/
daf/ (data access framework)
__init__.py (probably empty)
fitsio/ (fitsio package)
dbingest/ (dbingest package)
...
pipe/
__init__.py (probably empty)
log.py
queue.py
Policy.py
...(other modules and packages)
scripts/
dbingestinit
dbingestworker
queueget
queueput
...
pylsst/ (files to install basic empty lsst base)
setup.py
python
lsst/
__init__.py (empty)
apps/
__init__.py (empty)
mw
__init__.py (empty)
* Responsibilities:
- Developers are responsible for __init__.py files in their own
subpackages and setup.py files for their own independently installed
packages (but I am happy to help).
- I am happy to reorganize the svn repository and write the higher
level __init__.py files and the lsst.mw setup.py file once we do
agree on the standard.
-- Russell
Here is a thread that was relevant to developing this standard (with
minor editing/simplification):
***** At 9:28 AM -0700 2006-06-22, Russell E Owen wrote:
Some kinds of code, such as image processing algorithms, will
probably want to be individually versioned -- so that we can revert
to earlier versions or swap between trial versions as desired. These
would make sense to be individual packages, or subpackages....
By "subpackage" I mean packages that live down a level or two in the
namespace. Thus, for example:
import lsstapps.<subpackage>
or
import lsst.apps.<subpackage>
To do this, we have one installer that creates the root
directory/directories, then other installers that populate it (and a
script to run them all, but during development we'll run the
individual installers to install new versions).
To keep it sane, the __init__.py in their parent directory should
have no reference to the contents of the subpackages (so it doesn't
need to be modified as subpackages are installed or deinstalled).
I think this ties in perfectly with Robert Lupton's code management
system -- the code just has to be copied to the right subdirectory.
For the "framework" code -- general, reusable code that all other
code can rely on (the lsst version of "dervish" or "pslib"), I
suggest we have just one or a very few packages -- try to keep it
simple.
***** At 5:05 PM -0700 2006-06-22, [EMAIL PROTECTED] replied to Russell:
I like this idea, as I am a big fan of KISS (keep it simple stupid.. ;)
I especially like the idea of doing subpackage installs underneath without
affecting everything above it.
***** At 12:49 PM -0500 2006-06-23, Ray Plante replied to Russell:
On Thu, 22 Jun 2006, Russell E Owen wrote:
I think this ties in perfectly with Robert Lupton's code management
system -- the code just has to be copied to the right subdirectory.
This is what I was hoping for (in my Java/C++ skewed perspective).
To keep it sane, the __init__.py in their parent directory should
have no reference to the contents of the subpackages (so it doesn't
need to be modified as subpackages are installed or deinstalled).
Sounds good. What, then, should the __init__.py include?
For the "framework" code -- general, reusable code that all other
code can rely on (the lsst version of "dervish" or "pslib"), I
suggest we have just one or a very few packages -- try to keep it
simple.
What does this mean practically? Are we doing some kind of integrating of
the code for logging, policy, queues, FITS I/O, ...? Perhaps another way
to ask this is, should each of these be a *.py file under lsst/mw/pipe/fw
with one __init.py__ "to rule them all" or is each represented as a
directory, each with its own __init__.py?
**** At 12:36 PM -0700 2006-06-23, Russell E Owen replied to Ray:
At 12:49 PM -0500 2006-06-23, Ray Plante wrote:
On Thu, 22 Jun 2006, Russell E Owen wrote:
I think this ties in perfectly with Robert Lupton's code management
system -- the code just has to be copied to the right subdirectory.
This is what I was hoping for (in my Java/C++ skewed perspective).
Yes--your suggestion was a good one.
> To keep it sane, the __init__.py in their parent directory should
have no reference to the contents of the subpackages (so it doesn't
need to be modified as subpackages are installed or deinstalled).
Sounds good. What, then, should the __init__.py include?
That depends. If that package contains any general-purpose code (in
addition to the individually versioned sub-packages) then the
__init__.py may contain some import statements to make using that
code easier. Otherwise it'll probably be blank. I don't much care so
long as it doesn't contain any reference to the individually
versioned sub-packages.
> For the "framework" code -- general, reusable code that all other
code can rely on (the lsst version of "dervish" or "pslib"), I
suggest we have just one or a very few packages -- try to keep it
simple.
What does this mean practically? Are we doing some kind of integrating of
the code for logging, policy, queues, FITS I/O, ...? Perhaps another way
to ask this is, should each of these be a *.py file under lsst/mw/pipe/fw
with one __init.py__ "to rule them all" or is each represented as a
directory, each with its own __init__.py?
I hope we'll integrate those modules into a single package.
I'm not entirely sure this answers the rest of your question, but...
What would help most, in my opinion, is if each package or subpackage
that has its own installer and its own version number be organized
into the same directory structure as the package itself. Then it
could easily have one setup.py file to install it, and life would be
good.
Then the details of the various __init__.py files are not very
important. We fill them out in a way that makes the code import
statements "sane". It's not hard to do. The people who build the
various modules can pretty much ignore the issue *unless* they supply
a whole package, in which case they should supply a reasonable
__init__.py (that only applies to their package).
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
