The sample svn directory layout had some serious errors. Here is a
corrected version of the whole proposal without the appended email
thread.
We need a plan for organizing our python code for DC1. In particular:
- Organizing the installed python package structure (so developers
can import code).
- Organizing the python source code in the svn repository
- Managing installation (how many installers, what do they install?).
This is a preliminary standard put together by Russell, Ray and
Michelle. We tried to make this general enough that it can be used
for future data challenges and (presumably with some evolution) LSST
itself.
* The python package structure will approximately reflect the svn
source code layout, which in turn reflects the UML model. However:
- Names are abbreviated and lower case.
- Branches will be pruned as needed (and empty branches omitted)
Thus:
lsst
apps (contains sextractor, wcs, etc.)
mw
pipe (contains policy, log and queue)
daf (contains fitsio and dbingest)
* The python code will be divided into separately installable
packages based on common functionality and individually versioned
units. For DC1 we primarily see these as being:
- lsst.apps: each application (e.g. sextractor, wcs) will be a
separate installable package
- lsst.mw: middleware code, such as logging, policy, queue and
fitsio, will be combined into one installable package
* Command-line applications (including scripts) need some naming
convention to avoid name collisions. Ray suggests 3-letter prefixes
-- nice and short, but somebody has to pick them. Meanwhile, for the
examples below I just prepend the package name. (Alternatives include
doing both, or only prepending the last two components of the package
name). We also need to decide whether to use the language suffix
(e.g. ".py"). I personally prefer not to, but either standard is
better than none.
* The source code is arranged as follows. Each installable python
package is contained in its own directory, such that:
- It lives in the appropriate place in the svn hierarchy.
- Its name is py<package name> (e.g. "pymw")
- It includes all python code, documentation, an installer (setup.py
file), any python-specific C/C++ code (e.g. code required to
interface to existing libraries), swig wrappers, command-line
scripts, etc. -- organized using the same directory structure as the
rest of the svn repository.
Sample svn directory layout for python code:
applications/
pysextractor/ (source code for lsst.apps.sextractor)
...
pymw/ (source code for lsst.mw)
setup.py
docs/ (all documentation goes here; flat unless hierarchy needed)
python/
daf/ (data access framework)
__init__.py (probably empty)
fitsio/ (fitsio package)
dbingest/ (dbingest package)
...
pipe/
__init__.py (probably empty)
log.py
queue.py
Policy.py
...(other modules and packages)
scripts/
dbingestinit
dbingestworker
queueget
queueput
...
pylsst/ (files to install basic empty lsst base)
setup.py
python
lsst/
__init__.py (empty)
apps/
__init__.py (empty)
mw
__init__.py (empty)
(Note: pylsst should be one directory higher, but there is no higher
directory in the svn repository)
* Installers:
- Eventually we'll want a centralized installer for everything.
People are working on code tools for this. Meanwhile it's more
practical to install each package separately.
* Responsibilities:
- Developers are responsible for __init__.py files inside their own
packages, as always. Changes should not affect higher level
__init__.py files.
- Developers are responsible for updating the installer (setup.py
file) for their code as necessary. However, the pymw setup.py file
will not need any changes unless somebody includes code that uses
C/C++ python extensions (assuming we feel we can safely treat all
files inside scripts/ as command-line scripts).
- I am happy to reorganize the svn repository and write the higher
level __init__.py files and the lsst.mw setup.py file once we agree
on the standard.
-- Russell
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
