All,
I've spoken to Robyn, Russell, and Gregory about this, and I think we
have some consensus.
I would like to propose several changes to the way we handle policy files.
To my understanding, right now policy files aren't really kept in a
standardized place, either in the source tree or the the installed
system. It is thus up to the person writing a module to figure out
their own policy file location and construct Policy objects with a full
path as a parameter. This strikes me as being a slightly disorganized
way of doing things, and it means that either we have to hard-code the
locations of policy files into modules (which has obvious drawbacks) or
we have to pass each module the location of its policy file of as a
command-line parameter or at run-time through a function. This seems
like a dangerous idea as well, because a caller of a module now has to
know the locations of the policy files of all its dependents.
This is particularly irritating when using something like a logging
module - it it highly likely that we'll want the flexibility to read
from different policy files and thus write to different logs for
different sections of the code. But on the other hand, it also makes
logging less convenient if you write a foo.py which instantiates a
barClass from bar.py which calls a function in baz.py and you need to
pass the name of the baz policy file along to all of these components.
My suggested solution is to keep our policy files all in one directory
and keep that directory name in an environment variable, perhaps
POLICYDIR. The Policy class should be constructed with a call to
Policy("name-of-module") rather than
Policy("/path/to/name-of-module.policy"), and at construction time, the
policy class should look at the environment variable and construct
itself with the correct policy file in that directory.
This should give us quite a bit of flexibility; for instance, we can now
have naive modules which don't need to know the location of any policy
files, and we can change our policy file directory at run-time. But we
can also have very intelligent modules - if, say, we want two instances
of module foo running simultaneously, we can simply start them in two
separate subshells with one environment variable (POLICYDIR) changed.
And if we have a program that goes through several stages, e.g. foo and
bar, then we can have foo export POLICYDIR=/path/to/foo-policy/ and bar
export POLICYDIR=/path/to/bar-policy, and all their components will now
use a different set of policies (which, of course, could be created on
the fly).
And of course, environment variables are highly multiplatform, so we
won't have to worry about communication between C/C++ and Python code
for this.
As one added benefit, we'll also have a much less ugly setup.py - right
now, we have 35 packages, and the distutils setup() function asks for
data files to be specified as package_data= { [package-name] :
[data-files], [package2-name] : [package2-datafiles] : ...}, so
importing per-module data files will quickly get ugly.
Also note that we can still implement a Policy constructor which takes a
full path as an argument (for backwards-compatibility).
Any suggestions? My only self-criticism is that this might
overcomplicate things for now (as we aren't yet doing some of the things
described above, but Robyn said we may have multiple running pipelines
in the future) and that the nlog implementation Russell and I discussed
will need to be subtly nudged. Also, I was concerned that it would
become a headache to propagate these variables across multiple machines,
but I have been told that only the pipeline manager is really
"MPI-aware", so it looks like propagating the variable would have to be
done in just one place.
Cheers,
-Jon
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
