Keywords: DataAccWG
FYI...
-------- Original Message --------
Subject: Re: provenance
Date: Thu, 04 May 2006 15:08:52 -0700
From: Jacek Becla <[EMAIL PROTECTED]>
Organization: Stanford Linear Accelerator Center
To: [EMAIL PROTECTED]
CC: Greg Daues <[EMAIL PROTECTED]>
References: <[EMAIL PROTECTED]>
Joe,
I know people working on other pipelines face problems similar
to yours, it might be useful to come up with one solid approach
for the whole LSST. (Do you agree?)
cvs-based approach is not unreasonable, as long as you are
ok with the extra dependency on cvs. Also, it does not
avoid dealing with persisting provenance, you will still
need to persist some provenance: as you said
you need to record the cvs version, you probably
want to know the time pipeline started/stopped etc.
Also, I know some pipeline developers (eg. Robyn Allsman/
Tim Axelrod) are interested in recording provenance more
dynamically: they would like to modify parameters on the
fly while the pipeline is running. Do you need that?
(It you have many dynamic changes, probably database would
do a better job than cvs.)
In the "main" LSST database we will have mechanisms to
configure pipelines (through policies) and persist them.
What is the relation between your DES database and
the "main" LSST database? Would you be interested in working
with us on improving our approach and using it?
BTW, at this point we have some objects in UML,
but no concrete implementation.
(when do you need something working?)
It might be worth dedicating part of one of the DataAccessWG
teleconferences to come back to pipeline provenance issues,
I could invite people involved in provenance of different
pipelines. Would you be interested in attending?
If you don't mind, I'd suggest to move further discussion
to lsst-data mailing list (topic DataAccWG). We have
some quite knowledgeable people there that might help.
Regards,
Jacek
Joe Mohr wrote:
Dear Jacek,
I'm writing to ask whether you might be interested in a discussion of
mechanisms for keeping track of the state of a pipeline when particular
data are reduced. I'm leading the Dark Energy Survey data management
team, and Greg Daues is leading the pipeline development.
We've brainstormed about the problem a bit. Our DES database includes
tracking on an object by object basis of the originating images and the
software version (or instance) used to produce those images. We also
have key parts of our system in a cvs archive (and plan to move the
whole beast into an svn archive). A crazy idea we had this morning is
at the point when properties files have been set and pipelines are
springing into action to simply commit the changes into the svn archive
to save an instance of our system. This apparently generates an
instance number within svn. We could then record that instance number
within the DES database along with versioning information for some
software that we are using from outside our group (e.g. SExtractor). It
would then in principle be possible to track back from any object to
recreate the version of the pipeline (down to the parameters) that was
used to generate the data.
It almost seems too easy... just wanted to reach out for a reaction.
Any comments?
Thanks a bunch,
Joe
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
