Keywords: DataAccWG
Joe,
So do you want to store in database just the cvs tag,
or the actual contents of what the cvs contains?
(I would strongly advice not to do the latter).
I'll raise the issue of collaborating on designing
of the pipeline provenance at one of the next
DataAccWG meetings. I'll keep you posted...
Jacek
Joe Mohr wrote:
Jacek Becla wrote:
Joe,
I know people working on other pipelines face problems similar
to yours, it might be useful to come up with one solid approach
for the whole LSST. (Do you agree?)
Dear Jacek,
Thanks very much for writing back.
Yes... it would be great to come up with some optimal approach.
cvs-based approach is not unreasonable, as long as you are
ok with the extra dependency on cvs. Also, it does not
avoid dealing with persisting provenance, you will still
need to persist some provenance: as you said
you need to record the cvs version, you probably
want to know the time pipeline started/stopped etc.
I may not be understanding what "persisting provenance" means. Our
thinking is that by storing the pipeline svn/cvs instance in the
database we would be able to extract an exact copy of the
pipeline/parameters from the svn archive at any time in the future. We
could then, in principle, rerun the pipeline with the same input data to
reproduce the same output data. Start and stop times would be good,
too. Also, each pipeline execution generates a unique name, which we
use to locate the data within the filesystem of the processing nodes and
then later in the archive.
Also, I know some pipeline developers (eg. Robyn Allsman/
Tim Axelrod) are interested in recording provenance more
dynamically: they would like to modify parameters on the
fly while the pipeline is running. Do you need that?
(It you have many dynamic changes, probably database would
do a better job than cvs.)
Ahhh... this is something that I hadn't imagined. Currently we don't
have pipeline parameters that chance during execution. But it's
reasonable to expect that as our system gets more complicated we might.
Interesting... this may be the fatal flaw for the svn/cvs approach,
unless those dynamically changing parameters are altered in a
predictable/repeatable way given the same input data and initial state
of the pipelines.
In the "main" LSST database we will have mechanisms to
configure pipelines (through policies) and persist them.
What is the relation between your DES database and
the "main" LSST database? Would you be interested in working
with us on improving our approach and using it?
BTW, at this point we have some objects in UML,
but no concrete implementation.
(when do you need something working?)
Currently the only connection between the LSST dm project and the DES dm
project is in people like Greg Daues, who are paid 50% by LSST and 50%
by DES. Also, Ray Plante helped the DES dm project enormously in the
early design phase. The DES database is a developing beast, and because
Ray's group hasn't been involved in the LSST db effort we've been
working with altogether different people at NCSA (Dora Cai and Tanweer
Alam). I'd be happy to collaborate, but the timescales of the two
projects are sufficiently different that there would be challenges. We
hope (still not approved within DOE) to go into operations in fall 2009,
compared to the 2013 target date for LSST. There has been some positive
feedback between our efforts and Ray Plante's LSST middleware group...
basically we've used the middleware he is interested in exploring for
LSST to build a crude, working dm system... this is a sort of proof of
concept exercise from the LSST perspective. In principle this type of
prototyping relationship might also be helpful in the database
components of the project.
Our approach has been to develop a system for quickly for testing and
even discovery. We then use the problems we've discovered to refine our
development plan and cycle through again.
It might be worth dedicating part of one of the DataAccessWG
teleconferences to come back to pipeline provenance issues,
I could invite people involved in provenance of different
pipelines. Would you be interested in attending?
Sure... but again, our ideas are not very advanced. We've got some
structures built into our current database that will support provenance
at some level but have not developed any kind of mature thinking on the
subject.
If you don't mind, I'd suggest to move further discussion
to lsst-data mailing list (topic DataAccWG). We have
some quite knowledgeable people there that might help.
That's no problem, but cc me if you don't mind, because I'm not a member
of the lsst-data mailing list.
Again, thanks for the thoughtful response. I will think a bit more
about dynamically changing pipeline structures/parameters...
Best wishes,
Joe
Regards,
Jacek
Joe Mohr wrote:
Dear Jacek,
I'm writing to ask whether you might be interested in a discussion of
mechanisms for keeping track of the state of a pipeline when
particular data are reduced. I'm leading the Dark Energy Survey data
management team, and Greg Daues is leading the pipeline development.
We've brainstormed about the problem a bit. Our DES database
includes tracking on an object by object basis of the originating
images and the software version (or instance) used to produce those
images. We also have key parts of our system in a cvs archive (and
plan to move the whole beast into an svn archive). A crazy idea we
had this morning is at the point when properties files have been set
and pipelines are springing into action to simply commit the changes
into the svn archive to save an instance of our system. This
apparently generates an instance number within svn. We could then
record that instance number within the DES database along with
versioning information for some software that we are using from
outside our group (e.g. SExtractor). It would then in principle be
possible to track back from any object to recreate the version of the
pipeline (down to the parameters) that was used to generate the data.
It almost seems too easy... just wanted to reach out for a
reaction. Any comments?
Thanks a bunch,
Joe
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
