At 1:05 PM -0500 2006-10-05, Greg Daues wrote:
On Thu, 5 Oct 2006, Ray Plante wrote:
On Thu, 5 Oct 2006, Russell E Owen wrote:
From something Jacek said yesterday (but not from any email I've
seen) it sounds as if you've managed to create 6300 different
database tables, rather than 192 tables containing 32 files each. Is
this the case? If so, something needs to be fixed (either the policy
files or dbingest itself). Creating a new table for each file would
slow things down -- though how much, I would not care to guess.
I'm not sure that we are creating 6300 different database tables, unless
that is built into the dbingest service. We just write the files to the
ingest directory, and the service slurps them up.
No, we created 192 tables; each table obtained 6300 entries in
one of our examples, but I think that number depends on the number
of data files fed into the pipeline.
I could easily modify dbingest so that it processes all new files for
a given slice before checking the next slice. But if each image needs
significant time to be processed (as I assumed when I designed
dbingest) then I doubt this would have *any* effect at all during a
normal run.
Agreed.
It certainly could help if you started dbingest well
after data was being pumped out by the pipeline (such that there was
a collection of data files already waiting to be ingested).
To separate dbingest from the MPI commands in the "pipeline
workflow" it is most convenient to fork off dbingest before
pipelines have even started. The alternative could be to wait until
all MPI daemons exit, in which case it becomes like a
postprocessing step. I am unclear on the motivation for interweaving
them.
I agree. But whether interwoven or as a postprocessing step, dbingest
would be more efficient if it processed all new data files in one
slice directory before looking at the next slice directory. It
doesn't do that now, and I'm not sure I see any reason to change it.
If we switch to using messages to trigger (as per the EA model) then
that part of the existing code is entirely discarded.
-- Russell
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
