On Thu, 5 Oct 2006, Ray Plante wrote:
On Thu, 5 Oct 2006, Russell E Owen wrote:
From something Jacek said yesterday (but not from any email I've
seen) it sounds as if you've managed to create 6300 different
database tables, rather than 192 tables containing 32 files each. Is
this the case? If so, something needs to be fixed (either the policy
files or dbingest itself). Creating a new table for each file would
slow things down -- though how much, I would not care to guess.
I'm not sure that we are creating 6300 different database tables, unless
that is built into the dbingest service. We just write the files to the
ingest directory, and the service slurps them up.
No, we created 192 tables; each table obtained 6300 entries in
one of our examples, but I think that number depends on the number
of data files fed into the pipeline.
I could easily modify dbingest so that it processes all new files for
a given slice before checking the next slice. But if each image needs
significant time to be processed (as I assumed when I designed
dbingest) then I doubt this would have *any* effect at all during a
normal run.
Agreed.
It certainly could help if you started dbingest well
after data was being pumped out by the pipeline (such that there was
a collection of data files already waiting to be ingested).
To separate dbingest from the MPI commands in the "pipeline
workflow" it is most convenient to fork off dbingest before
pipelines have even started. The alternative could be to wait until
all MPI daemons exit, in which case it becomes like a
postprocessing step. I am unclear on the motivation for interweaving
them.
1PM central is a fine time for a call.
Greg
It becomes a question of when we need the data in the database. Right
away, I think, particularly, if the correllation with the standard stars
for WCS determination is done in the database.
Should we schedule a quick telecon today, say 1pm PT, to talk through
this? The people we should have on-line at least should be,
Jeff Kantor
Jacek Becla
Russell Owen
Ray Plante
Greg Daues
cheers,
Ray
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
