dbingest 1.2.1 has been checked into svn as revision 562.
The main change is that elapsed wall-clock time is now reported for
various operations. Specifically:
*** The time to ingest a file (loaded=filepath, elapsedSec=...).
For example (on my personal computer with a fake logger):
5 dbingestworker; loaded='/tmp/dbitest/s2/data.txt'; scanNum=5;
elapsedSec=0.025828123092651367
elapsedSec includes the time dbingest spends ingesting the file. This
includes: telling mysqld to:
- clear an in-memory temporary table
- ingest the data into that temporary table
- copy that to a real table
- commit the change
*** The time for one pass through the "scan and ingest" cycle
This includes the time to:
- Scan all slice directories (except those that are known to be done)
for new data files.
- Ingest the new data files that it finds -- at most one new data
file per slice.
It does NOT include the time that dbingestworker sleeps if it finds
no new new data files in any slice directory (but you do get a
message telling you that dbingest is about to sleep).
For example on my work computer and using a fake logger, if one file found:
5 dbingestworker; nSliceDirsScanned=2;
elapsedSec=0.028389930725097656; nFilesFound=1; scanNum=5;
nFilesFailed=0; nFilesIngested=1
and if no files found (note "sleeping=1", indicating it is about to
sleep for 1 second):
5 dbingestworker; nSliceDirsScanned=2;
elapsedSec=0.00045108795166015625; nFilesFound=0; scanNum=6;
sleeping=1; nFilesFailed=0; nFilesIngested=0
Note: many messages now include scanNum; this is incremented each
time dbingest starts scanning the slice directories again. This
should make it easier to compute the time spent scanning by
subtracting the time spent ingesting from the time spent scanning and
ingesting.
-- Russell
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
