On Oct 26, 2006, at 1:36 PM, Jonathan Myers wrote:
All,
I ran some of my Python scripts on the logs, and I'm a little
surprised to report that the "dry run" (without consumers) ran
about 20 sec per image and the full run was 150 sec per image,
bringing the total to ~ 130 sec per image. The main culprits are
stage 3 and 4 (of 0 through 5 inclusive), which (when examined in
NLV) look to be about 60 seconds each. Unfortunately, the policy
directory is not publically readable, so I'm not positive what is
causing the holdup.
Policy file are there in /usr/projects/LSST/stage3/Final/
LsstDC1Final4/NCSA/policy/ if needed .... sorry for the delay.
Greg
I checked the GFLOP fields with
[EMAIL PROTECTED] NCSA]$ nlgrep "STAGE: 3" lsst.central.log | grep GFLOP
s GFLOP: 0.165398
s GFLOP: 0.164503
s GFLOP: 0.164246
s GFLOP: 0.164471
s GFLOP: 0.161589
...
But they're all around .14 to .165ish. I'm trying to check file IO
times - it looks like we were writing to GPFS, which should be a
good bit slower than RAMFS, which I may have been using - but I
need to fix a bug in my getIOTimes script to establish just how
much time is being spent in there.
I'll present more information when I get it.
-Jon
PS: Ray, sorry if you get this twice. I had to do this in Outlook
webmail.
-----Original Message-----
From: [EMAIL PROTECTED] on behalf of Ray Plante
Sent: Wed 10/25/2006 3:14 PM
To: LSST Data Management
Subject: [LSST-data] DC1 status [DataChallenge]
Hi Tim and Jeff,
The long run of the pipeline we started yesterday ran for about 8
hours at
both SDSC and NCSA. The dbingest service died fairly early from the
too-many-open-files bug; otherwise, the pipeline seems to have run
fine.
I have not finished the timing analysis, yet. The results can be
found on
the NCSA teragrid machine in
/usr/projects/LSST/stage3/Final/LsstDC1Final3.
Steve and I dived into the logger bug and were able to trace it to
a bug
in the nlog module. We figured out and tested a fix, and now the
dbingest
service is running to ingest the remaining files. As I mentioned
yesterday, I increased the size of the files by a factor of 100 to
see if
this improves the overall B/s ingest rate.
We also did a 1 hour run today of the pipeline at NCSA with the
consumer
dials turned to zero. The results can be found on the NCSA teragrid
machine in /usr/projects/LSST/stage3/Final/LsstDC1Final4.
cheers,
Ray
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
<msg-25599-1011.txt>
<msg-25599-1021.txt>
_______________________________________________
LSST-data mailing list
[email protected]
http://www.lsstmail.org/mailman/listinfo/lsst-data
