All, I got NLV working a little better under GNOME (Ray, I think you mentioned problems running under FVWM - it seems to like Fluxbox, but today I'm on someone else's machine today, and I see what you meant about having some issues with it).
It looks like large ConsumeMemFlop calls *are* indeed the restrictively slow factor, even though we're getting some pretty good GFLOPS ratings. It's particularly the very large calls in stages 3 and 4 that are restricting us; though we *are* getting two ConsumeMemFlop events happening concurrently on each host (at least on the hosts I've checked). -Jon -----Original Message----- From: [EMAIL PROTECTED] on behalf of Jonathan Myers Sent: Thu 10/26/2006 11:36 AM To: LSST Data Management Subject: RE: [LSST-data] DC1 status [DataChallenge] All, I ran some of my Python scripts on the logs, and I'm a little surprised to report that the "dry run" (without consumers) ran about 20 sec per image and the full run was 150 sec per image, bringing the total to ~ 130 sec per image. The main culprits are stage 3 and 4 (of 0 through 5 inclusive), which (when examined in NLV) look to be about 60 seconds each. Unfortunately, the policy directory is not publically readable, so I'm not positive what is causing the holdup. I checked the GFLOP fields with [EMAIL PROTECTED] NCSA]$ nlgrep "STAGE: 3" lsst.central.log | grep GFLOP s GFLOP: 0.165398 s GFLOP: 0.164503 s GFLOP: 0.164246 s GFLOP: 0.164471 s GFLOP: 0.161589 ... But they're all around .14 to .165ish. I'm trying to check file IO times - it looks like we were writing to GPFS, which should be a good bit slower than RAMFS, which I may have been using - but I need to fix a bug in my getIOTimes script to establish just how much time is being spent in there. I'll present more information when I get it. -Jon PS: Ray, sorry if you get this twice. I had to do this in Outlook webmail. -----Original Message----- From: [EMAIL PROTECTED] on behalf of Ray Plante Sent: Wed 10/25/2006 3:14 PM To: LSST Data Management Subject: [LSST-data] DC1 status [DataChallenge] Hi Tim and Jeff, The long run of the pipeline we started yesterday ran for about 8 hours at both SDSC and NCSA. The dbingest service died fairly early from the too-many-open-files bug; otherwise, the pipeline seems to have run fine. I have not finished the timing analysis, yet. The results can be found on the NCSA teragrid machine in /usr/projects/LSST/stage3/Final/LsstDC1Final3. Steve and I dived into the logger bug and were able to trace it to a bug in the nlog module. We figured out and tested a fix, and now the dbingest service is running to ingest the remaining files. As I mentioned yesterday, I increased the size of the files by a factor of 100 to see if this improves the overall B/s ingest rate. We also did a 1 hour run today of the pipeline at NCSA with the consumer dials turned to zero. The results can be found on the NCSA teragrid machine in /usr/projects/LSST/stage3/Final/LsstDC1Final4. cheers, Ray _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
<<winmail.dat>>
_______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
