Hi Jim, You are delving too deeply into the numbers. They were generated for comparative reasons to give a ball park sense of differences between the various hardware/software options we have available on a small NCSA cluster. Fabrication of the table resulted from a chance remark to Tim about timing differences noticed when debugging the mpi code.
The code is a prototype built from a combination of handcrafted libraries and off-the-shelf tools and with appropriate glue to drive it the way we need. All data passed between the commodity interfaces is via files. The code is not efficient but it does demonstrate the WCS generation algorithm and its suitability as an mpi pipeline process. Robyn On Tue, Feb 14, 2006 at 10:18:45AM -0800, Jim Gray wrote: > Robyn: > Disk deliver data at 50MBps and disk arrays at 1.5 GBps > So the file system should be a non-issue. > But, I am unclear why there is a 4x advantage of MPI over non MPI (the > main import of your measuremnts). > I assume that the MPI implementation uses multiple cpus and the non-mpi > implementation is a single processor. > Spending 213 seconds to process a 400MB file means that you are using > about 1/2 cpu second/MB > If this is a 3Ghz processor and if pixels are 2 24 bytes that is > 40,000 instructions/pixel. > Is that about right? > Jim Gray > Microsoft Research, Suite 1690, 455 Market, SF CA 94105, > tel: 415 778 8222 fax: 425 706 7329 > [EMAIL PROTECTED] http://research.Microsoft.com/~gray > > > > -----Original Message----- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Robyn Allsman > Sent: Tuesday, February 14, 2006 8:05 AM > To: LSST Data Management > Subject: [LSST-data] Prototype WCS pipeline > > Timing Comparisons: IBRIX and RAM-disk usage for serial- and > mpi-processing > in the WCS generation pipeline > ======================================================================== > ==== > Purpose > ------- > The tests were done to give a rough idea of the prototype pipeline > throughput > for various hardware and software configurations. The prototype code is > not > optimized for best performance, so treat the numbers as indicative not > absolute. > > Setup > ----- > Each sample consisted of 10 timed interations of the prototype WCS > pipeline > code processing the same 12-CCD MEF file (size: 393 MB). > > The first sample set used a non-mpi code to illustrate various > permutations > of files resident on either the NCSA IBRIX filesystem or a 752MB > RAM-based > filesystem. > case 1: IBRIX disk for all files and code > > case 2: Source image file and source configuration files on > RAM-based FS, > Transient files and code on IBRIX > > case 3: Transient files and code on RAM-based FS, > Source image file and source configuration files on IBRIX > > case 4: RAM-based FS for all files and code > > The second sample set used an mpi code to illustrate the same filesystem > permutations as above. The mpi code processed all CCDs in parallel. In > this > case, the 12 CCDs were parceled across 5 mpi nodes. > > All input files were positioned in advance of the test. All transient > (output) > files were reamed at the end of each code processing to ensure an > identical > environment for each of the 10 runs. > > > Timings > ------- > The times were averaged - almost. In two cases, the first timings were > 1 minute longer than the subsequent 9. The average of the lower 9 > timimgs > is reported. The search for the cause of the anomalous time-consumer is > on-going. > > The timings did not include the time needed to place the files onto a > specific > media. The testbed systems remained in active production mode during > the > tests, so there might be unexpected user activity impacting the tests. > > Averaged 'elapsed' time is reported; 'user' and 'system' times showed > same > relative scaling. > > Multi-Processing > no-mpi mpi > > case 1: 3:38.72 0:50.7 IBRIX FS for all files and code > > case 2 3:34 0:50.43 Transient files and code on IBRIX > FS > MEF Image and Config files on RAM > FS, > > case 3 3:10.47 0:40.98 Transient files and code on RAM FS, > > MEF Image and Config files on IBRIX > FS > > case 4 3:08 0:40.75 RAM FS for all files and code > > > No surprisingly, the best overall time placed all code, input files and > > output files onto the RAM disk. Mpi was demonstrably faster than > non-mpi. > > The difference in time between the two mixed medium cases is probably > due > to the location where the output files were written. In case 3, the > much > larger number of output files (18) written onto RAM overshadowed the > fewer input files (~3) remaining on the IBRIX fs. > > > > 13 Feb 2006 RAA > -- > Roberta (Robyn) Allsman > Large Synoptic Survey Telescope phone: 520-322-8741 > Data Management FAX: 520-881-2627 > 4703 E. Camp Lowell Dr Suite 253 email: [EMAIL PROTECTED] > Tucson, AZ 85712 USA > -- Roberta (Robyn) Allsman Large Synoptic Survey Telescope phone: 520-322-8741 Data Management FAX: 520-881-2627 4703 E. Camp Lowell Dr Suite 253 email: [EMAIL PROTECTED] Tucson, AZ 85712 USA _______________________________________________ LSST-data mailing list [email protected] http://www.lsstmail.org/mailman/listinfo/lsst-data
