Barry, thanks, this is extremely helpful. I'll have the OLCF folks contact Nathan if they need any further info or have other experiments to try.
On 02/27/2015 11:03 AM, Barry Smith wrote: > > Same text also in the attachment. > > Barry > > David, > > Nathan Collier has kindly run a test on Titan, Satish on Mira and Hopper, > and Victor on Ranger with a basic optimized build of PETSc (all C code) > > Please find below some configure and make timings from the latest PETSc > master. > > The Titan times for both configure and make are unacceptable. For total > build time Titan is 3.5 times slower than Mira and Hopper and at least 10 > times slower than laptops. The "time" results on Titan are disturbing > > configure > real 14m32.169s (since the user + sys time is much less than real time, > what is it waiting on?) > user 1m51.527s > sys 3m40.734s > > make > real 15m56.004s > user 8m8.971s > sys 52m42.734s (why so much?) > > which I read as either the filesystem or the compiler system (location of the > compilers, license server of the compilers, ...) is really badly configured. > > The Hopper configure time with the default > TMPDIR=/scratch/scratchdirs/balay is is unacceptable but if you actually use > the real /tmp it becomes somewhat reasonable. > > Feel free to share this information with local experts, > > > > > I suggest you view the below table in a fixed width font editor like Emacs or > Vi so the columns line up. > > configure time make time Total compilers > filesystem > > Titan 14m32s 15m56s 30m28s Intel 14 > /lustre/atlas1/geo103/proj-shared/ > 41m38s 9m5s 50m43s > /ccs/home/ (no load on login node) > 13m > (no load on a different login node) > > Mira 6m59s 1m49s 8m48s IBM > /gpfs/mira-home/ > > Hopper 23m17 1m45s 25m2s > /global/u2/b/balay/petsc.clone default TMPDIR=/scratch/scratchdirs/balay > 6m17s 1m39s 7m57s > manually set TMPDIR=/tmp > > NSF Ranger UT Austin 5m10s 1m28s 6m38s > default, whatever it is > > Linux laptop 53s 1m13s 2m6s Gnu > compile and compiler local > > Apple laptop 1m14s 54s 2m8s clang > compile and compiler local > > Linux workstation 1m11s 22s 1m33s Gnu > compile and compiler local > 1m37s 29s 2m6s Gnu > compile directory local; compiler directory remote > 3m11s 25s 3m36s Intel 13 > compile directory local; compiler directory remote > > PETSc has about 1000 source files that need compiling > > The configure is essentially sequential, the make extremely parallel. > > During configure the source code is on the listed file system, all .o and > executables are on /tmp > > During the make the source code and all .o are on the listed file system > > >> On Feb 25, 2015, at 11:23 AM, David E. Bernholdt <[email protected]> >> wrote: >> >> At the kick-off meetings, one of the general complaints I heard >> expressed about the facilities was the slow build times compared to >> personal systems. >> >> If you have this complaint and are an OLCF user, and are willing to work >> with us a little to try to understand your experience in more detail, >> please contact me (individually, not reply-all). >> >> This is a facility thing, not an IDEAS thing, so I can't speak for the >> other facilities. But we've recently received some other similar >> comments, and we're trying to dig into what's happening. >> >> Thanks >> -- >> David E. Bernholdt | Email: [email protected] >> Oak Ridge National Laboratory | Phone: +1 865-574-3147 >> http://www.csm.ornl.gov/~bernhold | Fax: +1 865-576-5491 >> _______________________________________________ >> Ideas-team mailing list >> [email protected] >> https://lists.mcs.anl.gov/mailman/listinfo/ideas-team -- David E. Bernholdt | Email: [email protected] Oak Ridge National Laboratory | Phone: +1 865-574-3147 http://www.csm.ornl.gov/~bernhold | Fax: +1 865-576-5491
