Ok, on a whim I tried reconfiguring in home again this morning but from a different login node. There was no load last night (on ext4) and none this morning (on ext5), but now my configure time from /ccs/home/nate is 13m. While not appreciably better than the /lustre time, it is suddenly much faster than the 41m I sent previously.
Strikes me that this kind of experience is what confuses/frustrates users. Nate On Fri, Feb 27, 2015 at 8:20 AM, Nathan Collier <[email protected] > wrote: > Barry, > > No problem. > > No packages / master and on /ccs/home/nate > > [nate@titan] time python no_packages.py > ... > real 41m38.700s > user 1m50.075s > sys 1m30.034s > > [nate@titan] time make all > ... > real 9m4.977s > user 5m56.686s > sys 28m22.310s > > In addition to a much slower configure time, the home areas are not > accessible to batch submissions. So without the reconfigure script, I could > not do a full build from home. > > Nate > > > On Thu, Feb 26, 2015 at 11:19 PM, Barry Smith <[email protected]> wrote: > >> >> Nathan, >> >> Can you please do a configure make on your home directory system? >> >> It could be since Lustre is a parallel filesystem designed for high >> bandwidth IO from a parallel computer it is just bad for compiles etc. >> >> Barry >> >> >> >> https://www.olcf.ornl.gov/support/getting-started/ >> >> home areas >> Home areas/directories are provided on a Network File System (NFS). The >> user home areas are by default accessible only to the owning user. >> Similarly the project home areas are accessible by only member of the >> project. The areas are backed-up but available space is limited. Because >> space is limited, each area has a quota. Users should store small source >> code, scripts, and other similar items in the area. Users should not store >> large job output or input in the area. Job I/O should be performed in the >> system’s temporary work area. >> work areas >> Temporary work directories are provided to each user and project on >> Lustre file systems. Similar to the home areas, by default, the user and >> project areas are accessible to the user and project members. The areas >> provide a large amount of storage, but are not backed-up. In general, job >> I/O performance will be faster in the lustre areas than the NFS mounted >> home areas. The temporary work areas are regularly purged of data that has >> not been recently accessed, because of this all needed data should be >> backed-up to the HPSS. >> >> >> > On Feb 26, 2015, at 10:00 PM, Nathan Collier < >> [email protected]> wrote: >> > >> > > They never told you to do compiles on particular file systems or >> anything? For example did they ever say DON'T compile on the lustre file >> system? >> > >> > I have not received any guidance about where to or not to compile. A >> quick perusing of the user guide doesn't make that clear to me. If it isn't >> already obvious, I am far from a Titan expert :) >> > >> > Do you know someone here at ORNL who I can talk to to make sure I am >> not missing something silly? >> > >> > > Perhaps the 52m 'sys' time could be luster overhead >> > >> > It could be, I have heard a lot of complaints about this from people >> who need to write output or read from files in their applications. >> > >> > Nate >> > >> > >> > On Thu, Feb 26, 2015 at 10:36 PM, Satish Balay <[email protected]> >> wrote: >> > On Thu, 26 Feb 2015, Barry Smith wrote: >> > >> > > >> > > > On Feb 26, 2015, at 9:16 PM, Nathan Collier < >> [email protected]> wrote: >> > > > >> > > > Barry, >> > > > >> > > > > Do you know where /tmp is on the system? Presumably it is fast? >> > > > >> > > > I am not sure about that. There is a /tmp but I am not sure what it >> is or if/how I can use it. >> > > >> > > They never told you to do compiles on particular file systems or >> anything? For example did they ever say DON'T compile on the lustre file >> system? >> > >> > Ah - I misunderstood earlier. You'd like to have a petsc clone in /tmp >> > - and do the build there - and compare the difference [with the >> > current luster build]. >> > >> > Perhaps the 52m 'sys' time could be luster overhead - and that woud >> > disappear with a petsc build in /tmp. [and could be much faster]. >> > >> > Satish >> > >> >> >
