On Thu, Jan 24, 2013 at 1:12 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> On Thu, 24 Jan 2013, Barry Smith wrote: > > > > > > > Begin forwarded message: > > > > > From: Philip Papadopoulos <philip.papadopoulos at gmail.com> > > > Subject: Re: [petsc-maint #149715] Making a PetSc Roll and Rocks Module > > > Date: January 24, 2013 9:48:36 AM CST > > > To: "Schneider, Barry I." <bschneid at nsf.gov> > > > Cc: Barry Smith <bsmith at mcs.anl.gov> > > > > > > Dear Barry^2, > > > > > > The major observation about the build process is that building the > software so that it can livesw- > > > in a standard OS package is more work than it should be. Here is the > standard "meme". > > > Suppose you have three directories: > > > sw-src > > > tmpoot/sw-install > > > sw-install > > > > > > sw-src: the source directory tree for building the sw package (eg > petsc-3.3-p5) > > > sw-install: the directory where one wants sw installed on the final > system. We use something > > > like > /opt/petsc/compiler/mpi/interconnect/petsc-version/petsc-arch > > > > > > the last directory is an artifact of the way many people build > software for packaging. Instead of > > > installing into sw-install directly, you install into a > tmproot/sw-install. Then you direct the package > > > manager to grab all files in tmproot/sw-install to put in the package. > The package itself will > > > strip off the tmproot leading directory. In other words the package > labels all files as sw-install/. > > > > > > So the build issue that I ran into is that the build directory and/or > the tmproot directory path becomes embedded into a large number of files > (include files, mod files, etc). To get around this, > > > I did a bind mount of (tmproot/sw-install --> /sw-install) and told > petsc to install into /sw-install. I consider that to be a "hack", but > petsc isn't the only software that I've run into that requires this > > > mounting bit of legerdemain. > > > > > > Many software packages will support a dest= or DESTDIR variable for > "make install" that supports the "installation" into a tmproot directory > but leaves no trace of tmproot in any of its configuration files. ( > http://git.rocksclusters.org/cgi-bin/gitweb.cgi?p=core/python/.git;a=blob;f=src/python-2/Makefile;h=ced6cc1c6eb6a4f70dd0f3e1b0ccd1ac1e40f989;hb=HEAD) > shows a Makefile for python that supports this. > > We do support this install with DESTDIR. It might have rough edges - > but its supporsed to work the same way any other package that supports > it. > > One difference though is - since we also suppor the alternate > orngaization [default] for inplace install with PETSC_ARCH - one has > to use this PETSC_ARCH during the prefix build process aswell. [but > then PETSC_ARCH is nolonger used/needed after that] > > i.e > configure > --prefix=/opt/petsc/compiler/mpi/interconnect/petsc-version/arch1 > PETSC_ARCH=build1 > make PETSC_ARCH=build1 all > make install PETSC_ARCH=build1 install DESTDIR=/tmp/dest1 > <package up from DESTDIR, and install as root:> > Now user does: > make PETSC_DIR=/opt/petsc/compiler/mpi/interconnect/petsc-version/arch1 ex1 > > configure [different options] > --prefix=/opt/petsc/compiler/mpi/interconnect/petsc-version/arch2 > PETSC_ARCH=build2 > make PETSC_ARCH=build2 all > make install PETSC_ARCH=build2 install DESTDIR=/tmp/dest2 > <package up from DESTDIR, and install as root:> > Now user does: > make PETSC_DIR=/opt/petsc/compiler/mpi/interconnect/petsc-version/arch2 ex1 > Satish, would you mind putting this little blurb on the installation page in the section about using prefix? I could not find this anywhere in our documentation. Thanks, Matt > etc. > > There could be rough edges to this process as most folks use inplace > install. But if it doesn't work for you we should fix. > > > > Ultimately we create packages of software to simplify a number of > things. Including update of installed software on many machines. > > > > > > The other "issue" is not really an issue, but a request. When petsc is > built it creates a petscvariables (or similarly named) file with lots of > environment variables. It would be terrific if it could also create an > environment modules files with this same information. Users often want > different versions/different configs of the same software and environment > modules is now standard in Redhat/CentOS distributions. > > The stuff petsc creates is in petscvariables for the consumption by > 'make' and the only env variable the user needs [in this case with > prefix install] is PETSC_DIR [as shown above for a prefix install]. > > Since this PETSC_DIR value is a prefix specfied to configure [for eg: > --prefix=/opt/petsc/compiler/mpi/interconnect/petsc-version/arch2] its > value is known only to rocks pakcage builder. So it would know how to > create the correct module [or whatever mechanicsm used to > indentify/use] for a given petsc install. > > Satish > > > > > > > Hope this helps. > > > > > > > > > On Thu, Jan 24, 2013 at 2:44 AM, Schneider, Barry I. <bschneid at nsf.gov> > wrote: > > > Barry, > > > I am sending this to Phil Papadopoulos and he can make much more > precise recommendations. Glad you have thick skins. Being at NSF requires > the same physiology. > > > > > > -----Original Message----- > > > From: Barry Smith [mailto:bsmith at mcs.anl.gov] > > > Sent: Wednesday, January 23, 2013 10:19 PM > > > To: Schneider, Barry I. > > > Subject: Re: [petsc-maint #149715] Making a PetSc Roll and Rocks Module > > > > > > > > > Barry, > > > > > > By far the most common email we get is regarding installation, so > we are always trying to understand how to improve that process. If we can > eliminate just 20 percent of installation problems that would save us a > great deal of work, so yes, any critique/suggestions are greatly > appreciated, and after all these years we have thick skins so can survive > even the harshest suggestions. > > > > > > Barry > > > > > > On Jan 23, 2013, at 6:13 PM, "Schneider, Barry I." <bschneid at nsf.gov> > wrote: > > > > > > > Barry, > > > > First, thanks for getting back to me so quickly. I view this as a > way to make things easier for everyone for a library that is really > invaluable to many of us. Phil P. is a pro at making rolls and modules for > the Rocks distribution of CentOS that is widely used in the NSF > cyberinfrastructure and by others. I am both a program manager for the > XSEDE project and a large user as well so I appreciate the ability to use > the library in its most transparent fashion. After Phil did his thing we > tested it and it worked perfectly on our cluster. Basically, you load the > module and any associated modules such as the Intel compiler and > appropriate MPI module and then you just use it. No screwing around with > building and the rest of it. That's the good news. Of course, you need to > make a roll for every specific combination of compiler and MPI but I > believe what Phil has learned makes that pretty straightforward to do. > While I cannot speak for Phil, I would expect that anythi > ng he learned would be available to you folks if that's what you wanted. > The next step for us will be to incorporate what we did in the Rocks > distribution which will eventually propagate to lots of users. Eventually, > there will be an XSEDE distribution which will be used by an even larger > number of sites. We are talking about many thousands of users. So, I > guess what I am really asking is whether you are interested enough in what > Phil learned to perhaps modify the current PetSc so that it is easier for > users to install or make available the technology for folks to make their > own rolls. Of course, you could decide that is not on your agenda and > that's fine. But if you capitalize on our experience and your great > software that would be wonderful and perhaps offer users alternatives to > the current way of doing things that would make life easier. > > > > > > > > ************************************************** > > > > Dr Barry I. Schneider > > > > Program Director for Cyberinfrastructure Past Chair, APS Division of > > > > Computational Physics Physics Division National Science Foundation > > > > 4201 Wilson Blvd. > > > > Arlington, VA 22230 > > > > Phone:(703) 292-7383 > > > > FAX:(703) 292-9078 > > > > Cell:(703) 589-2528 > > > > ************************************************** > > > > > > > > > > > > -----Original Message----- > > > > From: Barry Smith [mailto:bsmith at mcs.anl.gov] > > > > Sent: Wednesday, January 23, 2013 5:56 PM > > > > To: petsc-maint at mcs.anl.gov; Schneider, Barry I. > > > > Subject: Re: [petsc-maint #149715] Making a PetSc Roll and Rocks > > > > Module > > > > > > > > > > > > Barry, > > > > > > > > Thanks for the feedback. We actually desire to support "system" > installs just as easily as "home directory" installs, though our > documentation emphasizes "home directory" installs since we feel that is > most practical for more users. We'd be interested in more specifics on the > difficulties and any suggestions there may be to make the process > (especially for multiple compilers/mpis) easier. That is what could we have > done differently to make it easier for you? > > > > > > > > Thanks > > > > > > > > Barry > > > > > > > > On Jan 23, 2013, at 1:59 PM, "Schneider, Barry I." <bschneid at nsf.gov> > wrote: > > > > > > > >> I thought I might pass on the following to the folks maintaining > PetSc. Recently we had the occasion to make a Rocks roll and module to be > used on a local cluster here at NSF. Phil Papadopoulos the developer of > Rocks at SDSC took the time to help us to do that not simply because we > wanted it but also because he wanted to be able to know how to do it and > distribute it to a much larger set users of NSF resources and also on out > NSF supported platforms. Here are his comments. Perhaps if you feel there > are things that could be made easier that would be great. It also could be > useful to you directly. > > > >> > > > >> BTW, Petsc is kind of a bear to package -- they really expect you > to build it in your home directory :-). > > > >> I took the time to make the roll pretty flexible in terms of > compiler/mpi/network support to build various versions. > > > >> This was modeled after other Triton rolls so that it is consistent > with other software -- it likely becomes part of the standard SDSC software > stack, so this was a good thing to do all the way around. > > > >> > > > >> I'm about ready to add this to our code repository, it will show up > on git.rocksclusters.org<http://git.rocksclusters.org> tomorrow morning. > > > >> > > > >> > > > >> ************************************************** > > > >> Dr Barry I. Schneider > > > >> Program Director for Cyberinfrastructure Past Chair, APS Division of > > > >> Computational Physics Physics Division National Science Foundation > > > >> 4201 Wilson Blvd. > > > >> Arlington, VA 22230 > > > >> Phone:(703) 292-7383 > > > >> FAX:(703) 292-9078 > > > >> Cell:(703) 589-2528 > > > >> ************************************************** > > > >> > > > >> > > > > > > > > > > > > > > > > > > > -- > > > Philip Papadopoulos, PhD > > > University of California, San Diego > > > 858-822-3628 (Ofc) > > > 619-331-2990 (Fax) > > > > > -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20130124/5ef300f8/attachment-0001.html>
