On Thu, 23 Aug 2007, Tom Cortese wrote:
> Hello, > > Sure, I would be happy to share my experiences; I have amassed a decent-sized > set of "when building PETSc on this sort of machine, use configure options > that look like that" data points, and there is no sense in re-inventing the > wheel. > > As for external packages connected with PETSc, the most commonly-requested > ones that I have heard of are HYPRE, MUMPS, and SLEPC (and also SuperLU, but > you mention that is already part of Cray LibSci -- is it possible to access > this SuperLU from within PETSc on Cray, or do users have to access SuperLU > routines directly?). > > I did make an installation of PETSc with BLOCKSOLVE95 for one particular user, > and also an installation of PETSc with parMETIS, SLEPC, complex scalars, > etc... for another specific user, but these appear to be isolated incidents. > > I am a bit hesitant to ask our users if there are any more; what if they > request 87 other things -- then not only would Cray have to figure out how to > make all of them available, but someone (most likely me) would have to figure > out how to install each of these 87 other things on all the OTHER types of HPC > machines that I have to deal with (this sounds like an experience which would > contain a less-than-optimal amount of fun). I suppose I could do an informal > survey, along with some sort of "we can't make any promises, but we're > interested in knowing which PETSc external packages would be useful" caveat... > > I have one more system-administrator-type question (I'm not a system > administrator, but I play one on TV): If PETSc is configured to download > packages such as MUMPS, HYPRE, etc..., can users wishing to access the > packages directly (i.e., use MUMPS without going through the PETSc interface), > still access them? If the answer is "yes", then the rest of this email is > just un-necessary speculation... Yes > > I'm asking this because we have disk-space limitations on some sites, and I > don't want to end up having, say, MUMPS and HYPRE installed separately by > themselves (for users who wish to access them directly), and then configuring > PETSc with "download MUMPS" and "download HYPRE", thereby creating extra > copies of these libraries. > > Of course, I know the intelligent answer would be "install MUMPS", "install > HYPRE", then "configure PETSc to point to these existing installations and > then build PETSc", but this simple-sounding procedure does not always work (I > often get error messages saying something like "You specified > --with-hypre=blahblahblah, but blahblahblah does not work", which is why I use > the "download HYPRE" PETSc configure option). > > -Tom Cortese > PET Computational Environment Onsite > > On Thu, 23 Aug 2007, Adrian Tate wrote: > > > > > Tom > > > > Thanks for your comments - I'm sure your experiences with PETSc on Crays > > will be helpful to us. Perhaps Keita and I could communicate directly with > > you on these issues and to find out more about the requirements of your > > users? > > > > It is our ultimate goal to provide everything that Cray customers need with > > PETSc, but I can't comment on how feasible that is without knowing more > > about the packages that are widely used. For our initial release we were > > thinking of providing HYPRE and ParMETIS, we also provide SuperLU already > > within Cray LibSci. MUMPS is certainly something we should consider. > > > > This is a perfect opportunity to ask PETSc developers that use Cray hardware > > to suggest packages that they'd like to see provided by Cray. > > > > Adrian > > > > -----Original Message----- > > From: Tom Cortese [mailto:tcortese at cs.utk.edu] > > Sent: Thursday, August 23, 2007 10:53 AM > > To: Adrian Tate > > Cc: Barry Smith; petsc-developers at mcs.anl.gov; petsc-dev at mcs.anl.gov; > > Keita > > Teranishi > > Subject: RE: Cray and PETSc > > > > Hello all, > > > > First of all, I am NOT a PETSc developer, but as a member of the DoD HPC > > Modernization Program I DO install PETSc on a wide variety of hardware > > platforms (including XT3, XD1, and even X1 [that was rough; had to > > cross-compile on one machine and transfer files]), and have had numerous > > email exchanges working out all of the little quirks involved in > > installing PETSc on these machines, so the PETSc folks have been kind > > enough to let me watch this development email list. > > > > I hope I'm not over-stepping my bounds by asking this, but I have two > > comments: > > > > 1) Making PETSc available on the Cray machines sounds like a GREAT idea > > > > 2) Several of our users also like to have packages such as HYPRE and > > MUMPS, etc... installed along with PETSc. Would such external packages > > also be installed by Cray, or would users requiring a combination of, say, > > PETSc and MUMPS still have to make their own installation? > > > > Thanks, > > > > -Tom Cortese > > PET Computational Environnment Onsite > > > > On Thu, 23 Aug 2007, Adrian Tate wrote: > > > > > > > > Hi Barry > > > > > > Thanks for your reply. > > > > > > We will expect to provide the new full releases of PETSc as soon as we > > > possibly can. Obviously there is something of a lag because Cray need to > > > test, package and release the library, but we'll try to do so with minimal > > > delay. Obviously any heads-up you can give us in advance will help reduce > > > the lag. With respect to patch versions, it is going to be very difficult > > > for Cray to release every new version, and by looking at the logs it seems > > > that many of the patches would not be relevant to a Cray specific build. > > > We will endeavor to monitor all patch releases and will go with an > > > unscheduled new release of PETSc when we see a relevant patch. One thing > > > that we can do to minimize confusion is to informally provide the most > > > currently patched version to any customer who is experiencing a problem, > > > although we might not go with a full release of the version in question. > > > > > > Regarding Cray modifications, we (Keita Teranishi or myself) will check in > > > periodically with you to show you our improvements and to see which, if > > > any need to be channeled back into the PETSc build. > > > > > > I hope this support structure works OK from your perspective and I look > > > forward to working with the PETSc team. > > > > > > Thanks! > > > Adrian > > > > > > -----Original Message----- > > > From: Barry Smith [mailto:bsmith at mcs.anl.gov] > > > Sent: Monday, August 20, 2007 11:12 AM > > > To: Adrian Tate > > > Cc: petsc-developers at mcs.anl.gov; petsc-dev at mcs.anl.gov > > > Subject: Re: Cray and PETSc > > > > > > > > > Adrian, > > > > > > Thank you for in the inquiry. > > > > > > > > > On Thu, 16 Aug 2007, Adrian Tate wrote: > > > > > > > Hello Barry > > > > > > > > I'm not sure we met in person - I am the lead of the libraries group > > > > at Cray. We decided some time ago that our iterative solver strategy > > > > would be to leverage PETSc, and to hopefully provide some Cray > > > > specific tunings to improve performance of the KSP solvers (largely > > > > through our custom sparse BLAS and some parallel tuning for our > > > > interconnect). I believe that John Lewis has been in communication > > > > with you with respect to the tuning of Sparse BLAS and their > > > > integration with the PETSc build. > > > > > > > > We are however, considering packaging and supplying PETSc along with > > > > the scientific library that we provide (libSci). This allows for > > > > better integration of our internal kernels and also it means that we > > > > are no longer requiring users (including benchmakers and internal > > > > users) to build their own PETSc. I see from your online page that > > > > doing so is acceptable as long as we use a copyright switch during > > > > configure. By applying this switch, do we make our PETSc library > > > > unsupported from your perspective? > > > > > > The GNU copyright code is tiny; it would not effect the usability > > > or support of PETSc > > > > > > > We do not expect to be able to > > > > provide anywhere near the degree of support that your team provide, > > > > and I was hoping to supply a pre-built library whose users could still > > > > seek assistance through your normal support channels - is this > > > > realistic? > > > > > > > Yes. > > > > > > The two isses that concern us with pre-packaged versions of PETSc are > > > 1) keeping up to date on our releases. We generally make two releases > > > a year and much prefer that users use the most recently release. > > > If they are using an older release it means we are less able to help > > > them. > > > 2) keeping up to date on our patches. We may make several bug patches > > > to each release. Users with a pre-packaged version have trouble keeping > > > up with the source code patches we provide. > > > > > > > > > > > > > > > > > > > > > > > Also, I would be interested to know your degree of interest in the > > > > Cray-specific modifications that we make - would you prefer those to > > > > be channeled back into the PETSc library? > > > > > > If they involved directly changing PETSc code, we much prefer to get > > > it channeled back into the master copy of the source code. Makes it > > > much easier to debug user code. If it is auxiliary code, like a faster > > > ddot() etc then it is more appropriate to not try to include it. > > > > > > > Any other comments you have > > > > on the way that Cray can contribute to the PETSc project, I would be > > > > very glad to hear. > > > > > > PETSc has a variety of "tuning factors" that could theoretically be > > > set optimally for a particular machine, these range from simple things > > > like compiler options, to a choice between C and Fortran versions of > > > the same code (what we call them Fortran kernels), to different loop > > > unrollings (in the inline.h file), even something like PetscMemzero() > > > which has five possible forms. Now currently we do not tune these > > > or even have a test harness for selecting a good tuning. One thing > > > you could/would like to do, is determine good choices for these > > > options on your machines. Just as a simple example, on some Linux systems > > > the basic libraries are just compiled with the GNU compiler, hence the > > > system memset() is not particularly effective. A version compiled of PETSc > > > compiled using a "Fortran memset" may be much faster, or Intel provides > > > its own _intel_fast_memset() which is better. I've seen a few percent > > > increase in performance of entire nonlinear PDE solver applications > > > just from using the non-default memset(). [This was specifically on > > > an Itanium system, but is likely on other configurations also]. > > > > > > > > > Barry > > > > > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > > > > Adrian Tate > > > > > > > > --- > > > > > > > > Technical Lead > > > > > > > > Math Software > > > > Cray Inc. > > > > > > > > (206) 349 5868 > > > > > > > > > > > > > > > > > >
