Thanks Dave, Jed and Shi: Also it looks like there is no 64 bit int support now ... you can't use much of that machine w/o 64 bit ints.
This is not urgent. I have a user that runs 128K+ core jobs with GAMG. The only seem to spend about 40% of the time in the solver but it might be fun to play with when ready. Mark On Fri, Nov 8, 2013 at 5:15 PM, Nystrom, William D <[email protected]> wrote: > Mark, > > I have not run gamg on Vulcan. Right now, I have been focusing on > doing some strong scaling studies on a variety of different machines > using the different more advanced capabilities of petsc that are > available in the "next" branch. With threadcomm, I have focused on > using the pthread support and have looked at the scaling on different > machines using 1 to 64 nodes. My testing has uncovered a problem > where the runs do not scale when going from 1 node to 2 nodes. I > have done enough testing with OpenMP to confirm that the problem > occurs with OpenMP as well as with Pthreads. The scaling problem > only exists when going from 1 node to 2 nodes. The pthread support > scales fine from 2 to 64 nodes. I have spent some time trying to > debug the problem and fix it. I did find and fix one issue with VecNorm > that was part of the scaling problem but there is still at least one > problem remaining as far as I can tell. I'm actually doing some runs > on Vulcan and on one of our Cray systems to try and assess the > current status of this problem. At the moment, I've run out of ideas > for how to continue debugging. So I'm thinking I will do a few more > runs and generate some scaling plots and then present the results > on petsc-dev with the goal of seeing if others have ideas for how > to make more progress on debugging the final problem. > > I've also done a lot of scaling studies for the Cuda gpu support and > some with the OpenCL/ViennaCL support. For the latter, I'm waiting > for Karli to finish some upgrades to ViennaCL. I'm not sure where > that work stands at the moment. > > On a different note, running the ex2.c problem with CG and Jacobi > preconditioning on Vulcan with only MPI results in really beautiful > strong scaling - much better than on a dual socket Sandy Bridge > cluster or a Cray XE-6. > > Best regards, > > Dave > > ------------------------------ > *From:* Mark Adams [[email protected]] > *Sent:* Friday, November 08, 2013 2:53 PM > *To:* Nystrom, William D > *Cc:* For users of the development version of PETSc > *Subject:* Re: [petsc-dev] Mira > > Dave, do you have any performance data with openMP? with GAMG would be > good too. > Mark > > > On Fri, Nov 8, 2013 at 4:20 PM, Nystrom, William D <[email protected]> wrote: > >> I've been using the next branch of petsc and have built and run >> src/ksp/ksp/examples/tutorials/ex2.c on Vulcan at LLNL with >> the threadcomm package using pthreads and openmp. >> >> Dave >> >> -- >> Dave Nystrom >> LANL HPC-5 >> Phone: 505-667-7913 >> Email: [email protected] >> Smail: Mail Stop B272 >> Group HPC-5 >> Los Alamos National Laboratory >> Los Alamos, NM 87545 >> >> ------------------------------ >> *From:* [email protected] [[email protected]] on >> behalf of Mark Adams [[email protected]] >> *Sent:* Friday, November 08, 2013 10:24 AM >> *To:* For users of the development version of PETSc >> *Subject:* [petsc-dev] Mira >> >> I have not been able to get checkout PETSc on Mira with: >> >> [adams@miralac1 ~]$ git clone [email protected]:petsc/petsc.git >> Initialized empty Git repository in /gpfs/mira-home/adams/petsc/.git/ >> Enter passphrase for key '/home/adams/.ssh/id_rsa': >> Permission denied (publickey). >> fatal: The remote end hung up unexpectedly >> >> >> But I have been able to get this to work: >> >> git clone -b maint https://bitbucket.org/petsc/petsc petsc >> >> With this I am trying to build an openMP version and I get these errors. >> >> Mark >> >> >
