On Jan 16, 2016, at 4:00 PM, Bhalla, Amneet Pal S <[email protected]<mailto:[email protected]>> wrote:
--Amneet Bhalla On Jan 16, 2016, at 10:21 AM, Barry Smith <[email protected]<mailto:[email protected]>> wrote: On Jan 16, 2016, at 7:12 AM, Griffith, Boyce Eugene <[email protected]<mailto:[email protected]>> wrote: On Jan 16, 2016, at 12:34 AM, Bhalla, Amneet Pal S <[email protected]<mailto:[email protected]>> wrote: On Jan 15, 2016, at 5:40 PM, Matthew Knepley <[email protected]<mailto:[email protected]>> wrote: I am inclined to try Barry's experiment first, since this may have bugs that we have not yet discovered. Ok, I tried Barry’s suggestion. The runtime for PetscOptionsFindPair_Private() fell from 14% to mere 1.6%. If I am getting it right, it’s the petsc options in the KSPSolve() that is sucking up nontrivial amount of time (14 - 1.6) and not KSPSetFromOptions() itself (1.6%). Barry / Matt / Jed, if we were using KSPReset here and reusing KSPs, would that also bypass these calls to PetscOptionsXXX? No that is a different issue. In the short term I recommend when running optimized/production you work with a PETSc with those Options checking in KSPSolve commented out, you don't use them anyways*. Since you are using ASM with many subdomains there are many "fast" calls to KSPSolve which is why for your particular case the the PetscOptionsFindPair_Private takes so much time. Now that you have eliminated this issue I would be very interested in seeing the HPCToolKit or Instruments profiling of the code to see hot spots in the PETSc solver configuration you are using. Thanks Barry --- the best way and the least back and forth way would be if I can send you the files (maybe off-list) that you can view in HPCViewer, which is a light weight java script app. You can view which the calling context (which petsc function calls which internal petsc routine) in a cascade form. If I send you an excel sheet, it would be in a flat view and not that useful for serious profiling. Amneet, can you just run with OS X Instruments, which Barry already knows how to use (right Barry?)? :-) Thanks, -- Boyce Let me know if you would like to try that. Barry * Eventually we'll switch to a KSPPreSolveMonitorSet() and KSPPostSolveMonitorSet() model to eliminate this overhead but still have the functionality. Thanks, -- Boyce
