Google PTL_NO_SPACE and you will find some NERSC presentations on how to go about fixing this. (I have run into these problems years ago but forget the issues)
Also, I would try running with a Jacobi solver to see if that fixes the problem. If so then you might try -pc_type gamg -pc_gamg_agg_nsmooths 1 -pc_gamg_type agg This is a built in AMG solver so perhaps it plays nicer with resources ... Mark On Jul 9, 2012, at 10:57 AM, John Mousel wrote: > I'm running on Kraken and am currently working with 4320 cores. I get the > following error in KSPSolve. > > [2711]: > (/ptmp/ulib/mpt/nightly/5.3/120211/mpich2/src/mpid/cray/src/adi/ptldev.c:2046) > PtlMEInsert failed with error : PTL_NO_SPACE > MHV_exe: > /ptmp/ulib/mpt/nightly/5.3/120211/mpich2/src/mpid/cray/src/adi/ptldev.c:2046: > MPIDI_CRAY_ptldev_desc_pkt: Assertion `0' failed. > forrtl: error (76): Abort trap signal > Image PC Routine Line Source > MHV_exe 00000000014758CB Unknown Unknown Unknown > MHV_exe 000000000182ED43 Unknown Unknown Unknown > MHV_exe 0000000001829460 Unknown Unknown Unknown > MHV_exe 00000000017EDE3E Unknown Unknown Unknown > MHV_exe 00000000017B3FE6 Unknown Unknown Unknown > MHV_exe 00000000017B3738 Unknown Unknown Unknown > MHV_exe 00000000017B2B12 Unknown Unknown Unknown > MHV_exe 00000000017B428F Unknown Unknown Unknown > MHV_exe 000000000177FCE1 Unknown Unknown Unknown > MHV_exe 0000000001590A43 Unknown Unknown Unknown > MHV_exe 00000000014F909B Unknown Unknown Unknown > MHV_exe 00000000014FF53B Unknown Unknown Unknown > MHV_exe 00000000014A4E25 Unknown Unknown Unknown > MHV_exe 0000000001487D57 Unknown Unknown Unknown > MHV_exe 000000000147F726 Unknown Unknown Unknown > MHV_exe 000000000137A8D3 Unknown Unknown Unknown > MHV_exe 0000000000E97BF2 Unknown Unknown Unknown > MHV_exe 000000000098EAF1 Unknown Unknown Unknown > MHV_exe 0000000000989C20 Unknown Unknown Unknown > MHV_exe 000000000097A9C2 Unknown Unknown Unknown > MHV_exe 000000000082FF2D axbsolve_ 539 > PetscObjectsOperations.F90 > > This is somewhere in KSPSolve. Is there an MPICH environment variable that > needs tweaking? I couldn't really find much on this particular error. > The solver is BiCGStab with Hypre as a preconditioner. > > -ksp_type bcgsl -pc_type hypre -pc_hypre_type boomeramg -ksp_monitor > > Thanks, > > John
