Take a look at http://www.mcs.anl.gov/petsc/petsc-as/documentation/faq.html#efficient-assembly
On Jul 13, 2011, at 4:08 PM, Clemens Domanig wrote: > I preallocate a sparse matrix 1107648x1107648 at the beginning. I checked all > indices when filling the matrix with MatSetValues and they are all smaller > than 1107648. > Just to give you some time measurements: > 1 MPI - "hangs" > 2 MPI - "hangs" even after 20 min no change > 3 MPI - 21s for assembly > 4 MPI - 19s for assembly > > will test with -start_in_debugger tomorrow morning > thx > > Am 2011-07-13 22:59, schrieb Barry Smith: >> >> If it really hangs on one process then just run with the option >> -start_in_debugger noxterm and type cont in the debugger; when you think it >> is hanging (after a few minutes I guess) hit control-c and then type where >> to see where it is "hanging". My guess is that it is not hanging but with >> the size dof your preallocation is way off and it is just taking a huge >> amount of time to set the values. Suggest revisit where you determine the >> preallocation. >> >> >> Barry >> >> On Jul 13, 2011, at 3:56 PM, Clemens Domanig wrote: >> >>> I tried with -mat_no_inode - no effect. Thats the output >>> >>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>> [0] MatStashScatterBegin_Private(): No of messages: 1 >>> [0] MatStashScatterBegin_Private(): Mesg_to: 1: size: 704692232 >>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 88086528 entries, uses 13 mallocs. >>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 553824 X 553824; storage space: >>> 24984360 unneeded,19875384 used >>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0 >>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 42 >>> >>> >>> Am 2011-07-13 22:10, schrieb Matthew Knepley: >>>> On Wed, Jul 13, 2011 at 4:08 PM, Clemens Domanig >>>> <clemens.domanig at uibk.ac.at<mailto:clemens.domanig at uibk.ac.at>> >>>> wrote: >>>> >>>> Hi everyone, >>>> >>>> maybe some can offer som debugging-hints for my problem. >>>> >>>> >>>> Its possible that there is a bug in the inode routines. Please try >>>> running with -mat_no_inode >>>> >>>> Thanks, >>>> >>>> Matt >>>> >>>> My FEM-program uses a shell-element that has depending on the >>>> geometry 5 or 6 dof per node. >>>> >>>> The program uses MPI for parallel solving (LU, mumps). >>>> It works fine with all examples that have onyl 5 dof per node and >>>> that have a mixture of 5 and 6 dof per node. >>>> When doing examples that have 6 dof per node this happens: >>>> * when using more than 2 MPI processes everything seems to be fine. >>>> * when using 1 or 2 MPI processes MatAssemblyBegin() never finishes >>>> >>>> This is the last output of -info, -mat_view_info, -vec_view_info >>>> (with 2 MPI processes, matrix size 1107648x1107648) >>>> >>>> [1] MatAssemblyBegin_MPIAIJ(): Stash has 0 entries, uses 0 mallocs. >>>> [0] MatStashScatterBegin_Private()__: No of messages: 1 >>>> [0] MatStashScatterBegin_Private()__: Mesg_to: 1: size: 704692232 >>>> [0] MatAssemblyBegin_MPIAIJ(): Stash has 88086528 entries, uses 13 >>>> mallocs. >>>> [0] MatAssemblyEnd_SeqAIJ(): Matrix size: 553824 X 553824; storage >>>> space: 24984360 unneeded,19875384 used >>>> [0] MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() >>>> is 0 >>>> [0] MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 42 >>>> [0] Mat_CheckInode(): Found 184608 nodes of 553824. Limit used: 5. >>>> Using Inode routines >>>> >>>> Thx for your help - respectfully C. Domanig >>>> >>>> >>>> >>>> >>>> -- >>>> What most experimenters take for granted before they begin their >>>> experiments is infinitely more interesting than any results to which >>>> their experiments lead. >>>> -- Norbert Wiener >>> >> >
