Thank you very much! I am writing it in parallel mode. I am using MPICH not MPICH2. I don't know the mechanism in MatView. If I write in paralle, there is no communication between nodes?
Regards, Yujie On Tue, Jun 29, 2010 at 9:36 PM, Jed Brown <jed at 59a2.org> wrote: > On Tue, Jun 29, 2010 at 19:35, Yujie <recrusader at gmail.com> wrote: >> >> Dear PETSc developers, >> >> I want to output an about 36K*36K dense matrix using MatView in binary >> format. I use RedHat Enterprise 5 64bits system. However, when the >> file size of output matrix reaches about 2.7G, the codes pause and >> don't response for a long time (almost 3 hours). Could you help me >> figure out what happened? Thanks a lot. > > Were you writing this in serial or parallel? ?MPICH2 and Open MPI don't > properly handle large message sizes fixes require ABI-incompatible changes > that they don't want to push out in a minor release. ?I believe the latest > versions of both will actually do the send, but MPI_Get_count does not > return the correct value, and it probably would not be surprising if some > MPI-IO functionality did not work correctly with large messages. ?Tickets > that I'm familiar with: > https://trac.mcs.anl.gov/projects/mpich2/ticket/1005 > https://svn.open-mpi.org/trac/ompi/ticket/2241 > I think they should both be fine for MPI-IO as long as each processor sends > less than 2 GiB (even though the final output may be much bigger). > If this happens again, you could attach a debugger to the running process > (gdb -pid XXX) and get a backtrace. ?Note that you can build "optimized" > with debugging symbols at a very small runtime penalty. > Jed
