I have used 9 processors. However, I didn't pay attention to the size of the blocks in each node. I will double check. Thanks a lot.
On Tue, Jun 29, 2010 at 9:57 PM, Barry Smith <bsmith at mcs.anl.gov> wrote: > > ?Until the MPI implementors fix this problem the only way you can work with > such large matrices is to use more processors. Make sure that no process has > more than 250,000,000 entries in the matrix. So for your 36k by 36k matrix > you need to use at least 5 processors, maybe six just try them. > > ? Barry > > > On Jun 29, 2010, at 9:48 PM, Yujie wrote: > >> Thank you very much! I am writing it in parallel mode. I am using >> MPICH not MPICH2. I don't know the mechanism in MatView. If I write in >> paralle, there is no communication between nodes? >> >> Regards, >> Yujie >> >> On Tue, Jun 29, 2010 at 9:36 PM, Jed Brown <jed at 59a2.org> wrote: >>> On Tue, Jun 29, 2010 at 19:35, Yujie <recrusader at gmail.com> wrote: >>>> >>>> Dear PETSc developers, >>>> >>>> I want to output an about 36K*36K dense matrix using MatView in binary >>>> format. I use RedHat Enterprise 5 64bits system. However, when the >>>> file size of output matrix reaches about 2.7G, the codes pause and >>>> don't response for a long time (almost 3 hours). Could you help me >>>> figure out what happened? Thanks a lot. >>> >>> Were you writing this in serial or parallel? ?MPICH2 and Open MPI don't >>> properly handle large message sizes fixes require ABI-incompatible changes >>> that they don't want to push out in a minor release. ?I believe the latest >>> versions of both will actually do the send, but MPI_Get_count does not >>> return the correct value, and it probably would not be surprising if some >>> MPI-IO functionality did not work correctly with large messages. ?Tickets >>> that I'm familiar with: >>> https://trac.mcs.anl.gov/projects/mpich2/ticket/1005 >>> https://svn.open-mpi.org/trac/ompi/ticket/2241 >>> I think they should both be fine for MPI-IO as long as each processor sends >>> less than 2 GiB (even though the final output may be much bigger). >>> If this happens again, you could attach a debugger to the running process >>> (gdb -pid XXX) and get a backtrace. ?Note that you can build "optimized" >>> with debugging symbols at a very small runtime penalty. >>> Jed > >
