Hello Sylvain Thanks for your explanation.
I tried using -mca coll basic,sync option also and I still see the same issue ie; the process size increases at an alarming rate. As you see, I am not allocating any memory inside my program. Regards Ananda ------------------- Original Message -------------------------------- Subject: Re: [OMPI devel] Possible memory leak From: Sylvain Jeaugey (sylvain.jeaugey_at_[hidden]) List-Post: devel@lists.open-mpi.org Date: 2010-09-01 04:07:36 * Next message: Chris Samuel: "[OMPI devel] OMPI 1.5 twitter notification plugin probably broken by switch to OAUTH" <http://www.open-mpi.org/community/lists/devel/2010/09/8435.php> * Previous message: Larry Baker: "[OMPI devel] Fwd: Fwd: 1.5rc5 has been posted" <http://www.open-mpi.org/community/lists/devel/2010/08/8433.php> * In reply to: ananda.mudar_at_[hidden]: "[OMPI devel] Possible memory leak" <http://www.open-mpi.org/community/lists/devel/2010/08/8431.php> ________________________________ Hi ananda, I didn't try to run your program, but this seems logical to me. The problem with calling MPI_Bcast repeatedly is that you may have an infinite desynchronization between the sender and the receiver(s). MPI_Bcast is an unidirectional operation. It does not necessary block until the receiver(s) gets the message, hence causing a huge number of messages to be buffered (and in the case of ft-enable-cr, I guess everything is saved until an operation going the other way is done). To "solve" this issue, the sync collective component has been created to perform a barrier every N operations. So, running with -mca coll basic,sync should make the problem disappear. I don't think it is really a memory leak, the memory used is needed (in case of fault) and should be freed at the next operation going the other way (reduce, barrier, recv/send). This seems to be the classical problem of MPI_Bcast benchmarks. Real applications usually don't suffer this kind of problems. Sylvain On Tue, 31 Aug 2010, ananda.mudar_at_[hidden] wrote: > Hi > > When I run the attached program with the following arguments, the size of MPI processes keep increasing alarmingly (I saw that the size grew from 18M to 12G in under 10 minutes) making me suspect that there is a major memory leak: > > mpirun -am ft-enable-cr --mca coll basic -np 2 <name of the executable> > > If I run this program without checkpoint control ie; remove "-am ft-enable-cr", the size of MPI processes stays constant. > > Also if I run this program without setting "--mca coll basic", the size of the MPI processes stays constant. > > I set the mca parameter to "--mca coll basic" during my debugging attempts of the problem related to checkpointing the program that has repetitive MPI_Bcast() calls. > > FYI, I am using OpenMPI v1.4.2 with BLCR 0.8.2 > > Thanks > Ananda > > Ananda B Mudar, PMP > Senior Technical Architect > Wipro Technologies > Ph: 972 765 8093 begin_of_the_skype_highlighting 972 765 8093 end_of_the_skype_highlighting > > Please do not print this email unless it is absolutely necessary. > > The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. > > www.wipro.com > Ananda B Mudar, PMP Senior Technical Architect Wipro Technologies Ph: 972 765 8093 ananda.mu...@wipro.com Please do not print this email unless it is absolutely necessary. The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. www.wipro.com
