Hi ananda,
I didn't try to run your program, but this seems logical to me.
The problem with calling MPI_Bcast repeatedly is that you may have an
infinite desynchronization between the sender and the receiver(s).
MPI_Bcast is an unidirectional operation. It does not necessary block
until the receiver(s) gets the message, hence causing a huge number of
messages to be buffered (and in the case of ft-enable-cr, I guess
everything is saved until an operation going the other way is done).
To "solve" this issue, the sync collective component has been created to
perform a barrier every N operations. So, running with -mca coll
basic,sync should make the problem disappear. I don't think it is really a
memory leak, the memory used is needed (in case of fault) and should be
freed at the next operation going the other way (reduce, barrier,
recv/send).
This seems to be the classical problem of MPI_Bcast benchmarks. Real
applications usually don't suffer this kind of problems.
Sylvain
On Tue, 31 Aug 2010, ananda.mu...@wipro.com wrote:
Hi
When I run the attached program with the following arguments, the size of MPI
processes keep increasing alarmingly (I saw that the size grew from 18M to 12G
in under 10 minutes) making me suspect that there is a major memory leak:
mpirun -am ft-enable-cr --mca coll basic -np 2 <name of the executable>
If I run this program without checkpoint control ie; remove "-am ft-enable-cr",
the size of MPI processes stays constant.
Also if I run this program without setting "--mca coll basic", the size of the
MPI processes stays constant.
I set the mca parameter to "--mca coll basic" during my debugging attempts of
the problem related to checkpointing the program that has repetitive MPI_Bcast() calls.
FYI, I am using OpenMPI v1.4.2 with BLCR 0.8.2
Thanks
Ananda
Ananda B Mudar, PMP
Senior Technical Architect
Wipro Technologies
Ph: 972 765 8093
Please do not print this email unless it is absolutely necessary.
The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments.
WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email.
www.wipro.com
