Re: [OMPI users] Configuration problem or network problem?
Open MPI includes VampirTrace to generate tracing info. In addition to Vampir (a commercial product), there are a few free tools that can read VT's Open Tracefile Format (OTF) output -- I'll leave this up to the VT guys to describe. Alternatively, you could also use MPE (http://www.mcs.anl.gov/research/projects/mpi/www/www4/MPE.html ). IIRC, MPE is included in MPICH2 distributions. There's also mpiP (http://mpip.sourceforge.net/ ). Other MPI tracing tools also exist -- you should be able to google around and find them. All of these tools basically interpose themselves on the MPI library and intercept your function calls. Stats are generated, and possibly even timelines which can be shown graphically. Such graphical timelines can be quite enlightening as to what is really happening in your MPI application at run-time. On Jul 9, 2009, at 5:11 AM, Zou, Lin (GE, Research, Consultant) wrote: Hi Jeff, I tried your suggestion, insert MPI_Barrier every few iterations, but it doesn't work, in fact it became even slower. i want to try tracing the communication avtivity, can you give me some more details about how to use mpitrace. Thank you for your attention. regards Lin -Original Message- From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Jeff Squyres Sent: 2009年7月7日 20:42 To: Open MPI Users Subject: Re: [OMPI users] Configuration problem or network problem? You might want to use a tracing library to see where exactly your synchronization issues are occurring. It may depend on the communication pattern between your nodes and the timing between them. Additionally, your network switch(es) performance characteristics may come into effect here: are there retransmissions, timeouts, etc.? It can sometimes be helpful to insert an MPI_BARRIER every few iterations just to keep all processes well-synchronized. It seems counter-intuitive, but sometimes waiting a short time in a barrier can increase overall throughput (rather than waiting progressively longer times in poorly-synchronized blocking communications). On Jul 6, 2009, at 11:33 PM, Zou, Lin (GE, Research, Consultant) wrote: > Thank you for your suggestion, I tried this solution, but it doesn't > work. In fact, the headnode doesn't participate the computing and > communication, it only malloc a large a memory, and when the loop in > every PS3 is over, the headnode gather the data from every PS3. > The strange thing is that sometimes the program can work well, but > when reboot the system, without any change to the program, it can't > work, so I think it should be some mechanism in OpenMPI that can > configure to let the program work well. > > Lin > > From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] > On Behalf Of Doug Reeder > Sent: 2009年7月7日 10:49 > To: Open MPI Users > Subject: Re: [OMPI users] Configuration problem or network problem? > > Lin, > > Try -np 16 and not running on the head node. > > Doug Reeder > On Jul 6, 2009, at 7:08 PM, Zou, Lin (GE, Research, Consultant) wrote: > >> Hi all, >> The system I use is a PS3 cluster, with 16 PS3s and a PowerPC as >> a headnode, they are connected by a high speed switch. >> There are point-to-point communication functions( MPI_Send and >> MPI_Recv ), the data size is about 40KB, and a lot of computings >> which will consume a long time(about 1 sec)in a loop.The co- >> processor in PS3 can take care of the computation, the main processor >> take care of point-to-point communication,so the computing and >> communication can overlap.The communication funtions should return >> much faster than computing function. >> My question is that after some circles, the time consumed by >> communication functions in a PS3 will increase heavily, and the whole >> cluster's sync state will corrupt.When I decrease the computing time, >> this situation just disappeare.I am very confused about this. >> I think there is a mechanism in OpenMPI that cause this case, does >> everyone get this situation before? >> I use "mpirun --mca btl tcp, self -np 17 --hostfile ...", is there >> something i should added? >> Lin >> ___ >> users mailing list >> us...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > ___ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems -- Jeff Squyres Cisco Systems
Re: [OMPI users] Configuration problem or network problem?
You might want to use a tracing library to see where exactly your synchronization issues are occurring. It may depend on the communication pattern between your nodes and the timing between them. Additionally, your network switch(es) performance characteristics may come into effect here: are there retransmissions, timeouts, etc.? It can sometimes be helpful to insert an MPI_BARRIER every few iterations just to keep all processes well-synchronized. It seems counter-intuitive, but sometimes waiting a short time in a barrier can increase overall throughput (rather than waiting progressively longer times in poorly-synchronized blocking communications). On Jul 6, 2009, at 11:33 PM, Zou, Lin (GE, Research, Consultant) wrote: Thank you for your suggestion, I tried this solution, but it doesn't work. In fact, the headnode doesn't participate the computing and communication, it only malloc a large a memory, and when the loop in every PS3 is over, the headnode gather the data from every PS3. The strange thing is that sometimes the program can work well, but when reboot the system, without any change to the program, it can't work, so I think it should be some mechanism in OpenMPI that can configure to let the program work well. Lin From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Doug Reeder Sent: 2009年7月7日 10:49 To: Open MPI Users Subject: Re: [OMPI users] Configuration problem or network problem? Lin, Try -np 16 and not running on the head node. Doug Reeder On Jul 6, 2009, at 7:08 PM, Zou, Lin (GE, Research, Consultant) wrote: Hi all, The system I use is a PS3 cluster, with 16 PS3s and a PowerPC as a headnode, they are connected by a high speed switch. There are point-to-point communication functions( MPI_Send and MPI_Recv ), the data size is about 40KB, and a lot of computings which will consume a long time(about 1 sec)in a loop.The co- processor in PS3 can take care of the computation, the main processor take care of point-to-point communication,so the computing and communication can overlap.The communication funtions should return much faster than computing function. My question is that after some circles, the time consumed by communication functions in a PS3 will increase heavily, and the whole cluster's sync state will corrupt.When I decrease the computing time, this situation just disappeare.I am very confused about this. I think there is a mechanism in OpenMPI that cause this case, does everyone get this situation before? I use "mpirun --mca btl tcp, self -np 17 --hostfile ...", is there something i should added? Lin ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users -- Jeff Squyres Cisco Systems
Re: [OMPI users] Configuration problem or network problem?
Thank you for your suggestion, I tried this solution, but it doesn't work. In fact, the headnode doesn't participate the computing and communication, it only malloc a large a memory, and when the loop in every PS3 is over, the headnode gather the data from every PS3. The strange thing is that sometimes the program can work well, but when reboot the system, without any change to the program, it can't work, so I think it should be some mechanism in OpenMPI that can configure to let the program work well. Lin From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf Of Doug Reeder Sent: 2009年7月7日 10:49 To: Open MPI Users Subject: Re: [OMPI users] Configuration problem or network problem? Lin, Try -np 16 and not running on the head node. Doug Reeder On Jul 6, 2009, at 7:08 PM, Zou, Lin (GE, Research, Consultant) wrote: Hi all, The system I use is a PS3 cluster, with 16 PS3s and a PowerPC as a headnode, they are connected by a high speed switch. There are point-to-point communication functions( MPI_Send and MPI_Recv ), the data size is about 40KB, and a lot of computings which will consume a long time(about 1 sec)in a loop.The co-processor in PS3 can take care of the computation, the main processor take care of point-to-point communication,so the computing and communication can overlap.The communication funtions should return much faster than computing function. My question is that after some circles, the time consumed by communication functions in a PS3 will increase heavily, and the whole cluster's sync state will corrupt.When I decrease the computing time, this situation just disappeare.I am very confused about this. I think there is a mechanism in OpenMPI that cause this case, does everyone get this situation before? I use "mpirun --mca btl tcp, self -np 17 --hostfile ...", is there something i should added? Lin ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
Re: [OMPI users] Configuration problem or network problem?
Lin, Try -np 16 and not running on the head node. Doug Reeder On Jul 6, 2009, at 7:08 PM, Zou, Lin (GE, Research, Consultant) wrote: Hi all, The system I use is a PS3 cluster, with 16 PS3s and a PowerPC as a headnode, they are connected by a high speed switch. There are point-to-point communication functions( MPI_Send and MPI_Recv ), the data size is about 40KB, and a lot of computings which will consume a long time(about 1 sec)in a loop.The co- processor in PS3 can take care of the computation, the main processor take care of point-to-point communication,so the computing and communication can overlap.The communication funtions should return much faster than computing function. My question is that after some circles, the time consumed by communication functions in a PS3 will increase heavily, and the whole cluster's sync state will corrupt.When I decrease the computing time, this situation just disappeare.I am very confused about this. I think there is a mechanism in OpenMPI that cause this case, does everyone get this situation before? I use "mpirun --mca btl tcp, self -np 17 --hostfile ...", is there something i should added? Lin ___ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
