Measuring communications is a very tricky process; there's a lot of
factors involved. Check out this FAQ item:
http://www.open-mpi.org/faq/?category=tuning#running-perf-numbers
You might want to use a well-known benchmark program (e.g., NetPIPE,
link checker, etc.) to run pair-wise communication performance
analysis rather than write your own application; it's typically not
as simple as just doing a few sends within a loop.
The issue is that MPI may make different decisions on how to send
messages, including factors such as:
- is this the first time you have sent between these peer pair?
- who are you sending to?
- what is the size of the message?
- are there other messages pending?
- are other messages incoming from different peers while you are
sending?
Your simplistic loop below can cause some "bad" things to happen
(i.e., not give a true/absolute measure of what max performance is
between a pair of peers) by unintentionally stepping on several of
the things that Open MPI does behind the scenes (e.g., we don't make
network connections until the first time a message is sent between a
given peer pair).
But on the flip side, there's a whole school of thought that micro
benchmarks are only useful in a limited sense (because they test
artificial scenarios), and the only thing that *really* matters is
your application's performance. Hence, micro benchmarks are good as
input for guiding tuning issues, but they are not the absolute
measure of how well a given OS/middleware/network are performing.
That being said, a poorly-written application will tend perform
poorly regardless of how well the OS/middleware/network performs.
And so on.
This is an age-old religious debate, and both sides have some good
points. I won't re-hash the entire debate here. :-)
On Jun 4, 2007, at 10:00 AM, Allan, Mark ((UK Filton)) wrote:
Hi,
I'm new to this list and wonder if anyone can help. I'm trying to
measure communication time between parallel processes using
openmpi. As an example I might be running on 4 dual core
processors (8 processes in total). I was hoping that communication
using shared memory (comms between dual cores on the same chip)
would be faster than that over the network. To measure
communication time I'm sending a block of data to each process
(from each process) using a blocking send, and am timing how long
it takes. I repeat this 50 times (for example) and take the
average time. The code is something like:
for(int i=0;i<numProcs;i++)
for(int j=0;j<numProcs;j++)
if(i!=j)
{
// // // i is the sending proc to j, others wait
double time = 0.0;
for(int kk=0; kk<50; kk++)
{
if(i==my_rank)
{
double start = MPI::Wtime();
MPI::COMM_WORLD.Send(&sendData
[0],dataSize,MPI::DOUBLE,j,i);
double end = MPI::Wtime();
time+=(end-start);
}
if(j==my_rank)
{
MPI::COMM_WORLD.Recv(&recvData
[0],dataSize,MPI::DOUBLE,i,i);
}
}
if(i==my_rank)
out << i << " " << j << " " << time/50.0 <<
std::endl;
MPI::COMM_WORLD.Barrier();
}
The problem I am having is that I'm not noticing any appreciable
difference in communication times between shared memory and network
protocols. I expected shared memory to be faster(!?!).
Does anyone have a better way of measuring communication times?
Thanks,
Mark.
********************************************************************
This email and any attachments are confidential to the intended
recipient and may also be privileged. If you are not the intended
recipient please delete it from your system and notify the sender.
You should not copy it or use it for any purpose nor disclose or
distribute its contents to any other person.
********************************************************************
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
