Following your advices and those in the FAQ pages, I have added the file $(HOME)/.openmpi/mca-params.conf with :
btl_mvapi_flags=6 mpi_leave_pinned=1 pml_ob1_leave_pinned_pipeline=1 mpool_base_use_mem_hooks=1The parameter btl_mvapi_eager_limit gives the best results, when set to 8 K or 16 K.
The pingpong test result is now : LOOPS: 1000 BYTES: 4096 SECONDS: 0.085643 MBytes/sec: 95.652825 LOOPS: 1000 BYTES: 8192 SECONDS: 0.050893 MBytes/sec: 321.931400 LOOPS: 1000 BYTES: 16384 SECONDS: 0.106791 MBytes/sec: 306.842281 LOOPS: 1000 BYTES: 32768 SECONDS: 0.154873 MBytes/sec: 423.159259 LOOPS: 1000 BYTES: 65536 SECONDS: 0.250849 MBytes/sec: 522.513526 LOOPS: 1000 BYTES: 131072 SECONDS: 0.443162 MBytes/sec: 591.530910 LOOPS: 1000 BYTES: 262144 SECONDS: 0.827640 MBytes/sec: 633.473448 LOOPS: 1000 BYTES: 524288 SECONDS: 1.596701 MBytes/sec: 656.714101 LOOPS: 1000 BYTES: 1048576 SECONDS: 3.134974 MBytes/sec: 668.953554 LOOPS: 1000 BYTES: 2097152 SECONDS: 6.210786 MBytes/sec: 675.325785 LOOPS: 1000 BYTES: 4194304 SECONDS: 12.384103 MBytes/sec: 677.369053 LOOPS: 1000 BYTES: 8388608 SECONDS: 27.377714 MBytes/sec: 612.805580 which is exactly what we can get also with mvapich on the same network.Since we do NOT have a PCI-X hardware, I believe this is the maximum we can get from the
hardware. Thanks a lot for your explanations for this tunning of OpenMPI Best Regards, Jean George Bosilca wrote:
On Thu, 16 Mar 2006, Jean Latour wrote:My questions are : a) Is OpenMPI doing in this case TCP/IP over IB ? (I guess so)If the path to the mvapi library is correct then Open MPI will use mvapi not TCP over IB. There is a simple way to check. "ompi_info --param btl mvapi" will print all the parameters attached to the mvapi driver. If there is no mvapi in the output, then mvapi was not correctly detected. But I don't think it's the case, because if I remember well we have a protection at configure time. If you specify one of the drivers and we're not able to correctly use the libraries, we will stop the configure.b) Is it possible to improve significantly these values by changing the defaults ?By default we are using a very conservative approach. We never leave the memory pinned down, and that decrease the performance for a ping-pong. There are pro and cons for that, too long to be explained here, but in general we're seeing better performance for real-life applications with our default approach, and that's our main goal.Now, if you want to get better performance for the ping-pong test please read the FAQ at http://www.open-mpi.org/faq/?category=infiniband.These are the 3 flags that affect the mvapi performance for the ping-pong case (add them in $(HOME)/.openmpi/mca-params.conf):btl_mvapi_flags=6 mpi_leave_pinned=1 pml_ob1_leave_pinned_pipeline=1I have used several mca btl parameters but without improving the maximum bandwith.For example : --mca btl mvapi --mca btl_mvapi_max_send_size 8388608It is difficult to improve the maximum bandwidth without the leave_pinned activated. But you can improve the bandwidth for medium size messages. Play with btl_mvapi_eager_limit to set the limit between short and rendez-vous protocol. "ompi_info --param btl mvapi" will give you a full list of parameters as well as their description.c) Is it possible that other IB hardware implementations have better performances with OpenMPI ?The maximum bandwidth depend on several factors. One of the most importants is the maximum bandwidth on your node bus. To reach 800 and more MB/s you definitively need a PCI-X 16 ...d) Is it possible to use specific IB drivers for optimal performance ? (should reach almost 800 MB/sec)Once the 3 options are set, you should see an improvement on the bandwidth.Let me know if it does not solve your problems. george. "We must accept finite disappointment, but we must never lose infinite hope." Martin Luther King _______________________________________________ users mailing list us...@open-mpi.org http://www.open-mpi.org/mailman/listinfo.cgi/users
<<attachment: latour.vcf>>
