Re: [OMPI users] Advices for parameter tuning for CUDA-aware MPI

2014-05-27 Thread Rolf vandeVaart
>-Original Message- >From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Maxime >Boissonneault >Sent: Tuesday, May 27, 2014 4:07 PM >To: Open MPI Users >Subject: Re: [OMPI users] Advices for parameter tuning for CUDA-aware MPI > >Answers inline to

Re: [OMPI users] Advices for parameter tuning for CUDA-aware MPI

2014-05-27 Thread Maxime Boissonneault
Answers inline too. 2) Is the absence of btl_openib_have_driver_gdr an indicator of something missing ? Yes, that means that somehow the GPU Direct RDMA is not installed correctly. All that check does is make sure that the file /sys/kernel/mm/memory_peers/nv_mem/version exists. Does that

Re: [OMPI users] Advices for parameter tuning for CUDA-aware MPI

2014-05-27 Thread Rolf vandeVaart
Answers inline... >-Original Message- >From: users [mailto:users-boun...@open-mpi.org] On Behalf Of Maxime >Boissonneault >Sent: Friday, May 23, 2014 4:31 PM >To: Open MPI Users >Subject: [OMPI users] Advices for parameter tuning for CUDA-aware MPI > >Hi, >I am

[OMPI users] Advices for parameter tuning for CUDA-aware MPI

2014-05-23 Thread Maxime Boissonneault
Hi, I am currently configuring a GPU cluster. The cluster has 8 K20 GPUs per node on two sockets, 4 PCIe bus (2 K20 per bus, 4 K20 per socket), with a single QDR InfiniBand card on each node. We have the latest NVidia drivers and Cuda 6.0. I am wondering if someone could tell me if all the