Hi Zhen, As Freddie and Brian suggest, I would try using round-robin mode (the default), but with your GPUs in compute exclusive mode.
Cheers Peter Dr Peter Vincent MSci ARCS DIC PhD Senior Lecturer and EPSRC Early Career Fellow Department of Aeronautics Imperial College London South Kensington London SW7 2AZ email: [email protected]<mailto:[email protected]> web: www.imperial.ac.uk/aeronautics/research/vincentlab/<http://www.imperial.ac.uk/aeronautics/research/vincentlab/> twitter: @Vincent_Lab<https://twitter.com/#!/Vincent_Lab> On 17 Apr 2015, at 15:56, Brian Vermeire <[email protected]<mailto:[email protected]>> wrote: Hi Zhen, It could be the case that you do not have your cards set to "compute exclusive mode". If you check nvidia-smi -h you should see the following option: -c, --compute-mode= Set MODE for compute applications: 0/DEFAULT, 1/EXCLUSIVE_THREAD, 2/PROHIBITED, 3/EXCLUSIVE_PROCESS You should try running nvidia-smi -c 3 to allow only one process per card. On Fri, Apr 17, 2015 at 3:53 PM, Zhen Zhang <[email protected]<mailto:[email protected]>> wrote: Hi Freddie, Thanks a lot, but you may misunderstand my idea. Assuming I have two CUDA GPUs on a single node, and I want to partition a mesh into two parts and solve two partition on two GPUs simultaneously. I can set the `devid`, but it is a sole number, which targets at only one GPU. But for local-rank, MPI ( I used MVAPICH2) gives all two processes to a single card. Thank you a lot! On Wednesday, April 15, 2015 at 6:41:56 PM UTC+8, Freddie Witherden wrote: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Zhen, On 15/04/2015 06:36, Freddie Witherden wrote: > I am curious about how to use the multiple CUDA GPU cards in a > single node. Every-time I use MPI with CUDA, there is only one GPU > per node could be used. > > I have checked out some information online, and find that, this > problem seems related with the MPI implementation adopted (For me, > I used Intel MPI), as well as how the program itself is written. By default the CUDA backend uses a 'round-robin' strategy to decide which GPU to use. The strategy tries to create a CUDA context on each CUDA capable device in the system until one succeeds. It is intended to be used when the GPUs are in 'compute exclusive' mode. Alternatively, you can set the device-id key in [backend-cuda] to be 'local-rank'. Here PyFR will use the node-local MPI rank to determine which CUDA device to use. Further information on these options can be found in the user guide. Regards, Freddie. -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org<http://gpgtools.org/> iEYEARECAAYFAlUuQHIACgkQ/J9EM/uoqVdxOgCdE/cSLLnkJfKm1RaQmpKZWr7R VNQAoIBtNTVRwOIP/VQNezt/m5BTkeCx =VdLR -----END PGP SIGNATURE----- -- You received this message because you are subscribed to the Google Groups "PyFR Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To post to this group, send email to [email protected]<mailto:[email protected]>. Visit this group at http://groups.google.com/group/pyfrmailinglist. For more options, visit https://groups.google.com/d/optout. -- Dr. Brian Vermeire BESc MESc PhD Postdoctoral Scholar Department of Aeronautics Imperial College London South Kensington London, UK, SW7 2AZ -- You received this message because you are subscribed to the Google Groups "PyFR Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]<mailto:[email protected]>. To post to this group, send email to [email protected]<mailto:[email protected]>. Visit this group at http://groups.google.com/group/pyfrmailinglist. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "PyFR Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send an email to [email protected]. Visit this group at http://groups.google.com/group/pyfrmailinglist. For more options, visit https://groups.google.com/d/optout.
