Hi, Others software doesn't assist the user as gromacs, they don't use CUDA_VISIBLE without an action by the user (with an option, or a specific environnement variable), in case the use this variable they wrote a bit more information, like what they detect and will be used. So i was convince gromacs works in the same way even though you wrote in your mail. sorry :(
Alexis On 01/12/16 18:28, Szilárd Páll wrote: > Hi, > > Welcome! > >> Thank for you reply. Yes, i have read them and know them. I use them > will success with other software. > > BTW, what did you refer to here? It seems like you were focused on > figuring out the detection issue and might have ignored my other > recommendations on using multi (which incidentally would have solved > the original issue too as well as would do correct job placement for > you) and 2+ ranks per GPU? > Cheers, > -- > Szilárd > > > On Thu, Dec 1, 2016 at 6:12 PM, Alexis Michon <alexis.mic...@ibcp.fr> wrote: >> Hi, >> >> You solve my problem, Thank you. >> >> The variable CUDA_VISIBLE_DEVICES is set up by the scheduler at the >> running time of my job. So after adding commandline "unset >> $CUDA_VISIBLE_DEVICES" in my script, mdrun detect and use both GPUs. >> >> Thank you for your patience. >> >> Cheers, >> Alexis >> >> On 01/12/16 17:53, Szilárd Páll wrote: >>> Hi, >>> >>> Do any other programs, e.g. CUDA SDK samples (like deviceQuery) detect >>> both GPUs? >>> I find it *very-very* unlikely that only mdrun would not be able >>> detect both GPUs. >>> >>> Again, are you sure it's not the use of CUDA_VISIBLE_DEVICES that >>> creates the confusion? Please share your full command line (with env >>> vars an all) and resulting log output? >>> >>> Cheers, >>> -- >>> Szilárd >>> >>> >>> On Thu, Dec 1, 2016 at 5:10 PM, Alexis Michon <alexis.mic...@ibcp.fr> wrote: >>>> Hi, >>>> >>>> Thank for you reply. Yes, i have read them and know them. I use them >>>> will success with other software. >>>> >>>> My first and main problem is mdrun doesn't see the second GPU on the >>>> system. And i try to find the root cause of and the remediation. >>>> >>>> Hardware is a bi xeon with 2 Titan-xp GPU, so my goal is to run 2 >>>> differentes simulations with two mdrun process each with one dedicated GPU. >>>> >>>> Do you have any idea ? >>>> >>>> Cheers >>>> Alexis >>>> >>>> On 01/12/16 16:38, Szilárd Páll wrote: >>>>> Alexis, >>>>> >>>>> What is the problem? Have you read the relevant docs of what >>>>> CUDA_VISIBLE_DEVICES does? >>>>> https://docs.nvidia.com/cuda/cuda-c-programming-guide/#env-vars >>>>> >>>>> >>>>> BTW, you can use, but you *do not need* CUDA_VISIBLE devices to >>>>> control the mapping of GPU(s) to mdrun process, the -gpu_id variable >>>>> (or equivalent env var) alone is enough. >>>>> >>>>> To run multiple processes per node, do consider: >>>>> - using mdrun -multi >>>>> - using 2 or more mdrun runs per GPU >>>>> >>>>> You can find the reason, motivating performance examples, and sample >>>>> commands here (in particular Fig 5 and related section): >>>>> dx.doi.org/10.1002/jcc.24030 >>>>> >>>>> Cheers, >>>>> -- >>>>> Szilárd >>>>> >>>>> >>>>> On Thu, Dec 1, 2016 at 3:38 PM, Alexis Michon <alexis.mic...@ibcp.fr> >>>>> wrote: >>>>>> Hi, >>>>>> >>>>>> thank you for your answer >>>>>> >>>>>> the Cmdline is: >>>>>> gmx mdrun -deffnm nvt >>>>>> >>>>>> I played a bit with option -gpu_id, use $CUDA_VISIBLE_DEVICES and read >>>>>> the page http://www.gromacs.org/GPU_acceleration without any succes. One >>>>>> output is in nvt.log.11, >>>>>> >>>>>> nvidia-smi returs is in nvidia-smi.log >>>>>> >>>>>> Nvidia-smi detect 2 GPUs, but not mdrun. >>>>>> >>>>>> nvidia-smi.log : https://icloud.ibcp.fr/index.php/s/08jqFsZpWNlSUGh >>>>>> nvt.log : https://icloud.ibcp.fr/index.php/s/vC5UxCCAhyw9FuC >>>>>> >>>>>> Alexis >>>>>> >>>>>> >>>>>> >>>>>> On 01/12/16 11:34, Mark Abraham wrote: >>>>>>> Hi, >>>>>>> >>>>>>> On Wed, Nov 30, 2016 at 5:38 PM Alexis Michon <alexis.mic...@ibcp.fr> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> We have build gromacs 2016.1 from source with "DMX_GPU=on" on a bi >>>>>>>> processor bi GPU machine, mdrun detect and run fine on only 1 GPU. How >>>>>>>> could we force mdrun to detect the second GPU ? >>>>>>>> >>>>>>> If it's compatible, powered and supported by your driver, then mdrun >>>>>>> will >>>>>>> find it. Presumably nvidia-smi tool will help you work out what's going >>>>>>> on. >>>>>>> >>>>>>> We would like to run 2 mdrun instances on a machine equiped with 2 GPUs, >>>>>>>> each mdrun will use 1GPU. How could we tell mdrun to use a specific >>>>>>>> GPU ? >>>>>>>> >>>>>>> Guidance is here >>>>>>> http://manual.gromacs.org/documentation/2016.1/user-guide/mdrun-performance.html#examples-for-mdrun-on-one-node >>>>>>> >>>>>>> Mark >>>>>>> >>>>>>> >>>>>>>> Cheers, >>>>>>>> Alexis >>>>>>>> >>>>>>>> -- >>>>>>>> Citation : "It’s not enough to be busy; so are the ants. The question >>>>>>>> is: >>>>>>>> what are we busy about?" - Henry David Thoreau >>>>>>>> Alexis MICHON, responsable informatique >>>>>>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France >>>>>>>> Mail : alexis.mic...@ibcp.fr Tel : 04.72.72.26.03 - 06.27.56.34.80 >>>>>>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/ >>>>>>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34 >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Gromacs Users mailing list >>>>>>>> >>>>>>>> * Please search the archive at >>>>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >>>>>>>> posting! >>>>>>>> >>>>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >>>>>>>> >>>>>>>> * For (un)subscribe requests visit >>>>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >>>>>>>> send a mail to gmx-users-requ...@gromacs.org. >>>>>> -- >>>>>> Citation : "It’s not enough to be busy; so are the ants. The question >>>>>> is: what are we busy about?" - Henry David Thoreau >>>>>> Alexis MICHON, responsable informatique >>>>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France >>>>>> Mail : alexis.mic...@ibcp.fr Tel : 04.72.72.26.03 - 06.27.56.34.80 >>>>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/ >>>>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34 >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Gromacs Users mailing list >>>>>> >>>>>> * Please search the archive at >>>>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before >>>>>> posting! >>>>>> >>>>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >>>>>> >>>>>> * For (un)subscribe requests visit >>>>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or >>>>>> send a mail to gmx-users-requ...@gromacs.org. >>>> -- >>>> Citation : "It’s not enough to be busy; so are the ants. The question is: >>>> what are we busy about?" - Henry David Thoreau >>>> Alexis MICHON, responsable informatique >>>> CNRS IBCP, 7 passage du vercors, 69007 LYON, France >>>> Mail : alexis.mic...@ibcp.fr Tel : 04.72.72.26.03 - 06.27.56.34.80 >>>> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/ >>>> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34 >>>> >>>> >>>> >>>> -- >>>> Gromacs Users mailing list >>>> >>>> * Please search the archive at >>>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! >>>> >>>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >>>> >>>> * For (un)subscribe requests visit >>>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send >>>> a mail to gmx-users-requ...@gromacs.org. >> -- >> Citation : "It’s not enough to be busy; so are the ants. The question is: >> what are we busy about?" - Henry David Thoreau >> Alexis MICHON, responsable informatique >> CNRS IBCP, 7 passage du vercors, 69007 LYON, France >> Mail : alexis.mic...@ibcp.fr Tel : 04.72.72.26.03 - 06.27.56.34.80 >> CNRS IBCP - UMS 5760 - http://www.ibcp.fr/ >> Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34 >> >> >> >> -- >> Gromacs Users mailing list >> >> * Please search the archive at >> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! >> >> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists >> >> * For (un)subscribe requests visit >> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a >> mail to gmx-users-requ...@gromacs.org. -- Citation : "It’s not enough to be busy; so are the ants. The question is: what are we busy about?" - Henry David Thoreau Alexis MICHON, responsable informatique CNRS IBCP, 7 passage du vercors, 69007 LYON, France Mail : alexis.mic...@ibcp.fr Tel : 04.72.72.26.03 - 06.27.56.34.80 CNRS IBCP - UMS 5760 - http://www.ibcp.fr/ Empreinte : C9:45:2D:7C:79:7F:0B:79:CA:C8:0B:68:41:A2:8C:EE:EA:72:82:34
-- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.