Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Stéphane Téletchéa]

OK, perfect, once the GPU is available (I'll kick the student ...) I 'll 
post reports of the results,

to have a better description of what happens.

Thanks for the short code, it will help!

Cheers,

Stéphane

Le 22/04/2016 21:58, Szilárd Páll a écrit :

Actually, here's a small program I hacked up that will help diagnose such
issues; you'll find the instructions and explanation of the expected output
in the header of the file:
https://github.com/pszi1ard/playground/blob/master/check-cuda-drv-runtime.cu

Cheers,

--
Szilárd


--
Lecturer, UFIP, UMR 6286 CNRS, Team Protein Design In Silico
UFR Sciences et Techniques, 2, rue de la Houssinière, Bât. 25, 44322 Nantes 
cedex 03, France
Tél : +33 251 125 636 / Fax : +33 251 125 632
http://www.ufip.univ-nantes.fr/ - http://www.steletch.org

--
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Szilárd Páll]

Actually, here's a small program I hacked up that will help diagnose such
issues; you'll find the instructions and explanation of the expected output
in the header of the file:
https://github.com/pszi1ard/playground/blob/master/check-cuda-drv-runtime.cu

Cheers,

--
Szilárd

On Fri, Apr 22, 2016 at 9:32 PM, Szilárd Páll 
wrote:

> Stéphane,
>
> Don't get me wrong, I don't think anyone was angry at your post - I
> certainly was not. On the contrary, describing this (rather annoying)
> problem and suggesting a solution is of good use for the GROMACS community
> -- and CUDA application user community in general.
>
> However, it is important to make the correct assessments and draw the
> correct conclusions. Therefore, in order to clarify things, I simply meant
> to point out that the behavior is not a GPU *detection* issue, but a
> software dependency issue. I suggested clarifying this assessment because a
> correct diagnosis will allow the readers of your post (both on this list
> and on your blog) to better understand the problem and seek the appropriate
> solution (to avoid conclusions like GROMACS+CUDA on Ubuntu is problematic
> hence one should use Fedora).
>
> If you wish to pinpoint the issue at hand, I suggest the following:
> - install an older CUDA version along 7.5, e.g. 6.5
> - grab a simple sample code that uses the CUDA runtime (e.g. from the SDK)
> - compile it with both nvcc's, older and newer,
> - try running both and you'll likely see that the one compiled with the
> newer compiler may fail while the other one will work.
>
> I hope this helps, let me know if you want to dig further but you get
> stuck.
>
> Cheers,
> --
> Szilárd
>
>
> On Fri, Apr 22, 2016 at 6:40 PM, Téletchéa Stéphane <
> stephane.teletc...@univ-nantes.fr> wrote:
>
>> Le 22/04/2016 14:33, Szilárd Páll a écrit :
>>
>>> Additionally, I suggest amending the post to note that the GDK and NVML
>>> that's picked up from it is optional and it is only useful with
>>> Quadro/Tesla cards.
>>>
>>
>> Dear Szilárd,
>>
>> I hope I'll find time to produce a proper bug report, but using the
>> Ubuntu binaries
>> with the ubuntu cuda runtime (so exact same version) 352.xx (same for
>> cuda and nvidia driver),
>> gromacs failed to detect the GPU. Up-to-now I have not traced down where
>> the problems lies.
>>
>> This was I think already stated in the previous entry (section at the
>> end: To sum up*
>> avoid ubuntu packages and install upstream drivers and dependencies*.)
>>
>> But since it seems not clear enough I have corrected some typos and added
>> the disclaimer
>> as requested. I hope Gromacs developper do not feel "angry", this was not
>> my goal, I am a long
>> term GNU/Linux contributor and keen to echaustive bug reports. This
>> article was more a short recipe
>> to get it working and not a complaint.
>>
>> I hope the clarifications on the web site are now better, and this it may
>> help others
>> in getting a proprely working installation ...
>>
>> Best,
>>
>> Stéphane
>>
>> --
>> Assistant Professor in BioInformatics, UFIP, UMR 6286 CNRS, Team Protein
>> Design In Silico
>> UFR Sciences et Techniques, 2, rue de la Houssinière, Bât. 25, 44322
>> Nantes cedex 03, France
>> Tél : +33 251 125 636 / Fax : +33 251 125 632
>> http://www.ufip.univ-nantes.fr/ - http://www.steletch.org
>>
>>
>> --
>> Gromacs Users mailing list
>>
>> * Please search the archive at
>> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
>> posting!
>>
>> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>>
>> * For (un)subscribe requests visit
>> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
>> send a mail to gmx-users-requ...@gromacs.org.
>>
>
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Szilárd Páll]

Stéphane,

Don't get me wrong, I don't think anyone was angry at your post - I
certainly was not. On the contrary, describing this (rather annoying)
problem and suggesting a solution is of good use for the GROMACS community
-- and CUDA application user community in general.

However, it is important to make the correct assessments and draw the
correct conclusions. Therefore, in order to clarify things, I simply meant
to point out that the behavior is not a GPU *detection* issue, but a
software dependency issue. I suggested clarifying this assessment because a
correct diagnosis will allow the readers of your post (both on this list
and on your blog) to better understand the problem and seek the appropriate
solution (to avoid conclusions like GROMACS+CUDA on Ubuntu is problematic
hence one should use Fedora).

If you wish to pinpoint the issue at hand, I suggest the following:
- install an older CUDA version along 7.5, e.g. 6.5
- grab a simple sample code that uses the CUDA runtime (e.g. from the SDK)
- compile it with both nvcc's, older and newer,
- try running both and you'll likely see that the one compiled with the
newer compiler may fail while the other one will work.

I hope this helps, let me know if you want to dig further but you get stuck.

Cheers,
--
Szilárd


On Fri, Apr 22, 2016 at 6:40 PM, Téletchéa Stéphane <
stephane.teletc...@univ-nantes.fr> wrote:

> Le 22/04/2016 14:33, Szilárd Páll a écrit :
>
>> Additionally, I suggest amending the post to note that the GDK and NVML
>> that's picked up from it is optional and it is only useful with
>> Quadro/Tesla cards.
>>
>
> Dear Szilárd,
>
> I hope I'll find time to produce a proper bug report, but using the Ubuntu
> binaries
> with the ubuntu cuda runtime (so exact same version) 352.xx (same for cuda
> and nvidia driver),
> gromacs failed to detect the GPU. Up-to-now I have not traced down where
> the problems lies.
>
> This was I think already stated in the previous entry (section at the end:
> To sum up*
> avoid ubuntu packages and install upstream drivers and dependencies*.)
>
> But since it seems not clear enough I have corrected some typos and added
> the disclaimer
> as requested. I hope Gromacs developper do not feel "angry", this was not
> my goal, I am a long
> term GNU/Linux contributor and keen to echaustive bug reports. This
> article was more a short recipe
> to get it working and not a complaint.
>
> I hope the clarifications on the web site are now better, and this it may
> help others
> in getting a proprely working installation ...
>
> Best,
>
> Stéphane
>
> --
> Assistant Professor in BioInformatics, UFIP, UMR 6286 CNRS, Team Protein
> Design In Silico
> UFR Sciences et Techniques, 2, rue de la Houssinière, Bât. 25, 44322
> Nantes cedex 03, France
> Tél : +33 251 125 636 / Fax : +33 251 125 632
> http://www.ufip.univ-nantes.fr/ - http://www.steletch.org
>
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Téletchéa Stéphane]

Le 22/04/2016 14:33, Szilárd Páll a écrit :

Additionally, I suggest amending the post to note that the GDK and NVML
that's picked up from it is optional and it is only useful with
Quadro/Tesla cards.


Dear Szilárd,

I hope I'll find time to produce a proper bug report, but using the 
Ubuntu binaries
with the ubuntu cuda runtime (so exact same version) 352.xx (same for 
cuda and nvidia driver),
gromacs failed to detect the GPU. Up-to-now I have not traced down where 
the problems lies.


This was I think already stated in the previous entry (section at the 
end: To sum up*

avoid ubuntu packages and install upstream drivers and dependencies*.)

But since it seems not clear enough I have corrected some typos and 
added the disclaimer
as requested. I hope Gromacs developper do not feel "angry", this was 
not my goal, I am a long
term GNU/Linux contributor and keen to echaustive bug reports. This 
article was more a short recipe

to get it working and not a complaint.

I hope the clarifications on the web site are now better, and this it 
may help others

in getting a proprely working installation ...

Best,

Stéphane

--
Assistant Professor in BioInformatics, UFIP, UMR 6286 CNRS, Team Protein Design 
In Silico
UFR Sciences et Techniques, 2, rue de la Houssinière, Bât. 25, 44322 Nantes 
cedex 03, France
Tél : +33 251 125 636 / Fax : +33 251 125 632
http://www.ufip.univ-nantes.fr/ - http://www.steletch.org

--
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Szilárd Páll]

On Thu, Apr 21, 2016 at 11:39 PM, Téletchéa Stéphane <
stephane.teletc...@univ-nantes.fr> wrote:

> Le 21/04/2016 03:54, treinz a écrit :
>
>> Hi,
>>
>>
>> Can you also explain why the function calls to cudaDriverGetVersion() and
>> cudaRuntimeGetVersion() both return 0, as in
>>
>>> >>CUDA driver:0.0
 >>CUDA runtime: 0.0

>>> Thanks,
>> Tim
>>
>
> Hi all,
>
> As I already posted on the list, it seems the combination of gromacs +
> cuda is tedious,
>

That's is a incorrect claim. Your problems have absolutely nothing to do
with GROMACS, but are caused by the CUDA driver / runtime mismatch on your
system. Any application that uses the CUDA runtime (i.e. libcudart) will
fail to work if you install an incompatible driver.

For clarity, please correct your blog too.

I had not time to dig it further, but wrote down what worked for me:
>
> http://www.steletch.org/spip.php?article89


Additionally, I suggest amending the post to note that the GDK and NVML
that's picked up from it is optional and it is only useful with
Quadro/Tesla cards.


>
>
> The driver version was very close from the drivers included in the default
> ubuntu
> installation, nvidia-smi was working fine also, but gromacs was not able
> to detect it.
>

Again, for the sake of clarity: this is not a matter of detection. It is an
incompatibility of two external dependencies.


>
> Using the driver from the cuda toolkit solved the problem for me.
> I only sent a mail on the mailing list, if I have time to dig further I'll
> try to provide more
> insight and a proper bug report. Up to now you can check if the driver
> version is the one
> expected from the cuda toolkit (nvidia-smi -a).


I'll ask my NVIDIA contact, but I suspect the only way to know this is to
dig into some header files -- or maybe even that doesn't help.

Cheers,
--
Szilárd



>


> HTH,
>
> Stéphane
>
> --
> Assistant Professor in BioInformatics, UFIP, UMR 6286 CNRS, Team Protein
> Design In Silico
> UFR Sciences et Techniques, 2, rue de la Houssinière, Bât. 25, 44322
> Nantes cedex 03, France
> Tél : +33 251 125 636 / Fax : +33 251 125 632
> http://www.ufip.univ-nantes.fr/  -http://www.steletch.org
>
>
> --
> Gromacs Users mailing list
>
> * Please search the archive at
> http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before
> posting!
>
> * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
> * For (un)subscribe requests visit
> https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or
> send a mail to gmx-users-requ...@gromacs.org.
>
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Téletchéa Stéphane]

Le 22/04/2016 01:50, treinz a écrit :

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL


Unfortunately for any machine installed by now the setup is now as 
described,

and works.

There is one machine were this setup was not updated and YES I have the 
same result as you :


./deviceQuery
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL


On this machine the driver is rather old:

dpkg -l |grep nvidia
ii  nvidia-331 340.96-0ubuntu0.14.04.1 amd64 
Transitional package for nvidia-331
ii  nvidia-340 340.96-0ubuntu0.14.04.1 amd64 
NVIDIA binary driver - version 340.96
ii  nvidia-340-uvm 340.96-0ubuntu0.14.04.1 
amd64 Transitional package for nvidia-340
ii  nvidia-libopencl1-340 
340.96-0ubuntu0.14.04.1 amd64 NVIDIA OpenCL 
Driver and ICD Loader library
ii  nvidia-opencl-icd-340 
340.96-0ubuntu0.14.04.1 amd64 NVIDIA OpenCL ICD
ii  nvidia-prime 0.6.2   
amd64 Tools to enable NVIDIA's Prime
ii  nvidia-settings 331.20-0ubuntu8 
amd64 Tool for configuring the NVIDIA graphics driver


But I wrote my "tutorial" since the latest package available in Ubuntu 
is "nvidia-352 - NVIDIA binary driver - version 352.63",
and apparently cuda expects 352.79, the minor revision in the driver is 
sufficient to have CUDA + GROMACS failing,

and apparently this is from CUDA :-)

You have the answer: use the exact driver of the CUDA runtime and 
nothing else ...


Best,

Stéphane




--
Assistant Professor in BioInformatics, UFIP, UMR 6286 CNRS, Team Protein Design 
In Silico
UFR Sciences et Techniques, 2, rue de la Houssinière, Bât. 25, 44322 Nantes 
cedex 03, France
Tél : +33 251 125 636 / Fax : +33 251 125 632
http://www.ufip.univ-nantes.fr/ - http://www.steletch.org

--
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[treinz]

Hi Stéphane,


Did you see the following or similar output from running 
/path/to/cuda/samples/1_Utilities/deviceQuery/deviceQuery when GROMACS wasn't 
able to detect GPU?


./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
-> CUDA driver version is insufficient for CUDA runtime version
Result = FAIL


In my case this happens.


Thanks,
Tim





At 2016-04-22 05:39:08, "Téletchéa Stéphane" 
 wrote:
>Le 21/04/2016 03:54, treinz a écrit :
>> Hi,
>>
>>
>> Can you also explain why the function calls to cudaDriverGetVersion() and 
>> cudaRuntimeGetVersion() both return 0, as in
 >>CUDA driver:0.0
 >>CUDA runtime: 0.0
>> Thanks,
>> Tim
>
>Hi all,
>
>As I already posted on the list, it seems the combination of gromacs + 
>cuda is tedious,
>I had not time to dig it further, but wrote down what worked for me:
>
>http://www.steletch.org/spip.php?article89
>
>The driver version was very close from the drivers included in the 
>default ubuntu
>installation, nvidia-smi was working fine also, but gromacs was not able 
>to detect it.
>
>Using the driver from the cuda toolkit solved the problem for me.
>I only sent a mail on the mailing list, if I have time to dig further 
>I'll try to provide more
>insight and a proper bug report. Up to now you can check if the driver 
>version is the one
>expected from the cuda toolkit (nvidia-smi -a).
>
>HTH,
>
>Stéphane
>
>-- 
>Assistant Professor in BioInformatics, UFIP, UMR 6286 CNRS, Team Protein 
>Design In Silico
>UFR Sciences et Techniques, 2, rue de la Houssinière, Bât. 25, 44322 Nantes 
>cedex 03, France
>Tél : +33 251 125 636 / Fax : +33 251 125 632
>http://www.ufip.univ-nantes.fr/  -http://www.steletch.org
>
>-- 
>Gromacs Users mailing list
>
>* Please search the archive at 
>http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!
>
>* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists
>
>* For (un)subscribe requests visit
>https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
>mail to gmx-users-requ...@gromacs.org.
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Szilárd Páll]

On Thu, Apr 21, 2016 at 3:54 AM, treinz  wrote:
> Hi,
>
>
> Can you also explain why the function calls to cudaDriverGetVersion() and 
> cudaRuntimeGetVersion() both return 0, as in

Not really, but it is the normal behavior on hosts where the runtime
is not compatible with the driver or there is simply no driver
installed.

Cheers,
--
Szilárd

>>> CUDA driver:0.0
>>> CUDA runtime: 0.0
>
>
> Thanks,
> Tim
> At 2016-04-21 08:49:17, "Szilárd Páll"  wrote:
>>On Thu, Apr 21, 2016 at 12:22 AM, treinz  wrote:
>>> Hi all,
>>>
>>>
>>> I recently built 5.1.2 with GPU support and the config options are:
>>>
>>>
>>> module load cuda/7.5.18
>>> cmake .. -DCMAKE_C_COMPILER=gcc-4.9 \
>>>  -DCMAKE_CXX_COMPILER=g++-4.9 \
>>>  -DGMX_MPI=OFF \
>>>  -DGMX_THREAD_MPI=ON \
>>>  -DGMX_GPU=ON \
>>>  -DCMAKE_PREFIX_PATH=$HOME/local \
>>>  -DCMAKE_INSTALL_PREFIX=$HOME/local/gromacs/grid_frontend \
>>>  -DGMX_BUILD_OWN_FFTW=ON \
>>>  -DGMX_DEFAULT_SUFFIX=OFF \
>>>  -DGMX_BINARY_SUFFIX=_gpu \
>>>  -DGMX_LIBS_SUFFIX=_gpu
>>>
>>>
>>> and the installation was successful. But when I tried running mdrun, it 
>>> wasn't able to detect the GPU:
>>>
>>>
>>> Build OS/arch:  Linux 2.6.32-573.1.1.el6.x86_64 x86_64
>>> Build CPU vendor:   GenuineIntel
>>> Build CPU brand:Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>>> Build CPU family:   6   Model: 63   Stepping: 2
>>> Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt 
>>> lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd 
>>> rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
>>> C compiler: /net/noble/vol1/home/dejunlin/local/bin/mpicc GNU 4.9.3
>>> C compiler flags:-march=core-avx2-Wextra 
>>> -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall 
>>> -Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG 
>>> -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
>>> C++ compiler:   /net/noble/vol1/home/dejunlin/local/bin/mpicxx GNU 4.9.3
>>> C++ compiler flags:  -march=core-avx2-Wextra 
>>> -Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function  
>>> -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
>>> Boost version:  1.59.0 (external)
>>> CUDA compiler:  
>>> /net/gs/vol3/software/modules-sw/cuda/7.5.18/Linux/RHEL6/x86_64/bin/nvcc 
>>> nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2015 NVIDIA 
>>> Corporation;Built on Tue_Aug_11_14:27:32_CDT_2015;Cuda compilation tools, 
>>> release 7.5, V7.5.17
>>> CUDA compiler 
>>> flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_52,code=compute_52;-use_fast_math;;
>>>  
>>> ;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-Wpointer-arith;-Wall;-Wno-unused-function;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
>>> CUDA driver:0.0
>>> CUDA runtime:   0.0
>>>
>>>
>>> NOTE: Error occurred during GPU detection:
>>>   CUDA driver version is insufficient for CUDA runtime version
>>>   Can not use GPU acceleration, will fall back to CPU kernels.
>>>
>>
>>^^^ This is the message you should be focusing on. For some reason,
>>the runtime mdrun was compiled with is incompatible with the driver.
>>This should not be the case if the driver is indeed 352.xx (which is
>>driver API v 7.5) as shown below and the runtime API v 7.5.
>>
>>> The command lines for this run is:
>>>
>>>
>>> module load cuda/7.5.18
>>> source $HOME/local/gromacs/grid_frontend/bin/GMXRC
>>> env
>>> nvidia-smi -a
>>> gmx_gpu mdrun -ntmpi 24 -ntomp 1 -gpu_id 1 -deffnm
>>
>>FYI: That's going to be inefficient, probably the worst possible
>>option in fact. Try 1-6 ranks/GPU.
>>
>>>
>>> I look at the stdout from nvidia-smi, it looks like the nvidia driver was 
>>> installed (there are 4 GPUs but I'm only showing one of them):
>>>
>>>
>>> ==NVSMI LOG==
>>>
>>>
>>> Timestamp   : Wed Apr 20 15:05:52 2016
>>> Driver Version  : 352.39
>>>
>>>
>>> Attached GPUs   : 4
>>> GPU :02:00.0
>>> Product Name: Tesla K40c
>>> Product Brand   : Tesla
>>> Display Mode: Disabled
>>> Display Active  : Disabled
>>> Persistence Mode: Disabled
>>> Accounting Mode : Disabled
>>> Accounting Mode Buffer Size : 1920
>>> Driver Model
>>> Current : N/A
>>> Pending : N/A
>>> Serial Number   : 0321715040048
>>> GPU UUID: 
>>> 

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[treinz]

Hi,


Can you also explain why the function calls to cudaDriverGetVersion() and 
cudaRuntimeGetVersion() both return 0, as in
>> CUDA driver:0.0 
>> CUDA runtime: 0.0 


Thanks,
Tim
At 2016-04-21 08:49:17, "Szilárd Páll"  wrote:
>On Thu, Apr 21, 2016 at 12:22 AM, treinz  wrote:
>> Hi all,
>>
>>
>> I recently built 5.1.2 with GPU support and the config options are:
>>
>>
>> module load cuda/7.5.18
>> cmake .. -DCMAKE_C_COMPILER=gcc-4.9 \
>>  -DCMAKE_CXX_COMPILER=g++-4.9 \
>>  -DGMX_MPI=OFF \
>>  -DGMX_THREAD_MPI=ON \
>>  -DGMX_GPU=ON \
>>  -DCMAKE_PREFIX_PATH=$HOME/local \
>>  -DCMAKE_INSTALL_PREFIX=$HOME/local/gromacs/grid_frontend \
>>  -DGMX_BUILD_OWN_FFTW=ON \
>>  -DGMX_DEFAULT_SUFFIX=OFF \
>>  -DGMX_BINARY_SUFFIX=_gpu \
>>  -DGMX_LIBS_SUFFIX=_gpu
>>
>>
>> and the installation was successful. But when I tried running mdrun, it 
>> wasn't able to detect the GPU:
>>
>>
>> Build OS/arch:  Linux 2.6.32-573.1.1.el6.x86_64 x86_64
>> Build CPU vendor:   GenuineIntel
>> Build CPU brand:Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>> Build CPU family:   6   Model: 63   Stepping: 2
>> Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt 
>> lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd 
>> rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
>> C compiler: /net/noble/vol1/home/dejunlin/local/bin/mpicc GNU 4.9.3
>> C compiler flags:-march=core-avx2-Wextra 
>> -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall 
>> -Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG 
>> -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
>> C++ compiler:   /net/noble/vol1/home/dejunlin/local/bin/mpicxx GNU 4.9.3
>> C++ compiler flags:  -march=core-avx2-Wextra 
>> -Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function  
>> -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
>> Boost version:  1.59.0 (external)
>> CUDA compiler:  
>> /net/gs/vol3/software/modules-sw/cuda/7.5.18/Linux/RHEL6/x86_64/bin/nvcc 
>> nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2015 NVIDIA 
>> Corporation;Built on Tue_Aug_11_14:27:32_CDT_2015;Cuda compilation tools, 
>> release 7.5, V7.5.17
>> CUDA compiler 
>> flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_52,code=compute_52;-use_fast_math;;
>>  
>> ;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-Wpointer-arith;-Wall;-Wno-unused-function;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
>> CUDA driver:0.0
>> CUDA runtime:   0.0
>>
>>
>> NOTE: Error occurred during GPU detection:
>>   CUDA driver version is insufficient for CUDA runtime version
>>   Can not use GPU acceleration, will fall back to CPU kernels.
>>
>
>^^^ This is the message you should be focusing on. For some reason,
>the runtime mdrun was compiled with is incompatible with the driver.
>This should not be the case if the driver is indeed 352.xx (which is
>driver API v 7.5) as shown below and the runtime API v 7.5.
>
>> The command lines for this run is:
>>
>>
>> module load cuda/7.5.18
>> source $HOME/local/gromacs/grid_frontend/bin/GMXRC
>> env
>> nvidia-smi -a
>> gmx_gpu mdrun -ntmpi 24 -ntomp 1 -gpu_id 1 -deffnm
>
>FYI: That's going to be inefficient, probably the worst possible
>option in fact. Try 1-6 ranks/GPU.
>
>>
>> I look at the stdout from nvidia-smi, it looks like the nvidia driver was 
>> installed (there are 4 GPUs but I'm only showing one of them):
>>
>>
>> ==NVSMI LOG==
>>
>>
>> Timestamp   : Wed Apr 20 15:05:52 2016
>> Driver Version  : 352.39
>>
>>
>> Attached GPUs   : 4
>> GPU :02:00.0
>> Product Name: Tesla K40c
>> Product Brand   : Tesla
>> Display Mode: Disabled
>> Display Active  : Disabled
>> Persistence Mode: Disabled
>> Accounting Mode : Disabled
>> Accounting Mode Buffer Size : 1920
>> Driver Model
>> Current : N/A
>> Pending : N/A
>> Serial Number   : 0321715040048
>> GPU UUID: 
>> GPU-647b8474-e09f-7f98-3ac5-4604e02f1c75
>> Minor Number: 0
>> VBIOS Version   : 80.80.3E.00.02
>> MultiGPU Board  : No
>> Board ID: 0x200
>> Inforom Version
>> Image Version   : 2081.0206.01.04
>> OEM Object  : 1.1
>> ECC 

Re: [gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[Szilárd Páll]

On Thu, Apr 21, 2016 at 12:22 AM, treinz  wrote:
> Hi all,
>
>
> I recently built 5.1.2 with GPU support and the config options are:
>
>
> module load cuda/7.5.18
> cmake .. -DCMAKE_C_COMPILER=gcc-4.9 \
>  -DCMAKE_CXX_COMPILER=g++-4.9 \
>  -DGMX_MPI=OFF \
>  -DGMX_THREAD_MPI=ON \
>  -DGMX_GPU=ON \
>  -DCMAKE_PREFIX_PATH=$HOME/local \
>  -DCMAKE_INSTALL_PREFIX=$HOME/local/gromacs/grid_frontend \
>  -DGMX_BUILD_OWN_FFTW=ON \
>  -DGMX_DEFAULT_SUFFIX=OFF \
>  -DGMX_BINARY_SUFFIX=_gpu \
>  -DGMX_LIBS_SUFFIX=_gpu
>
>
> and the installation was successful. But when I tried running mdrun, it 
> wasn't able to detect the GPU:
>
>
> Build OS/arch:  Linux 2.6.32-573.1.1.el6.x86_64 x86_64
> Build CPU vendor:   GenuineIntel
> Build CPU brand:Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
> Build CPU family:   6   Model: 63   Stepping: 2
> Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt 
> lahf_lm mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd 
> rdtscp sse2 sse3 sse4.1 sse4.2 ssse3 tdt x2apic
> C compiler: /net/noble/vol1/home/dejunlin/local/bin/mpicc GNU 4.9.3
> C compiler flags:-march=core-avx2-Wextra 
> -Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall 
> -Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG 
> -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
> C++ compiler:   /net/noble/vol1/home/dejunlin/local/bin/mpicxx GNU 4.9.3
> C++ compiler flags:  -march=core-avx2-Wextra 
> -Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function  
> -O3 -DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
> Boost version:  1.59.0 (external)
> CUDA compiler:  
> /net/gs/vol3/software/modules-sw/cuda/7.5.18/Linux/RHEL6/x86_64/bin/nvcc 
> nvcc: NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2015 NVIDIA 
> Corporation;Built on Tue_Aug_11_14:27:32_CDT_2015;Cuda compilation tools, 
> release 7.5, V7.5.17
> CUDA compiler 
> flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_52,code=compute_52;-use_fast_math;;
>  
> ;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-Wpointer-arith;-Wall;-Wno-unused-function;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
> CUDA driver:0.0
> CUDA runtime:   0.0
>
>
> NOTE: Error occurred during GPU detection:
>   CUDA driver version is insufficient for CUDA runtime version
>   Can not use GPU acceleration, will fall back to CPU kernels.
>

^^^ This is the message you should be focusing on. For some reason,
the runtime mdrun was compiled with is incompatible with the driver.
This should not be the case if the driver is indeed 352.xx (which is
driver API v 7.5) as shown below and the runtime API v 7.5.

> The command lines for this run is:
>
>
> module load cuda/7.5.18
> source $HOME/local/gromacs/grid_frontend/bin/GMXRC
> env
> nvidia-smi -a
> gmx_gpu mdrun -ntmpi 24 -ntomp 1 -gpu_id 1 -deffnm

FYI: That's going to be inefficient, probably the worst possible
option in fact. Try 1-6 ranks/GPU.

>
> I look at the stdout from nvidia-smi, it looks like the nvidia driver was 
> installed (there are 4 GPUs but I'm only showing one of them):
>
>
> ==NVSMI LOG==
>
>
> Timestamp   : Wed Apr 20 15:05:52 2016
> Driver Version  : 352.39
>
>
> Attached GPUs   : 4
> GPU :02:00.0
> Product Name: Tesla K40c
> Product Brand   : Tesla
> Display Mode: Disabled
> Display Active  : Disabled
> Persistence Mode: Disabled
> Accounting Mode : Disabled
> Accounting Mode Buffer Size : 1920
> Driver Model
> Current : N/A
> Pending : N/A
> Serial Number   : 0321715040048
> GPU UUID: GPU-647b8474-e09f-7f98-3ac5-4604e02f1c75
> Minor Number: 0
> VBIOS Version   : 80.80.3E.00.02
> MultiGPU Board  : No
> Board ID: 0x200
> Inforom Version
> Image Version   : 2081.0206.01.04
> OEM Object  : 1.1
> ECC Object  : 3.0
> Power Management Object : N/A
> GPU Operation Mode
>
>
> One catch is that I didn't have the CUDA GPU Deployment Kit or the NVML 
> library installed -- would that matter in detecting GPUs? Another catch is 
> that I was running the job in a SGE cluster. I believe I have requested the 
> GPU resources correctly because I could 

[gmx-users] gromacs 5.1.2 mdrun can't detect GPU

[treinz]

Hi all,


I recently built 5.1.2 with GPU support and the config options are:


module load cuda/7.5.18
cmake .. -DCMAKE_C_COMPILER=gcc-4.9 \
 -DCMAKE_CXX_COMPILER=g++-4.9 \
 -DGMX_MPI=OFF \
 -DGMX_THREAD_MPI=ON \
 -DGMX_GPU=ON \
 -DCMAKE_PREFIX_PATH=$HOME/local \
 -DCMAKE_INSTALL_PREFIX=$HOME/local/gromacs/grid_frontend \
 -DGMX_BUILD_OWN_FFTW=ON \
 -DGMX_DEFAULT_SUFFIX=OFF \
 -DGMX_BINARY_SUFFIX=_gpu \
 -DGMX_LIBS_SUFFIX=_gpu


and the installation was successful. But when I tried running mdrun, it wasn't 
able to detect the GPU:


Build OS/arch:  Linux 2.6.32-573.1.1.el6.x86_64 x86_64
Build CPU vendor:   GenuineIntel
Build CPU brand:Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
Build CPU family:   6   Model: 63   Stepping: 2
Build CPU features: aes apic avx avx2 clfsh cmov cx8 cx16 f16c fma htt lahf_lm 
mmx msr nonstop_tsc pcid pclmuldq pdcm pdpe1gb popcnt pse rdrnd rdtscp sse2 
sse3 sse4.1 sse4.2 ssse3 tdt x2apic
C compiler: /net/noble/vol1/home/dejunlin/local/bin/mpicc GNU 4.9.3
C compiler flags:-march=core-avx2-Wextra 
-Wno-missing-field-initializers -Wno-sign-compare -Wpointer-arith -Wall 
-Wno-unused -Wunused-value -Wunused-parameter  -O3 -DNDEBUG -funroll-all-loops 
-fexcess-precision=fast  -Wno-array-bounds
C++ compiler:   /net/noble/vol1/home/dejunlin/local/bin/mpicxx GNU 4.9.3
C++ compiler flags:  -march=core-avx2-Wextra 
-Wno-missing-field-initializers -Wpointer-arith -Wall -Wno-unused-function  -O3 
-DNDEBUG -funroll-all-loops -fexcess-precision=fast  -Wno-array-bounds
Boost version:  1.59.0 (external)
CUDA compiler:  
/net/gs/vol3/software/modules-sw/cuda/7.5.18/Linux/RHEL6/x86_64/bin/nvcc nvcc: 
NVIDIA (R) Cuda compiler driver;Copyright (c) 2005-2015 NVIDIA 
Corporation;Built on Tue_Aug_11_14:27:32_CDT_2015;Cuda compilation tools, 
release 7.5, V7.5.17
CUDA compiler 
flags:-gencode;arch=compute_20,code=sm_20;-gencode;arch=compute_30,code=sm_30;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_52,code=compute_52;-use_fast_math;;
 
;-march=core-avx2;-Wextra;-Wno-missing-field-initializers;-Wpointer-arith;-Wall;-Wno-unused-function;-O3;-DNDEBUG;-funroll-all-loops;-fexcess-precision=fast;-Wno-array-bounds;
CUDA driver:0.0
CUDA runtime:   0.0


NOTE: Error occurred during GPU detection:
  CUDA driver version is insufficient for CUDA runtime version
  Can not use GPU acceleration, will fall back to CPU kernels.


The command lines for this run is:


module load cuda/7.5.18
source $HOME/local/gromacs/grid_frontend/bin/GMXRC
env
nvidia-smi -a
gmx_gpu mdrun -ntmpi 24 -ntomp 1 -gpu_id 1 -deffnm


I look at the stdout from nvidia-smi, it looks like the nvidia driver was 
installed (there are 4 GPUs but I'm only showing one of them):


==NVSMI LOG==


Timestamp   : Wed Apr 20 15:05:52 2016
Driver Version  : 352.39


Attached GPUs   : 4
GPU :02:00.0
Product Name: Tesla K40c
Product Brand   : Tesla
Display Mode: Disabled
Display Active  : Disabled
Persistence Mode: Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 1920
Driver Model
Current : N/A
Pending : N/A
Serial Number   : 0321715040048
GPU UUID: GPU-647b8474-e09f-7f98-3ac5-4604e02f1c75
Minor Number: 0
VBIOS Version   : 80.80.3E.00.02
MultiGPU Board  : No
Board ID: 0x200
Inforom Version
Image Version   : 2081.0206.01.04
OEM Object  : 1.1
ECC Object  : 3.0
Power Management Object : N/A
GPU Operation Mode


One catch is that I didn't have the CUDA GPU Deployment Kit or the NVML library 
installed -- would that matter in detecting GPUs? Another catch is that I was 
running the job in a SGE cluster. I believe I have requested the GPU resources 
correctly because I could compile GROMACS with GPU support on the GPU nodes, 
although the resulting mdrun still wasn't able to detect any GPU if I ran it on 
the same GPU node where it was compiled.


Can anyone tell me if there's any flag I have to turn on during compilation or 
running mdrun in order to make it see the GPUs? Also, is NVML or the dev kit a 
must for this?


Thanks,
Tim
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit