Power9 (for HPC) is 4-way SMT, so make sure to try 1,2, and 4 threads per core (stride 4, 2, and 1 respectively). Especially if you are offloading all force computing to the GPU, what remains on the couch may not be able to benefit from more than 1-2 threads per core.
-- Szilárd On Thu, May 2, 2019, 01:19 Alex <nedoma...@gmail.com> wrote: > Well, unless something important has changed within a year, I distinctly > remember being advised here not to offload anything to GPU for EM. Not > that we ever needed to, to be honest... > > In any case, we appear to be dealing with build issues here. > > Alex > > On 5/1/2019 5:09 PM, Kevin Boyd wrote: > > Hi, > > > >> Of course, i am not. This is the EM. ;) > > I haven't looked back at the code, but IIRC EM can use GPUs for the > > nonbondeds, just not the PME. I just double-checked on one of my systems > > with 10 cores and a GTX 1080 Ti, offloading to the GPU more than doubled > > the minimization speed. > > > > Kevin > > > > On Wed, May 1, 2019 at 6:33 PM Alex <nedoma...@gmail.com> wrote: > > > >> Of course, i am not. This is the EM. ;) > >> > >> On Wed, May 1, 2019, 4:30 PM Kevin Boyd <kevin.b...@uconn.edu> wrote: > >> > >>> Hi, > >>> > >>> In addition to what Mark said (and I've also found pinning to be > critical > >>> for performance), you're also not using the GPUs with "-pme cpu -nb > cpu". > >>> > >>> Kevin > >>> > >>> On Wed, May 1, 2019 at 5:56 PM Alex <nedoma...@gmail.com> wrote: > >>> > >>>> Well, my experience so far has been with the EM, because the rest of > >> the > >>>> script (with all the dynamic things) needed that to finish. And it > >>>> "finished" by hitting the wall. However, your comment does touch upon > >>> what > >>>> to do with thread pinning and I will try to set '-pin on' throughout > to > >>> see > >>>> if things make a difference for the better. I am less confident about > >>>> setting strides because it is unclear what the job manager provides in > >>>> terms of the available core numbers. I will play around some more and > >>>> report here. > >>>> > >>>> Thanks! > >>>> > >>>> Alex > >>>> > >>>> On Wed, May 1, 2019 at 3:49 PM Mark Abraham <mark.j.abra...@gmail.com > > > >>>> wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> As with x86, GROMACS uses SIMD intrinsics on POWER9 and is thus > >> fairly > >>>>> insensitive to the compiler's vectorisation abilities. GCC is the > >> only > >>>>> compiler we've tested, as xlc can't compile simple C++11. As > >>> everywhere, > >>>>> you should use the latest version of gcc, as IBM spent quite some > >> years > >>>>> landing improvements for POWER9. > >>>>> > >>>>> EM is useless as a performance indicator of a dynamical simulation, > >>> avoid > >>>>> that - it runs serial code much much more often. > >>>>> > >>>>> Your run deliberately didn't fill the available cores, so just like > >> on > >>>> x86, > >>>>> mdrun will leave the thread affinity handling to the environment, > >> which > >>>> is > >>>>> often a path to bad performance. So, if you plan on doing that often, > >>>>> you'll want to check out the mdrun performance guide docs about the > >>> mdrun > >>>>> -pin and related options. > >>>>> > >>>>> Mark > >>>>> > >>>>> > >>>>> On Wed., 1 May 2019, 23:21 Alex, <nedoma...@gmail.com> wrote: > >>>>> > >>>>>> Hi all, > >>>>>> > >>>>>> Our institution decided to be all fancy, so now we have a bunch of > >>>> Power9 > >>>>>> nodes, each with 80 cores + 4 Volta GPUs. Stuff is managed by > >> slurm. > >>>>> Today > >>>>>> I did a simple EM ('gmx mdrun -ntomp 4 -ntmpi 4 -pme cpu -nb cpu') > >>> and > >>>>> the > >>>>>> performance is abysmal, I would guess 100 times slower than on > >>> anything > >>>>>> I've ever seen before. > >>>>>> > >>>>>> Our admin person emailed me the following: > >>>>>> "-- it would not surprise me if the GCC compilers were relatively > >> bad > >>>> at > >>>>>> taking advantage of POWER9 vectorization, they're likely optimized > >>> for > >>>>>> x86_64 vector stuff like SSE and AVX operations. This was an issue > >>> in > >>>>> the > >>>>>> build, I selected "-DGMX_SIMD=IBM_VSX" for the config, but > >> according > >>> to > >>>>> my > >>>>>> notes, that was part of an attempt to fix the "unimplemented SIMD" > >>>> error > >>>>>> that was dogging me at first, and/but which was eventually cleared > >> by > >>>>>> switching to gcc-6." > >>>>>> > >>>>>> Does anyone have any comments/suggestions on building and running > >> GMX > >>>> on > >>>>>> Power9? > >>>>>> > >>>>>> Thank you, > >>>>>> > >>>>>> Alex > >>>>>> -- > >>>>>> Gromacs Users mailing list > >>>>>> > >>>>>> * Please search the archive at > >>>>>> > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0 > >>>> before > >>>>>> posting! > >>>>>> > >>>>>> * Can't post? Read > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0 > >>>>>> * For (un)subscribe requests visit > >>>>>> > >> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=mJ%2FfYvTgmL49ZCAUYzSRJqz%2FJY8MxQdGpoYwKtbN39U%3D&reserved=0 > >>>> or > >>>>>> send a mail to gmx-users-requ...@gromacs.org. > >>>>>> > >>>>> -- > >>>>> Gromacs Users mailing list > >>>>> > >>>>> * Please search the archive at > >>>>> > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0 > >>>> before > >>>>> posting! > >>>>> > >>>>> * Can't post? Read > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0 > >>>>> * For (un)subscribe requests visit > >>>>> > >> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=mJ%2FfYvTgmL49ZCAUYzSRJqz%2FJY8MxQdGpoYwKtbN39U%3D&reserved=0 > >>>> or > >>>>> send a mail to gmx-users-requ...@gromacs.org. > >>>>> > >>>> -- > >>>> Gromacs Users mailing list > >>>> > >>>> * Please search the archive at > >>>> > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=zejDS0OvUCl%2BSch%2BzVtxic%2B%2BDFIPEhB1DygmpmQ2dvw%3D&reserved=0 > >>>> before posting! > >>>> > >>>> * Can't post? Read > >>>> > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018052656&sdata=X87Kk%2FtkodePJ9uhDb9XPIA0Xhaqi52e6Z9%2FhqY35fo%3D&reserved=0 > >>>> * For (un)subscribe requests visit > >>>> > >>>> > >> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0 > >>>> or send a mail to gmx-users-requ...@gromacs.org. > >>>> > >>> -- > >>> Gromacs Users mailing list > >>> > >>> * Please search the archive at > >>> > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=VXuy%2Bwx5Pcv3vgqNNLcXVbcsiAgWHI3nLdhqD4Yc7Io%3D&reserved=0 > >> before > >>> posting! > >>> > >>> * Can't post? Read > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=hNfd9HQH4i7TmU4wVUaos%2BzxqkRuKpEcVyswRFBrnyk%3D&reserved=0 > >>> * For (un)subscribe requests visit > >>> > >> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0 > >> or > >>> send a mail to gmx-users-requ...@gromacs.org. > >>> > >> -- > >> Gromacs Users mailing list > >> > >> * Please search the archive at > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists%2FGMX-Users_List&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=VXuy%2Bwx5Pcv3vgqNNLcXVbcsiAgWHI3nLdhqD4Yc7Io%3D&reserved=0 > >> before posting! > >> > >> * Can't post? Read > >> > https://nam01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.gromacs.org%2FSupport%2FMailing_Lists&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=hNfd9HQH4i7TmU4wVUaos%2BzxqkRuKpEcVyswRFBrnyk%3D&reserved=0 > >> > >> * For (un)subscribe requests visit > >> > >> > https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmaillist.sys.kth.se%2Fmailman%2Flistinfo%2Fgromacs.org_gmx-users&data=02%7C01%7Ckevin.boyd%40uconn.edu%7C5ae99d654910469ebe9008d6ce8502d1%7C17f1a87e2a254eaab9df9d439034b080%7C0%7C0%7C636923468018062645&sdata=jxfG7UF4cZYh%2FOnDgFR%2F%2FcNgCrD7DytsWNXgvtd8wlU%3D&reserved=0 > >> or send a mail to gmx-users-requ...@gromacs.org. > >> > -- > Gromacs Users mailing list > > * Please search the archive at > http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before > posting! > > * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists > > * For (un)subscribe requests visit > https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or > send a mail to gmx-users-requ...@gromacs.org. > -- Gromacs Users mailing list * Please search the archive at http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting! * Can't post? Read http://www.gromacs.org/Support/Mailing_Lists * For (un)subscribe requests visit https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a mail to gmx-users-requ...@gromacs.org.