Hi Junting, PyFR is a modern code. As such when running with the OpenMP backend you only want to use one MPI rank per socket. If you are running on a 12 core CPU there should be one MPI rank (the remaining cores will be saturated by OpenMPI threads). Using more ranks is likely to degrade performance -- especially if you do not tell OpenMP to use less threads (as you may end up with 12*12 = 144 threads which is a substantial amount of over-subscription).
In the case of the CUDA backend all computation is offloaed to the GPU. Thus there should be one MPI rank per CPU. Here, it is expected that all but one of the cores will be idle. Running more ranks is just going to over-subscribe the GPU and degrade performance. Regards, Freddie. On 12/06/2019 15:57, Junting Chen wrote: > Hello Dr.Vincent, > > Thanks for the quick reply. I did try to run the case with both openmp > and cuda backends. > > With CUDA backend (Tesla K80) on a virtual machine, I am constantly > seeing this error message: > > 1.png <about:invalid#zClosurez> > > > > > > > > > > I found out a way to get rid of the message, but there was no > improvement on the performance. (foot note: nvidia-smi tells me 12 cores > are used in both cases) > > 2.png <about:invalid#zClosurez> > > > > Also, I realized that run speed remains the same regardless how many > cores I call (mpirun -n ##). Single core, 12 cores and 20 cores all > requiring 14hrs to run 20s of this 3d cylinder case with 360k cell at > 4th order. I believe there is still some technical issues in here. I am > suspecting the openmpi version is holding me back at the moment. Here it > is openmpi 1.10 and latest version is 4.0. > > As I am running this case with openmp backend on a local machine, I have > encountered similar issue: regardless how many processors I am calling, > there was no improvement on performance. The run speed is extremely > slow. It is openmpi 2.1 on this local machine. > > > I am currently rebuilding the latest openmpi on the local machine (there > is some difficulties on updating cuda aware openmpi on the virtual > machine). Besides the openmpi issue, anything else might go wrong? > Thanks again. > > Junting Chen > > -- > You received this message because you are subscribed to the Google > Groups "PyFR Mailing List" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] > <mailto:[email protected]>. > To post to this group, send email to [email protected] > <mailto:[email protected]>. > Visit this group at https://groups.google.com/group/pyfrmailinglist. > To view this discussion on the web, visit > https://groups.google.com/d/msgid/pyfrmailinglist/101cc570-45fe-454a-8374-9a29d277b46a%40googlegroups.com > <https://groups.google.com/d/msgid/pyfrmailinglist/101cc570-45fe-454a-8374-9a29d277b46a%40googlegroups.com?utm_medium=email&utm_source=footer>. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "PyFR Mailing List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send an email to [email protected]. Visit this group at https://groups.google.com/group/pyfrmailinglist. To view this discussion on the web, visit https://groups.google.com/d/msgid/pyfrmailinglist/55cde3a0-e62c-d8cb-da60-b13c17c9f1a4%40witherden.org. For more options, visit https://groups.google.com/d/optout.
