Dear Mohammad,
> 2. You are using DFT+U, more precisely > U_projection_type = 'ortho-atomic' > The GPU acceleration of this functionality is limited and some portions > of the algorithm will still run on the CPU. In addition, I fear that the > evaluation of forces with that projection method scales pretty bad with > the number of atoms, but I let the experts (and developers) of this new > functionality further comment this last point. Yes, indeed, in DFT+U with the 'ortho-atomic' Hubbard manifold the calculations of Hubbard forces and stress take much more time than with the 'atomic' Hubbard manifold. See Phys. Rev. B 102, 235159 (2020), in particular see Appendix C. As Pietro said, the ortho-atomic Hubbard forces and stress were ported to the GPU version of QE but this is still not optimal and can be improved in the future. Greetings, Iurii -- Dr. Iurii TIMROV Postdoctoral Researcher STI - IMX - THEOS and NCCR - MARVEL Swiss Federal Institute of Technology Lausanne (EPFL) CH-1015 Lausanne, Switzerland +41 21 69 34 881 http://people.epfl.ch/265334 ________________________________ From: users <[email protected]> on behalf of Pietro Bonfa' <[email protected]> Sent: Monday, January 4, 2021 11:52:31 AM To: [email protected] Subject: Re: [QE-users] [QE-GPU] computing force and stress - time cost Dear Mohammad, Paolo is right, but a couple of comments can already be made: 1. you are not using OpenMP parallelism, and I believe you have more than 2 cores in your system. In order to achieve a decent speedup (at least in the SCF) it's mandatory to enable openmp and exploit the whole CPU power. 2. You are using DFT+U, more precisely U_projection_type = 'ortho-atomic' The GPU acceleration of this functionality is limited and some portions of the algorithm will still run on the CPU. In addition, I fear that the evaluation of forces with that projection method scales pretty bad with the number of atoms, but I let the experts (and developers) of this new functionality further comment this last point. Best regards and happy new year, Pietro On 1/4/21 10:07 AM, Paolo Giannozzi wrote: > The most important piece of information (the final time report) is not > contained in your 45Mb output. > > Paolo > > On Mon, Jan 4, 2021 at 6:33 AM Mohammad Moaddeli > <[email protected] <mailto:[email protected]>> wrote: > > Dear Pietro, > > It takes about 22 hours to perform an scf (about 2 hours to perform > diagonalization until convergence is achieved, and about 20 hours to > compute force and stress). > Here is the google drive link containing input and output files: > > https://drive.google.com/file/d/1DFtLqFvrc8CFo1_q_jjnMFErvXpWEAHB/view?usp=sharing > > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1DFtLqFvrc8CFo1_q_jjnMFErvXpWEAHB%2Fview%3Fusp%3Dsharing&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257890985%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=s5PSyKLXmuHXgzPSj9xcsCqPXt%2BmKPw9mj%2BZACc9sSk%3D&reserved=0> > > hp.x will be performed after vc-relax is done. > > Thanks in advance, > Mohammad > > On Mon, Jan 4, 2021 at 2:38 AM Pietro Bonfa' <[email protected] > <mailto:[email protected]>> wrote: > > Dear Mohammad, > > the performance of the GPU code depends dramatically on the > portions of > computation that are still performed on the CPU. Only a portion > of all > contributions to forces have been accelerated, and what is left > out may > be optimized for MPI parallelism rather than openmp. > > That being said, the behavior that you report is definitively > unusual. > Would you mind sharing input and output files? > > Best regards, > Pietro > > > > On 1/3/21 8:52 AM, Mohammad Moaddeli wrote: > > Dear all, > > > > GPU enabled QE v.6.7 is compiled on a VOLTA card. I am trying > to run a > > vc-relax for a bulk containing 48 atoms. Although > diagonalization > > (davidson) is about 3x faster than CPU, it takes a lot of > time (a couple > > of hours) to compute force and stress. Is this something > related to the > > code itself? > > > > Best, > > > > Mohammad Moaddeli > > ShirazU > > > > _______________________________________________ > > Quantum ESPRESSO is supported by MaX (<http://>www.max-centre.eu > > <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257900986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=ayiwKOVdIHx6hQ4VpfrjS1rTXOEiHF3ZWgGTRVbawCo%3D&reserved=0>) > > users mailing list [email protected] > <mailto:[email protected]> > > https://lists.quantum-espresso.org/mailman/listinfo/users > > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257900986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=nnTzluNl%2FpWGpQ3HPy0Zm%2Bjvt4qUlNIW5mNyncKQWoU%3D&reserved=0> > > > _______________________________________________ > Quantum ESPRESSO is supported by MaX (<http://>www.max-centre.eu > > <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257910979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=VDmrELj9IT2pHITCuC6xEAj%2Bc9PJuhhij7noi0qPqsM%3D&reserved=0>) > users mailing list [email protected] > <mailto:[email protected]> > https://lists.quantum-espresso.org/mailman/listinfo/users > > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257920970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=IBKLu4F2OC%2BLtjc0eeKxBt1m1dMEzLizqXZ1EbH3%2FSI%3D&reserved=0> > > _______________________________________________ > Quantum ESPRESSO is supported by MaX (<http://>www.max-centre.eu > > <https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257920970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=OYY1he5ImqA5ypHVCP0oxQLCMVoqVa3hXbngEJRmZhw%3D&reserved=0>) > users mailing list [email protected] > <mailto:[email protected]> > https://lists.quantum-espresso.org/mailman/listinfo/users > > <https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257930965%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=TliW5iLzHDjHe1J2j51oBqQAi2iQ8VIb12QQesVXmZA%3D&reserved=0> > > > > -- > Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche, > Univ. Udine, via delle Scienze 208, 33100 Udine, Italy > Phone +39-0432-558216, fax +39-0432-558222 > > > _______________________________________________ > Quantum ESPRESSO is supported by MaX > (www.max-centre.eu<http://www.max-centre.eu>) > users mailing list [email protected] > https://lists.quantum-espresso.org/mailman/listinfo/users > _______________________________________________ Quantum ESPRESSO is supported by MaX (www.max-centre.eu<http://www.max-centre.eu>) users mailing list [email protected] https://lists.quantum-espresso.org/mailman/listinfo/users
_______________________________________________ Quantum ESPRESSO is supported by MaX (www.max-centre.eu) users mailing list [email protected] https://lists.quantum-espresso.org/mailman/listinfo/users
