Thanks Davide, I am running using the smearing option since the system is metallic.
I also noticed an interesting relation. GIPAW runs succeed if number of cores (np) <= number of k points/npool. I checked this in the 38 atom case which kept failing whenever I chose a number of processors higher than the number of kpoints per pool. Although the SCF runs was finishing successfully all the time. This was also observed in other cases. Is this a general rule? I will send you the files privately. Yasser -----Original Message----- From: Davide Ceresoli [mailto:[email protected]] Sent: Sunday, July 16, 2017 1:49 PM To: Yasser Fowad AlWahedi <[email protected]>; PWSCF Forum <[email protected]> Subject: Re: [Pw_forum] GIPAW acceleration Dear Yasser, no problem! First of all, it seems to me that I/O is not a problem. In fact cputime ~= walltime and davcio routines consume only 1.88 s. I compared calculations of similar size and I've got: wollastonite: 30 atoms, 36 k-points: 10h40m coesite: 48 atoms, 32 k-points: 19h20m on a rather old (2008) Xeon E5520 2.27 GHz, 8 cores. My timings are more favorable than your C1 results. However, if your system is a slab, the empty space carries a non-neglibigle extra cost. You can try to minimize it as much as possible. NMR interactions are short-ranged, contrary to electrostatic interactions Is your system metallic? even if it has a small band gap, I suggest using occupations='smearing'. This will speed up the linear-response in GIPAW, and convergence wrt k-points. Finally, the clock difference between the i7 (3.5 GHz) and the Xeon (2.2 GHz) can explain the difference in timing. The clock ratio is ~1.6, similar to the walltime ratio. In any case, if you send me privately input and output files, I can look them in detail. Best wishes, Davide On 07/16/2017 10:26 AM, Yasser Fowad AlWahedi wrote: > Dear Davide, > > Thanks for your support and my apologies for the late reply. PW and GIPAW > are compiled using GNU compilers and the intel MKL libs. > > I am running DFT of Ni2P clusters of various surfaces over two computational > rigs: > > 1) The university cluster: Each node consist of dual 8 cores/8 threads > CPUs Xeons clocked at 2.2 GHz with 64 GB ram. I only use one node per > simulation. For storage it uses a mechanical hard drive . (Later > called C1) > > 2) My home pc: which is equipped with i7 5930K processor 6 cores 12 threads > clocked at 3.9 GHz with 128 GB ram (Later called C2). For storage I use a > Samsung 850 EVO SSD. > > Below table summarize the cases performed/running and the time of finish or > expected time of finishing assuming linear extrapolation. > > > # of atoms npool Cores # kpoints per pool Computer Time > (hrs) > 30 2 16 17 C1 28.9 > 38 1 16 25 C1 31.3 > 49 1 16 34 C1 124.9* > 50 2 16 17 C1 474.6* > 52 1 10 34 C2 295.2* > > * estimated time of finish > > I understand that the cases are different and as such they will require more > or less time to finish. > > But I noticed that the 50 and 52 cases which are quite similar (same k points > and similar number of atoms) but done over two different systems attain > substantially different time of finish. My guess it is probably due to the > SSD being used to write off the data. Considering that C2 uses less > computational threads and more atoms but is expected to finish faster. > > I also noticed an interesting relation. GIPAW runs succeed if number of > cores (np) <= number of k points/npool. I checked this in the 38 atom case > which kept failing whenever I chose a number of processors higher than the > number of kpoints per pool. Although the SCF runs was finishing successfully > all the time. This was also observed in other cases. Is this a general rule? > > Below is the timing output of the 38 atoms case: > > gipaw_setup : 0.46s CPU 0.50s WALL ( 1 calls) > > Linear response > greenf : 20177.91s CPU 20207.68s WALL ( 600 calls) > cgsolve : 20057.24s CPU 20086.82s WALL ( 600 calls) > ch_psi : 19536.93s CPU 19563.75s WALL ( 44231 calls) > h_psiq : 13685.97s CPU 13707.40s WALL ( 44231 calls) > > Apply operators > h_psi : 44527.30s CPU 46802.35s WALL ( 5434310 calls) > apply_vel : 262.98s CPU 263.30s WALL ( 525 calls) > > Induced current > j_para : 559.19s CPU 560.39s WALL ( 675 calls) > biot_savart : 0.05s CPU 0.06s WALL ( 1 calls) > > Other routines > > General routines > calbec : 39849.22s CPU 37474.79s WALL (10917262 calls) > fft : 0.12s CPU 0.15s WALL ( 42 calls) > ffts : 0.01s CPU 0.01s WALL ( 10 calls) > fftw : 8220.39s CPU 9116.72s WALL (27084278 calls) > davcio : 0.02s CPU 1.88s WALL ( 400 calls) > > Parallel routines > fft_scatter : 3533.10s CPU 3242.29s WALL (27084330 calls) > > Plugins > > GIPAW : 112557.79s CPU 112726.12s WALL ( 1 calls) > > Yasser > > > > > -----Original Message----- > From: [email protected] [mailto:[email protected]] > On Behalf Of Davide Ceresoli > Sent: Thursday, July 13, 2017 8:30 PM > To: PWSCF Forum <[email protected]> > Subject: Re: [Pw_forum] GIPAW acceleration > > Dear Yasser, > how many atoms? how many k-points? I/O can always be the reason, but in > my experience if the system is very large, time is dominated by computation, > not I/O. > You should get some speedup if diagonalization='cg' in GIPAW. > > Anyway, if I have time, I will introduce a "disk_io" variable in the input > file, to try to keep more data in memory instead that on disk. > > Best regards, > Davide > > > On 07/13/2017 10:02 AM, Yasser Fowad AlWahedi wrote: >> Dear GIPAW users, >> >> >> >> For nmr shifts calculations, I am suffering from the extreme slowness >> of GIPAW nmr shifts calculations. I have noticed that GIPAW write >> off the results frequently for restart purposes. In our clusters we >> have mechanical hard drives which stores the off data for. Could that be a >> reason for its slowness? >> >> >> >> Yasser Al Wahedi >> >> Assistant Professor >> >> Khalifa University of Science and Technology >> > -- +--------------------------------------------------------------+ Davide Ceresoli CNR Institute of Molecular Science and Technology (CNR-ISTM) c/o University of Milan, via Golgi 19, 20133 Milan, Italy Email: [email protected] Phone: +39-02-50314276, +39-347-1001570 (mobile) Skype: dceresoli +--------------------------------------------------------------+ _______________________________________________ Pw_forum mailing list [email protected] http://pwscf.org/mailman/listinfo/pw_forum
