Dear Pavel, Many thanks again for your patience and guidance. With the libopenblas-openmp-dev package it seems to work well!
$ ldd lapw1 linux-vdso.so.1 (0x00007ffca83d8000) *libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0 (0x000014563f924000) * libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 (0x000014563f65c000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014563f50d000) libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 (0x000014563f4e1000) libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x000014563f49f000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x000014563f47c000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014563f288000) /lib64/ld-linux-x86-64.so.2 (0x0000145641b2d000) libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 (0x000014563f23e000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x000014563f223000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000014563f21d000) and these options in siteconfig: L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-gnu/openblas-openmp R R_LIBS (LAPACK+BLAS): -lopenblas I also get better timings now (though the TIME HAMILT are slightly longer, but overall improvement): $ ../x lapw1 STOP LAPW1 END 119.400u 1.937s 0:32.53 372.9% 0+0k 0+37864io 0pf+0w $ grep HORB *output1* test_case.output1: TIME HAMILT (CPU) = 17.3, HNS = 18.4, HORB = 0.0, DIAG = 85.0, SYNC = 0.0 test_case.output1: TIME HAMILT (WALL) = 4.6, HNS = 5.2, HORB = 0.0, DIAG = 22.0, SYNC = 0.0 Thanks for your help, David --- *Dr David Holec* *Computational Materials Science group* Department of Materials Science Montanuniversität Leoben Franz-Josef-Strasse 18, A-8700 Leoben, Austria tel. +43-(0)3842-4024211 fax. +43-(0)3842-4024202 materials.unileoben.ac.at cms.unileoben.ac.at ________________________________ WHERE RESEARCH MEETS FUTURE On Wed, 24 Nov 2021 at 12:39, Pavel Ondračka <pavel.ondra...@email.cz> wrote: > Hi David, > > well, it is hard to say without the debug info why the OpenBLAS crahes. > My guess is that you link with the 64bit interface, try to install the > standard one (openblas-openmp-devel) and replace openblas64-openmp with > openblas-openmp everywhere in you config. Also remove the -lpthread > (just to be safe, but in theory should not matter), it is not needed > with OpenMP. If it still crashes, please recompile with debug info > enabled (add -g to compiler options) and send me the x lapw1 output via > PM. > > BTW my response was mostly motivated by me suspecting you actually link > against slow netlib BLAS (which turned out to be the case) and I wanted > to warn others in case someone in the future would be using your > settings as a reference :-) > > Best regards > Pavel > > On Wed, 2021-11-24 at 11:09 +0100, David Holec wrote: > > Hi Pavel, > > > > Many thanks for your insights. As you know, I am not an expert on how > > to compile codes, for me, this is sadly a trial and error adventure. > > > > I tried to compile it against the openblas library, but although the > > compilation ends without any errors, I get a segmentation fault when > > running lapw1 (on the test case > > from http://www.wien2k.at/reg_user/benchmark/). The current setting > > are: > > > > L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux- > > gnu/openblas64-openmp > > R R_LIBS (LAPACK+BLAS): /usr/lib/x86_64-linux-gnu/openblas64- > > openmp/libopenblas64.so.0 -lpthread > > > > > > (The rest is as I wrote in my first email.) Here is the list of > > linked libraries: > > $ ldd lapw1 > > linux-vdso.so.1 (0x00007ffea57d6000) > > libopenblas64.so.0 => /lib/x86_64-linux- > > gnu/libopenblas64.so.0 (0x000014fe2b2e5000) > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > > (0x000014fe2b2c2000) > > libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 > > (0x000014fe2affa000) > > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 > > (0x000014fe2aeab000) > > libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 > > (0x000014fe2ae7f000) > > libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 > > (0x000014fe2ae3d000) > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 > > (0x000014fe2ac49000) > > /lib64/ld-linux-x86-64.so.2 (0x000014fe2d4d3000) > > libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 > > (0x000014fe2abff000) > > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > > (0x000014fe2abe4000) > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > > (0x000014fe2abde000) > > > > And here is the stacking fault (it doesn't tell me anything): > > $x lapw1 > > > > Program received signal SIGSEGV: Segmentation fault - invalid memory > > reference. > > > > Backtrace for this error: > > #0 0x148fb48b8d21 in ??? > > #1 0x148fb48b7ef5 in ??? > > #2 0x148fb452d20f in ??? > > #3 0x148fb54495fb in ??? > > Segmentation fault > > 0.949u 0.472s 0:00.84 167.8% 0+0k 14400+10992io 31pf+0w > > > > Any idea? > > > > With the settings I reported yesterday, everything works just fine > > (though probably not very efficiently - but this is not a problem for > > me as this binary is not a "production" binary on any HPC): > > > > $ ldd lapw1 > > linux-vdso.so.1 (0x00007ffc60765000) > > libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3 > > (0x0000150cec90f000) > > liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3 > > (0x0000150cec26b000) > > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 > > (0x0000150cec248000) > > libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5 > > (0x0000150cebf80000) > > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 > > (0x0000150cebe31000) > > libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1 > > (0x0000150cebe05000) > > libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 > > (0x0000150cebdc1000) > > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 > > (0x0000150cebbcf000) > > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 > > (0x0000150cebbb4000) > > /lib64/ld-linux-x86-64.so.2 (0x0000150cec9fc000) > > libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0 > > (0x0000150cebb6a000) > > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 > > (0x0000150cebb64000) > > > > and: > > > > $ x lapw1 > > STOP LAPW1 END > > 162.045u 0.918s 2:31.25 107.7% 0+0k 8976+37856io 35pf+0w > > $ grep HORB *output1* > > test_case.output1: TIME HAMILT (CPU) = 16.3, HNS = 20.0, > > HORB = 0.0, DIAG = 125.9, SYNC = 0.0 > > test_case.output1: TIME HAMILT (WALL) = 4.4, HNS = 20.0, > > HORB = 0.0, DIAG = 126.0, SYNC = 0.0 > > > > (I am using it on my 10-year old "type writer": > > $lscpu > > Architecture: x86_64 > > CPU op-mode(s): 32-bit, 64-bit > > Byte Order: Little Endian > > Vendor ID: GenuineIntel > > CPU family: 6 > > Model: 26 > > Model name: Intel(R) Xeon(R) CPU W3550 > > @ 3.07GHz > > ) > > > > > > --- > > Dr David Holec > > Computational Materials Science group > > Department of Materials Science > > Montanuniversität Leoben > > > > > > > > Franz-Josef-Strasse 18, A-8700 Leoben, Austria > > tel. +43-(0)3842-4024211 > > fax. +43-(0)3842-4024202 > > materials.unileoben.ac.at > > cms.unileoben.ac.at > > ________________________________ > > WHERE RESEARCH MEETS FUTURE > > > > > > On Wed, 24 Nov 2021 at 08:27, Pavel Ondračka > > <pavel.ondra...@email.cz> wrote: > > > Hi David, > > > > > > as you said it works for you, so feel free to ignore, but I have > > > some > > > further tips if you are interested. Ubuntu switches between the > > > different blas and lapack using the "alternatives", so its > > > difficult > > > to > > > say if you actually link with the correct one. > > > > > > "ldd lapw1" in WIENROOT should show which one is actually linked, > > > what > > > you want to have is the openmp openblas > > > /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so > > > /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so > > > or alternatively > > > /usr/lib/x86_64-linux-gnu/openblas-openmp/libopenblas.so > > > It looks like you linked with the pthread one. This is not a > > > problem > > > when running at single thread but at higher thread number this > > > might > > > lead to oversubscription and slowdowns as the pthreaded openblas > > > doesn't respect the OMP_NUM_THREADS set by Wien2k. So I would > > > recommend > > > to relink with the openmp OpenBLAS. BTW it is usually safer to link > > > with OpenBLAS explicitly using the -lopenblas instead of the - > > > llapack > > > - > > > lblas to be sure you don't accidentally link the netlib one > > > (libopenblas is just the libblas and libblapack provided by > > > OpenBLAS > > > merged together). > > > > > > In general easy way how to check performance is to run the serial > > > test_case from http://www.wien2k.at/reg_user/benchmark/ On modern > > > CPUs > > > (at least avx2) the runtime should be around 15-25 seconds at > > > single > > > thread. > > > > > > I see total runtime of ~18seconds on Fedora 35 with gfortran 11.2.1 > > > OpenBLAS and AMD Ryzen 9 3900X 12-Core Processor. > > > Also look for the following line in test_case.output1, this is what > > > I > > > have: > > > TIME HAMILT (WALL) = 2.2, HNS = 1.7, HORB = 0.0, DIAG > > > = > > > > > > 14.0, SYNC = 0.0 > > > The time in HAMILT mostly depends on you compiler and vectorizing > > > settings, while the DIAG is 99% lapack/blas related, so this can > > > help > > > with the diagnostics if things are slow. > > > > > > You might also get extra speedup of the HAMILT part by adding "- > > > DHAVE_LIBMVEC" to the Compiler options. > > > > > > Best regards > > > Pavel > > > > > > On Tue, 2021-11-23 at 11:07 +0100, David Holec wrote: > > > > Dear all, > > > > > > > > I have just spent some time making Wien2k run on my single > > > > machine > > > > running Ubuntu 20.04 with gfortran/gcc. Since I am not an expert, > > > it > > > > was a trial and error, but it seems that I found a working > > > combination > > > > (sadly, the default parameters didn't work for me). Maybe this > > > > will > > > > help someone. Here are the settings that did the job for me: > > > > > > > > M OpenMP switch: -fopenmp > > > > O Compiler options: -ffree-form -O2 -ftree-vectorize - > > > > march=native -ffree-line-length-none -ffpe-summary=none > > > > L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux- > > > > gnu > > > > P Preprocessor flags '-DParallel' > > > > R R_LIBS (LAPACK+BLAS): -lblas -llapack -lpthread > > > > F FFTW options: -DFFTW3 -DFFTW_OMP -I/usr/include > > > > FFTW-LIBS: -L/usr/lib/x86_64-linux-gnu - > > > > lfftw3 > > > - > > > > lfftw3_omp > > > > > > > > where the FFTW options were: > > > > > > > > R FFTWROOT: /usr/ > > > > V FFTW_VERSION: FFTW3 > > > > L FFTW_LIB: lib/x86_64-linux-gnu > > > > N FFTW_LIBNAME: fftw3 > > > > > > > > Compiler versions: > > > > $ gcc --version > > > > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 > > > > gfortran --version > > > > GNU Fortran (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0 > > > > > > > > And I used the generic lapack and openblas packages provides by > > > Ubuntu > > > > repos: > > > > liblapack-dev/focal,now 3.9.0-1build1 amd64 [installed] > > > > liblapack3/focal,now 3.9.0-1build1 amd64 [installed,automatic] > > > > > > > > liblapack64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic] > > > > liblapack64-dev/focal,now 3.9.0-1build1 amd64 [installed] > > > > > > > > libblas-dev/focal,now 3.9.0-1build1 amd64 [installed] > > > > libblas3/focal,now 3.9.0-1build1 amd64 [installed,automatic] > > > > libblas64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic] > > > > libblas64-dev/focal,now 3.9.0-1build1 amd64 [installed,automatic] > > > > > > > > libopenblas64-0/focal-updates,now 0.3.8+ds-1ubuntu0.20.04.1 amd64 > > > > [installed] > > > > libopenblas64-0-openmp/focal-updates,now 0.3.8+ds- > > > > 1ubuntu0.20.04.1 > > > > amd64 [installed] > > > > libopenblas64-0-pthread/focal-updates,now 0.3.8+ds- > > > > 1ubuntu0.20.04.1 > > > > amd64 [installed,automatic] > > > > > > > > (I am not totally sure if I need all the libraries above, but > > > > certainly, with these, the compilation seems to work and I am > > > > able > > > to > > > > run SCF cycles & Telnes calculations without errors :-) > > > > > > > > All the best, > > > > David > > > > --- > > > > Dr David Holec > > > > Computational Materials Science group > > > > Department of Materials Science > > > > Montanuniversität Leoben > > > > > > > > > > > > > > > > Franz-Josef-Strasse 18, A-8700 Leoben, Austria > > > > tel. +43-(0)3842-4024211 > > > > fax. +43-(0)3842-4024202 > > > > materials.unileoben.ac.at > > > > cms.unileoben.ac.at > > > > ________________________________ > > > > WHERE RESEARCH MEETS FUTURE > > > > _______________________________________________ > > > > Wien mailing list > > > > Wien@zeus.theochem.tuwien.ac.at > > > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien > > > > SEARCH the MAILING-LIST at: > > > > > > > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html > > > > > > _______________________________________________ > > > Wien mailing list > > > Wien@zeus.theochem.tuwien.ac.at > > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien > > > SEARCH the MAILING-LIST at: > > > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html > > _______________________________________________ > > Wien mailing list > > Wien@zeus.theochem.tuwien.ac.at > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien > > SEARCH the MAILING-LIST at: > > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html > > _______________________________________________ > Wien mailing list > Wien@zeus.theochem.tuwien.ac.at > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien > SEARCH the MAILING-LIST at: > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html >
_______________________________________________ Wien mailing list Wien@zeus.theochem.tuwien.ac.at http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien SEARCH the MAILING-LIST at: http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html