Dear Pavel,
Many thanks again for your patience and guidance. With the
libopenblas-openmp-dev package it seems to work well!
$ ldd lapw1
linux-vdso.so.1 (0x00007ffca83d8000)
*libopenblas.so.0 => /lib/x86_64-linux-gnu/libopenblas.so.0
(0x000014563f924000) *
libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
(0x000014563f65c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014563f50d000)
libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1
(0x000014563f4e1000)
libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1
(0x000014563f49f000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x000014563f47c000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014563f288000)
/lib64/ld-linux-x86-64.so.2 (0x0000145641b2d000)
libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
(0x000014563f23e000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
(0x000014563f223000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x000014563f21d000)
and these options in siteconfig:
L Linker Flags: $(FOPT)
-L/usr/lib/x86_64-linux-gnu/openblas-openmp
R R_LIBS (LAPACK+BLAS): -lopenblas
I also get better timings now (though the TIME HAMILT are slightly longer,
but overall improvement):
$ ../x lapw1
STOP LAPW1 END
119.400u 1.937s 0:32.53 372.9% 0+0k 0+37864io 0pf+0w
$ grep HORB *output1*
test_case.output1: TIME HAMILT (CPU) = 17.3, HNS = 18.4, HORB
= 0.0, DIAG = 85.0, SYNC = 0.0
test_case.output1: TIME HAMILT (WALL) = 4.6, HNS = 5.2, HORB
= 0.0, DIAG = 22.0, SYNC = 0.0
Thanks for your help,
David
---
*Dr David Holec*
*Computational Materials Science group*
Department of Materials Science
Montanuniversität Leoben
Franz-Josef-Strasse 18, A-8700 Leoben, Austria
tel. +43-(0)3842-4024211
fax. +43-(0)3842-4024202
materials.unileoben.ac.at
cms.unileoben.ac.at
________________________________
WHERE RESEARCH MEETS FUTURE
On Wed, 24 Nov 2021 at 12:39, Pavel Ondračka <pavel.ondra...@email.cz>
wrote:
> Hi David,
>
> well, it is hard to say without the debug info why the OpenBLAS crahes.
> My guess is that you link with the 64bit interface, try to install the
> standard one (openblas-openmp-devel) and replace openblas64-openmp with
> openblas-openmp everywhere in you config. Also remove the -lpthread
> (just to be safe, but in theory should not matter), it is not needed
> with OpenMP. If it still crashes, please recompile with debug info
> enabled (add -g to compiler options) and send me the x lapw1 output via
> PM.
>
> BTW my response was mostly motivated by me suspecting you actually link
> against slow netlib BLAS (which turned out to be the case) and I wanted
> to warn others in case someone in the future would be using your
> settings as a reference :-)
>
> Best regards
> Pavel
>
> On Wed, 2021-11-24 at 11:09 +0100, David Holec wrote:
> > Hi Pavel,
> >
> > Many thanks for your insights. As you know, I am not an expert on how
> > to compile codes, for me, this is sadly a trial and error adventure.
> >
> > I tried to compile it against the openblas library, but although the
> > compilation ends without any errors, I get a segmentation fault when
> > running lapw1 (on the test case
> > from http://www.wien2k.at/reg_user/benchmark/). The current setting
> > are:
> >
> > L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-
> > gnu/openblas64-openmp
> > R R_LIBS (LAPACK+BLAS): /usr/lib/x86_64-linux-gnu/openblas64-
> > openmp/libopenblas64.so.0 -lpthread
> >
> >
> > (The rest is as I wrote in my first email.) Here is the list of
> > linked libraries:
> > $ ldd lapw1
> > linux-vdso.so.1 (0x00007ffea57d6000)
> > libopenblas64.so.0 => /lib/x86_64-linux-
> > gnu/libopenblas64.so.0 (0x000014fe2b2e5000)
> > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> > (0x000014fe2b2c2000)
> > libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
> > (0x000014fe2affa000)
> > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
> > (0x000014fe2aeab000)
> > libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1
> > (0x000014fe2ae7f000)
> > libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1
> > (0x000014fe2ae3d000)
> > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
> > (0x000014fe2ac49000)
> > /lib64/ld-linux-x86-64.so.2 (0x000014fe2d4d3000)
> > libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
> > (0x000014fe2abff000)
> > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> > (0x000014fe2abe4000)
> > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> > (0x000014fe2abde000)
> >
> > And here is the stacking fault (it doesn't tell me anything):
> > $x lapw1
> >
> > Program received signal SIGSEGV: Segmentation fault - invalid memory
> > reference.
> >
> > Backtrace for this error:
> > #0 0x148fb48b8d21 in ???
> > #1 0x148fb48b7ef5 in ???
> > #2 0x148fb452d20f in ???
> > #3 0x148fb54495fb in ???
> > Segmentation fault
> > 0.949u 0.472s 0:00.84 167.8% 0+0k 14400+10992io 31pf+0w
> >
> > Any idea?
> >
> > With the settings I reported yesterday, everything works just fine
> > (though probably not very efficiently - but this is not a problem for
> > me as this binary is not a "production" binary on any HPC):
> >
> > $ ldd lapw1
> > linux-vdso.so.1 (0x00007ffc60765000)
> > libblas.so.3 => /lib/x86_64-linux-gnu/libblas.so.3
> > (0x0000150cec90f000)
> > liblapack.so.3 => /lib/x86_64-linux-gnu/liblapack.so.3
> > (0x0000150cec26b000)
> > libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
> > (0x0000150cec248000)
> > libgfortran.so.5 => /lib/x86_64-linux-gnu/libgfortran.so.5
> > (0x0000150cebf80000)
> > libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
> > (0x0000150cebe31000)
> > libmvec.so.1 => /lib/x86_64-linux-gnu/libmvec.so.1
> > (0x0000150cebe05000)
> > libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1
> > (0x0000150cebdc1000)
> > libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
> > (0x0000150cebbcf000)
> > libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
> > (0x0000150cebbb4000)
> > /lib64/ld-linux-x86-64.so.2 (0x0000150cec9fc000)
> > libquadmath.so.0 => /lib/x86_64-linux-gnu/libquadmath.so.0
> > (0x0000150cebb6a000)
> > libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
> > (0x0000150cebb64000)
> >
> > and:
> >
> > $ x lapw1
> > STOP LAPW1 END
> > 162.045u 0.918s 2:31.25 107.7% 0+0k 8976+37856io 35pf+0w
> > $ grep HORB *output1*
> > test_case.output1: TIME HAMILT (CPU) = 16.3, HNS = 20.0,
> > HORB = 0.0, DIAG = 125.9, SYNC = 0.0
> > test_case.output1: TIME HAMILT (WALL) = 4.4, HNS = 20.0,
> > HORB = 0.0, DIAG = 126.0, SYNC = 0.0
> >
> > (I am using it on my 10-year old "type writer":
> > $lscpu
> > Architecture: x86_64
> > CPU op-mode(s): 32-bit, 64-bit
> > Byte Order: Little Endian
> > Vendor ID: GenuineIntel
> > CPU family: 6
> > Model: 26
> > Model name: Intel(R) Xeon(R) CPU W3550
> > @ 3.07GHz
> > )
> >
> >
> > ---
> > Dr David Holec
> > Computational Materials Science group
> > Department of Materials Science
> > Montanuniversität Leoben
> >
> >
> >
> > Franz-Josef-Strasse 18, A-8700 Leoben, Austria
> > tel. +43-(0)3842-4024211
> > fax. +43-(0)3842-4024202
> > materials.unileoben.ac.at
> > cms.unileoben.ac.at
> > ________________________________
> > WHERE RESEARCH MEETS FUTURE
> >
> >
> > On Wed, 24 Nov 2021 at 08:27, Pavel Ondračka
> > <pavel.ondra...@email.cz> wrote:
> > > Hi David,
> > >
> > > as you said it works for you, so feel free to ignore, but I have
> > > some
> > > further tips if you are interested. Ubuntu switches between the
> > > different blas and lapack using the "alternatives", so its
> > > difficult
> > > to
> > > say if you actually link with the correct one.
> > >
> > > "ldd lapw1" in WIENROOT should show which one is actually linked,
> > > what
> > > you want to have is the openmp openblas
> > > /usr/lib/x86_64-linux-gnu/openblas-openmp/libblas.so
> > > /usr/lib/x86_64-linux-gnu/openblas-openmp/liblapack.so
> > > or alternatively
> > > /usr/lib/x86_64-linux-gnu/openblas-openmp/libopenblas.so
> > > It looks like you linked with the pthread one. This is not a
> > > problem
> > > when running at single thread but at higher thread number this
> > > might
> > > lead to oversubscription and slowdowns as the pthreaded openblas
> > > doesn't respect the OMP_NUM_THREADS set by Wien2k. So I would
> > > recommend
> > > to relink with the openmp OpenBLAS. BTW it is usually safer to link
> > > with OpenBLAS explicitly using the -lopenblas instead of the -
> > > llapack
> > > -
> > > lblas to be sure you don't accidentally link the netlib one
> > > (libopenblas is just the libblas and libblapack provided by
> > > OpenBLAS
> > > merged together).
> > >
> > > In general easy way how to check performance is to run the serial
> > > test_case from http://www.wien2k.at/reg_user/benchmark/ On modern
> > > CPUs
> > > (at least avx2) the runtime should be around 15-25 seconds at
> > > single
> > > thread.
> > >
> > > I see total runtime of ~18seconds on Fedora 35 with gfortran 11.2.1
> > > OpenBLAS and AMD Ryzen 9 3900X 12-Core Processor.
> > > Also look for the following line in test_case.output1, this is what
> > > I
> > > have:
> > > TIME HAMILT (WALL) = 2.2, HNS = 1.7, HORB = 0.0, DIAG
> > > =
> > >
> > > 14.0, SYNC = 0.0
> > > The time in HAMILT mostly depends on you compiler and vectorizing
> > > settings, while the DIAG is 99% lapack/blas related, so this can
> > > help
> > > with the diagnostics if things are slow.
> > >
> > > You might also get extra speedup of the HAMILT part by adding "-
> > > DHAVE_LIBMVEC" to the Compiler options.
> > >
> > > Best regards
> > > Pavel
> > >
> > > On Tue, 2021-11-23 at 11:07 +0100, David Holec wrote:
> > > > Dear all,
> > > >
> > > > I have just spent some time making Wien2k run on my single
> > > > machine
> > > > running Ubuntu 20.04 with gfortran/gcc. Since I am not an expert,
> > > it
> > > > was a trial and error, but it seems that I found a working
> > > combination
> > > > (sadly, the default parameters didn't work for me). Maybe this
> > > > will
> > > > help someone. Here are the settings that did the job for me:
> > > >
> > > > M OpenMP switch: -fopenmp
> > > > O Compiler options: -ffree-form -O2 -ftree-vectorize -
> > > > march=native -ffree-line-length-none -ffpe-summary=none
> > > > L Linker Flags: $(FOPT) -L/usr/lib/x86_64-linux-
> > > > gnu
> > > > P Preprocessor flags '-DParallel'
> > > > R R_LIBS (LAPACK+BLAS): -lblas -llapack -lpthread
> > > > F FFTW options: -DFFTW3 -DFFTW_OMP -I/usr/include
> > > > FFTW-LIBS: -L/usr/lib/x86_64-linux-gnu -
> > > > lfftw3
> > > -
> > > > lfftw3_omp
> > > >
> > > > where the FFTW options were:
> > > >
> > > > R FFTWROOT: /usr/
> > > > V FFTW_VERSION: FFTW3
> > > > L FFTW_LIB: lib/x86_64-linux-gnu
> > > > N FFTW_LIBNAME: fftw3
> > > >
> > > > Compiler versions:
> > > > $ gcc --version
> > > > gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> > > > gfortran --version
> > > > GNU Fortran (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
> > > >
> > > > And I used the generic lapack and openblas packages provides by
> > > Ubuntu
> > > > repos:
> > > > liblapack-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > > > liblapack3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > >
> > > > liblapack64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > > liblapack64-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > > >
> > > > libblas-dev/focal,now 3.9.0-1build1 amd64 [installed]
> > > > libblas3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > > libblas64-3/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > > libblas64-dev/focal,now 3.9.0-1build1 amd64 [installed,automatic]
> > > >
> > > > libopenblas64-0/focal-updates,now 0.3.8+ds-1ubuntu0.20.04.1 amd64
> > > > [installed]
> > > > libopenblas64-0-openmp/focal-updates,now 0.3.8+ds-
> > > > 1ubuntu0.20.04.1
> > > > amd64 [installed]
> > > > libopenblas64-0-pthread/focal-updates,now 0.3.8+ds-
> > > > 1ubuntu0.20.04.1
> > > > amd64 [installed,automatic]
> > > >
> > > > (I am not totally sure if I need all the libraries above, but
> > > > certainly, with these, the compilation seems to work and I am
> > > > able
> > > to
> > > > run SCF cycles & Telnes calculations without errors :-)
> > > >
> > > > All the best,
> > > > David
> > > > ---
> > > > Dr David Holec
> > > > Computational Materials Science group
> > > > Department of Materials Science
> > > > Montanuniversität Leoben
> > > >
> > > >
> > > >
> > > > Franz-Josef-Strasse 18, A-8700 Leoben, Austria
> > > > tel. +43-(0)3842-4024211
> > > > fax. +43-(0)3842-4024202
> > > > materials.unileoben.ac.at
> > > > cms.unileoben.ac.at
> > > > ________________________________
> > > > WHERE RESEARCH MEETS FUTURE
> > > > _______________________________________________
> > > > Wien mailing list
> > > > Wien@zeus.theochem.tuwien.ac.at
> > > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > > > SEARCH the MAILING-LIST at:
> > > >
> > > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> > >
> > > _______________________________________________
> > > Wien mailing list
> > > Wien@zeus.theochem.tuwien.ac.at
> > > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > > SEARCH the MAILING-LIST at:
> > > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
> > _______________________________________________
> > Wien mailing list
> > Wien@zeus.theochem.tuwien.ac.at
> > http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> > SEARCH the MAILING-LIST at:
> > http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
>
> _______________________________________________
> Wien mailing list
> Wien@zeus.theochem.tuwien.ac.at
> http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
> SEARCH the MAILING-LIST at:
> http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
>
_______________________________________________
Wien mailing list
Wien@zeus.theochem.tuwien.ac.at
http://zeus.theochem.tuwien.ac.at/mailman/listinfo/wien
SEARCH the MAILING-LIST at:
http://www.mail-archive.com/wien@zeus.theochem.tuwien.ac.at/index.html
