Hi Paolo, I will have a look more closely to the links you sent me.
I tried to specify the location of fftw3.f03 file by setting INCLUDE_FFTW variable to the location of our FFTW library, compiled with NVHPC. Unfortunately, it didn't work. I have checked the m4 macro from the QE-7.0 used to detect the location of FFTW, x_ac_qe_fft.m4, and compared it with the tip of the develop branch on gitlab. The only thing that differs is the FFLAGS variable, -ffree-form option is added (FFLAGS="$orig_fflags -I$inc -ffree-form"). This means that either Gitlab doesn't provide the latest development version or that the improved FFTW3 auto-detection macro is available in the 7.0 release tarball. I have added the location of FFTW include directory directly to make.inc. This helped to continue the compilation process until: mpif90 -mp -cuda -gpu=cc80,cuda11.0 -acc -o pw.x \ pwscf.o libpw.a ../../KS_Solvers/libks_solvers.a ../../dft-d3/libdftd3qe.a /mnt/tier2/users/apps/scratchpad/qe-7.0//Modules/libqemod.a /mnt/tier2/users/apps/scratchpad/qe-7.0//upflib/libupf.a /mnt/tier2/users/apps/scratchpad/qe-7.0//XClib/xc_lib.a /mnt/tier2/users/apps/scratchpad/qe-7.0//FFTXlib/libqefft.a /mnt/tier2/users/apps/scratchpad/qe-7.0//LAXlib/libqela.a /mnt/tier2/users/apps/scratchpad/qe-7.0//UtilXlib/libutil.a /mnt/tier2/users/apps/scratchpad/qe-7.0//MBD/libmbd.a -cudalib=cufft,cublas,cusolver,curand /mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src/libdevXlib.a -L/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -ldevXlib -L/apps/USE/easybuild/staging/2021.1/software/NVHPC/22.3/Linux_x86_64/22.3/compilers/lib -llapack -L/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/lib -lFoX_dom -lFoX_sax -lFoX_wxml -lFoX_common -lFoX_utils -lFoX_fsys /apps/USE/easybuild/staging/2021.1/software/FFTW/3.3.9-nvompic-2021a/lib -lfftw3_omp -lfftw3 -L/apps/USE/easybuild/staging/2021.1/software/NVHPC/22.3/Linux_x86_64/22.3/compilers/lib -lblas /apps/USE/easybuild/staging/2021.1/software/binutils/2.36.1-GCCcore-10.3.0/bin/ld: read in flex scanner failed As you can see, there I reach another issue. I need to search through the users mailing list to see if somebody else experienced something similar. In case you have some suggestions, I'm happy to hear them. Best regards, Robert -----Original Message----- From: Paolo Giannozzi <[email protected]> Sent: Monday, 23 May 2022 21:46 To: Quantum ESPRESSO users Forum <[email protected]>; Robert MIJAKOVIC <[email protected]> Subject: Re: [QE-users] [QE 7.0/GPU] Build configured with Autotools fails Here some explanations: https://www.mail-archive.com/[email protected]/msg41494.html https://www.mail-archive.com/[email protected]/msg40215.html The path to the fftw3.f03 file, needed by fftw3, must be specified in INCLUDE_FFTW or similar variables. The latest development version (soon to be released) has somewhat better FFTW3 auto-detection Paolo On 23/05/2022 17:55, Robert MIJAKOVIC wrote: > # Summary > > QE 7.0/GPU compilation with Autotools fails with "Can't find include > file fftw3.f03 (fft_scalar.FFTW3.f90: 40)" > > # Version > > qe-7.0-ReleasePack.tgz > > # Environment > > ## Hardware > > 1. 2xAMD EPYC 7452 > > 2. 4xNVIDIA A100 > > 3. 512 GB RAM > > ## Software > > 1. OS: Rocky Linux release 8.5 (Green Obsidian) > > 2. NVHPC 22.3 > > 3. OpenMPI 4.1.3 built with NVHPC 22.3 > > 4. CUDA 11.3.1 with Driver 470.82.01 > > 5. libxc 5.1.5 > > 6. Autotools 20210128 > > # Steps to reproduce > > ## Configured with: > > `./configure --prefix=$HOME/install/qe-7.0 --enable-openmp > --with-scalapack=no --enable-cuda-env-check=no > --with-cuda=$EBROOTNVHPC/Linux_x86_64/22.3/cuda/11.0 > --with-cuda-runtime=11.0 --with-cuda-cc=80` > > ## Prebuild options > > `cp $EBROOTLIBXC/include/*.mod Modules/mod/qe_modules && export > FPP='nvfortran -Mpreprocess -E' && export CPP='cpp -E' && export > FCPP='cpp -E' && ` > > ## make options > > `make all epw` > > # Observed behavior > > Compilation fails with: > > ``` > > make[1]: Entering directory > '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib' > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_param.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_support.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > stick_base.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_types.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > scatter_mod.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_scatter_2d.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_buffers.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_scatter_gpu.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_scatter_2d_gpu.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_ggen.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_scalar.DFTI.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_scalar.ESSL.f90 > > NVFORTRAN-W-0006-Input file empty (fft_scalar.ESSL.f90) > > NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fftw_interfaces.f90 > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_scalar.FFTW.f90 > > NVFORTRAN-W-0006-Input file empty (fft_scalar.FFTW.f90) > > NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings > > mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI > -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0 > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include > -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include > -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c > fft_scalar.FFTW3.f90 > > NVFORTRAN-F-0906-Can't find include file fftw3.f03 > (fft_scalar.FFTW3.f90: 40) > > NVFORTRAN/x86-64 Linux 22.3-0: compilation aborted > > make[1]: *** [../make.inc:16: fft_scalar.FFTW3.o] Error 2 > > make[1]: Leaving directory '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib' > > make: *** [Makefile:196: libfft] Error 1 > > ``` > > # Questions > > 1. What do I do wrong? > > 2. Why configure didn't fail if FFTW is mandatory? > > 3. Is it possible to configure the build system to use cuFFTinstead? > > Dr. rer. nat. Robert Mijaković | HPC System Software Architect > > > *Lux**Provide* > 3, Op der Poukewiss | L-7795 Bissen > > Grand-Duchy of Luxembourg > M (+352) 691 396 474 > > [email protected] > <mailto:[email protected]> |www.luxprovide.lu > <http://www.luxprovide.lu/> > > > _______________________________________________ > The Quantum ESPRESSO community stands by the Ukrainian people and > expresses its concerns about the devastating effects that the Russian > military offensive has on their country and on the free and peaceful > scientific, cultural, and economic cooperation amongst peoples > _______________________________________________ > Quantum ESPRESSO is supported by MaX (www.max-centre.eu) users mailing > list [email protected] > https://lists.quantum-espresso.org/mailman/listinfo/users -- Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche, Univ. Udine, via delle Scienze 208, 33100 Udine, Italy Phone +39-0432-558216, fax +39-0432-558222 _______________________________________________ The Quantum ESPRESSO community stands by the Ukrainian people and expresses its concerns about the devastating effects that the Russian military offensive has on their country and on the free and peaceful scientific, cultural, and economic cooperation amongst peoples _______________________________________________ Quantum ESPRESSO is supported by MaX (www.max-centre.eu) users mailing list [email protected] https://lists.quantum-espresso.org/mailman/listinfo/users
