Re: [petsc-users] Question on MatMatmult
Ah ok, Then I will have a look at matconvert. And then maybe later switch to AIJ as well. Thanks of the help, Frank > On 29 May 2024, at 16:57, Barry Smith wrote: > > > You can use MatConvert() > > >> On May 29, 2024, at 10:53 AM, Frank Bramkamp wrote: >> >> This Message Is From an External Sender >> This message came from outside your organization. >> Hello Hong, >> >> Thank you for the clarification. >> If I already have a BAIJ matrix format, can I then convert it later into AIJ >> format as well ?! >> In that case I would have two matrices, but that would be ok for testing. >> I think that you sometimes convert different matrix formats into each other >> ?! >> >> >> Since I typically have BAIJ format, I also use a blocked ILU, which would >> turn into a point wise ILU >> for an AIJ matrix. That is why I typically have the BAIJ format. >> >> Otherwise, I have to change it into an AIJ format from the beginning. >> >> >> Thanks for the quick help, >> >> Frank >> >> >> >> >
Re: [petsc-users] Question on MatMatmult
Hello Hong, Thank you for the clarification. If I already have a BAIJ matrix format, can I then convert it later into AIJ format as well ?! In that case I would have two matrices, but that would be ok for testing. I think that you sometimes ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Hello Hong, Thank you for the clarification. If I already have a BAIJ matrix format, can I then convert it later into AIJ format as well ?! In that case I would have two matrices, but that would be ok for testing. I think that you sometimes convert different matrix formats into each other ?! Since I typically have BAIJ format, I also use a blocked ILU, which would turn into a point wise ILU for an AIJ matrix. That is why I typically have the BAIJ format. Otherwise, I have to change it into an AIJ format from the beginning. Thanks for the quick help, Frank
[petsc-users] Question on MatMatmult
Dear PETSc Team, I would like to make a matrix-matrix product of two matrices. I try to use CALL MatMatMult(Mat_A,MAT_B,MAT_INITIAL_MATRIX,PETSC_DEFAULT_REAL,MAT_AB,IERROR). // calling from fortran When I try to use this function I get the ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Dear PETSc Team, I would like to make a matrix-matrix product of two matrices. I try to use CALL MatMatMult(Mat_A,MAT_B,MAT_INITIAL_MATRIX,PETSC_DEFAULT_REAL,MAT_AB,IERROR). // calling from fortran When I try to use this function I get the following error message: "Unspecified symbolic phase for product AB with A seqbaij, B seqbaij. The product is not supported” I am using the seqbaij matrix format. Is MatMatMult and MatProductSymbolic only defined for the standard point-wise matrix format but not for a blocked format ?! In the documentation, I could not see a hint on supported matrix formats or any limitations. The examples also just use a point-wise format (AIJ), as I can see so far. Greetings, Frank Bramkamp
Re: [petsc-users] Problem with NVIDIA compiler and OpenACC
Dear Barry, That looks very good now. The -lnvc is gone now. I also tested my small fortran program. There I can see that libnvc is automatically added as well, but this time is comes after the libaccdevice. so. library for openacc. And then ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Dear Barry, That looks very good now. The -lnvc is gone now. I also tested my small fortran program. There I can see that libnvc is automatically added as well, but this time is comes after the libaccdevice.so. library for openacc. And then my openacc commands also work again. I also mentioned some issues with some cuda nvJitlink library. I just found out that some path in our cuda compiler module was not set correctly. I will try to compile it with cuda again as well. We just start to get PETSC on GPUs with the cuda backend, and I start with openccc for our fortran code to get first experience how everything works with GPU porting. Good that you could fix the issue. Thanks for the great help. Have a nice weekend, Frank Bramkamp
Re: [petsc-users] Problem with NVIDIA compiler and OpenACC
Thanks for effort, Barry. I will get it and give it another try. Thanks a lot, Frank > On 5 Apr 2024, at 15: 56, Barry Smith wrote: > > > There was a bug in my attempted fix so it actually did not skip the ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for effort, Barry. I will get it and give it another try. Thanks a lot, Frank > On 5 Apr 2024, at 15:56, Barry Smith wrote: > > > There was a bug in my attempted fix so it actually did not skip the option. > > Try git pull and then run configure again. > > >> On Apr 5, 2024, at 6:30 AM, Frank Bramkamp wrote: >> >> Dear Barry, >> >> I tried your fix for -lnvc. Unfortunately it did not work so far. >> Here I send you the configure.log file again. >> >> One can see that you try to skip something, but later it still always includes -lnvc for the linker. >> In the file petscvariables it also appears as before. >> >> As I see it, it lists the linker options including -lnvc also before you try to skip it. >> Maybe it is already in the linker options before the skipping. >> >> >> Greetings, Frank >> >> >> >
Re: [petsc-users] Problem with NVIDIA compiler and OpenACC
Thanks for the response, My code is in fortran. I will try to explicitly set LIBS=. . as you suggested. At the moment I skip cuda, but later I also want to use cuda as well. Barry also tried to skip the “-lnvc”, but that did not work yet. Thanks ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Thanks for the response, My code is in fortran. I will try to explicitly set LIBS=.. as you suggested. At the moment I skip cuda, but later I also want to use cuda as well. Barry also tried to skip the “-lnvc”, but that did not work yet. Thanks a lot for the suggestions, Frank
Re: [petsc-users] Problem with NVIDIA compiler and OpenACC
Ok, I will have a look. It is already evening here in Sweden, so it might take until tomorrow. Thanks Frank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Ok, I will have a look. It is already evening here in Sweden, so it might take until tomorrow. Thanks Frank
Re: [petsc-users] Problem with NVIDIA compiler and OpenACC
Ok, I will look for the config. log file. Frank ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Ok, I will look for the config.log file. Frank
Re: [petsc-users] Problem with NVIDIA compiler and OpenACC
Thanks for the reply, Do you know if you actively include the libnvc library ?! Or is this somehow automatically included ?! Greetings, Frank > On 4 Apr 2024, at 15:56, Satish Balay wrote: > > > On Thu, 4 Apr 2024, Frank Bramkamp wrote: > >> Dear PETSC Team, >> >> I found the following problem: >> I compile petsc 3.20.5 with Nvidia compiler 23.7. >> >> >> I use a pretty standard configuration, including >> >> --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpifort COPTFLAGS="-O2 -g" >> CXXOPTFLAGS="-O2 -g" FOPTFLAGS="-O2 -g" --with-debugging=0 --with-log=1 >> --download-fblaslapack --with-cuda=0 >> >> I exclude cuda, since I was not sure if the problem was cuda related. > > Can you try using (to exclude cuda): --with-cudac=0 > >> >> >> The problem is now, if I have s simple fortran program where I link the >> petsc library, but I actually do not use petsc in that program >> (Just for testing). I want to use OpenACC directives in my program, e.g. >> !$acc parallel loop . >> The problem is now, as soon I link with the petsc library, the openacc >> commands do not work anymore. >> It seems that openacc is not initialised and hence it cannot find a GPU. >> >> The problem seems that you link with -lnvc. >> In “petscvariables” => PETSC_WITH_EXTERNAL_LIB you include “-lnvc”. >> If I take this out, then openacc works. With “-lnvc” something gets messed >> up. >> >> The problem is also discussed here: >> https://urldefense.us/v3/__https://forums.developer.nvidia.com/t/failed-cuda-device-detection-when-explicitly-linking-libnvc/203225/1__;!!G_uCfscf7eWS!dlXNyKBzSbximQ13OXxwO506OF71yRM_H5KEnarqXE75D6Vg-ePZr2u6SJ5V3YpRETatvb9pMOUVmpyN0-19SFlbug$ >> >> <https://urldefense.us/v3/__https://forums.developer.nvidia.com/t/failed-cuda-device-detection-when-explicitly-linking-libnvc/203225/1__;!!G_uCfscf7eWS!dlXNyKBzSbximQ13OXxwO506OF71yRM_H5KEnarqXE75D6Vg-ePZr2u6SJ5V3YpRETatvb9pMOUVmpyN0-19SFlbug$><https://urldefense.us/v3/__https://forums.developer.nvidia.com/t/failed-cuda-device-detection-when-explicitly-linking-libnvc/203225/1__;!!G_uCfscf7eWS!dlXNyKBzSbximQ13OXxwO506OF71yRM_H5KEnarqXE75D6Vg-ePZr2u6SJ5V3YpRETatvb9pMOUVmpyN0-19SFlbug$ >> >> <https://urldefense.us/v3/__https://forums.developer.nvidia.com/t/failed-cuda-device-detection-when-explicitly-linking-libnvc/203225/1__;!!G_uCfscf7eWS!dlXNyKBzSbximQ13OXxwO506OF71yRM_H5KEnarqXE75D6Vg-ePZr2u6SJ5V3YpRETatvb9pMOUVmpyN0-19SFlbug$> >> > >> >> My understanding is that libnvc is more a runtime library that does not need >> to be included by the linker. >> Not sure if there is a specific reason to include libnvc (I am not so >> familiar what this library does). >> >> If I take out -lnvc from “petscvariables”, then my program with openacc >> works as expected. I did not try any more realistic program that includes >> petsc. >> >> >> >> 2) >> When compiling petsc with cuda support, I also found that in the petsc >> library the library libnvJitLink.so.12 >> Is not found. On my system this library is in $CUDA_ROOT/lib64 >> I am not sure where this library is on your system ?! > > Hm - good if you can send configure.log for this. configure attempts '$CC -v' > to determine the link libraries to get c/c++/fortran compatibility libraries. > But it can grab other libraries that the compilers are using internally here. > > To avoid this - you can explicitly list these libraries to configure. For ex: > for gcc/g++/gfortran > > ./configure CC=gcc CXX=g++ FC=gfortran LIBS="-lgfortran -lstdc++" > > Satish > >> >> >> Thanks a lot, Frank Bramkamp
[petsc-users] Problem with NVIDIA compiler and OpenACC
Dear PETSC Team, I found the following problem: I compile petsc 3.20.5 with Nvidia compiler 23.7. I use a pretty standard configuration, including --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpifort COPTFLAGS="-O2 -g" CXXOPTFLAGS="-O2 -g" FOPTFLAGS="-O2 -g" --with-debugging=0 --with-log=1 --download-fblaslapack --with-cuda=0 I exclude cuda, since I was not sure if the problem was cuda related. The problem is now, if I have s simple fortran program where I link the petsc library, but I actually do not use petsc in that program (Just for testing). I want to use OpenACC directives in my program, e.g. !$acc parallel loop . The problem is now, as soon I link with the petsc library, the openacc commands do not work anymore. It seems that openacc is not initialised and hence it cannot find a GPU. The problem seems that you link with -lnvc. In “petscvariables” => PETSC_WITH_EXTERNAL_LIB you include “-lnvc”. If I take this out, then openacc works. With “-lnvc” something gets messed up. The problem is also discussed here: https://urldefense.us/v3/__https://forums.developer.nvidia.com/t/failed-cuda-device-detection-when-explicitly-linking-libnvc/203225/1__;!!G_uCfscf7eWS!dlXNyKBzSbximQ13OXxwO506OF71yRM_H5KEnarqXE75D6Vg-ePZr2u6SJ5V3YpRETatvb9pMOUVmpyN0-19SFlbug$ <https://urldefense.us/v3/__https://forums.developer.nvidia.com/t/failed-cuda-device-detection-when-explicitly-linking-libnvc/203225/1__;!!G_uCfscf7eWS!dlXNyKBzSbximQ13OXxwO506OF71yRM_H5KEnarqXE75D6Vg-ePZr2u6SJ5V3YpRETatvb9pMOUVmpyN0-19SFlbug$ > My understanding is that libnvc is more a runtime library that does not need to be included by the linker. Not sure if there is a specific reason to include libnvc (I am not so familiar what this library does). If I take out -lnvc from “petscvariables”, then my program with openacc works as expected. I did not try any more realistic program that includes petsc. 2) When compiling petsc with cuda support, I also found that in the petsc library the library libnvJitLink.so.12 Is not found. On my system this library is in $CUDA_ROOT/lib64 I am not sure where this library is on your system ?! Thanks a lot, Frank Bramkamp
[petsc-users] MATSETVALUES: Fortran problem
Dear PETSc Team, I am using the latest petsc version 3. 20. 5. I would like to create a matrix using MatCreateSeqAIJ To insert values, I use MatSetValues. It seems that the Fortran interface/stubs are missing for MatsetValues, as the linker does ZjQcmQRYFpfptBannerStart This Message Is From an External Sender This message came from outside your organization. ZjQcmQRYFpfptBannerEnd Dear PETSc Team, I am using the latest petsc version 3.20.5. I would like to create a matrix using MatCreateSeqAIJ To insert values, I use MatSetValues. It seems that the Fortran interface/stubs are missing for MatsetValues, as the linker does not find any subroutine with that name. MatSetValueLocal seems to be fine. Typically I am using a blocked matrix format (BAIJ), which works fine in fortran. Soon we want to try PETSC on GPUs, using the format MATAIJCUSPARSE, since there seems not to be a blocked format available in PETSC for GPUs so far. Therefore I first want to try the pointwise format MatCreateSeqAIJ format on a CPU, before using the GPU format. I think that CUDA also supports a block format now ?! Maybe that would be also useful to have one day. Greetings, Frank Bramkamp
[petsc-users] Fortran problem MatGetValuesLocal
Dear PETSc team, We are using the latest petsc version 3.20.1, intel compiler 2023, and we found the following problem: We want to call the function MatGetValuesLocal to extract a block sub-matrix from an assembled matrix (e.g. a 5x5 blocked sub matrix). We use the matrix format MatCreateBAIJ in parallel. In particular we try to call MatGetValuesLocal in Fortran. It seems that the linked does not find the subroutine MatGetValuesLocal. The subroutine MatGetValues seems to be fine. I guess that the fortran stubs/fortran interface is missing for this routine. On the documentation side, you also write a note for developers that the fortran stubs and interface Is not automatically generated for MatGetValuesLocal. So maybe that has been forgotten to do. Unfortunately I do not have any small test example, since we just incorporated the function call into our own software. Otherwise I would first have to set a small test example for the parallel case. I think there is also an include file where one can check the fortran interfaces ?! I forgot where to look this up. Greetings, Frank Bramkamp
[petsc-users] KSPAGMRES Question
Dear PETSc team, I have seen that there is the KSP method: KSPAGMRES, https://petsc.org/release/docs/manualpages/KSP/KSPAGMRES.html <https://petsc.org/release/docs/manualpages/KSP/KSPAGMRES.html> I wanted to test this method, as it also seems to reduce the amount of MPI communication, compared to the standard GMRES. I supposed that the class is called “KSPAGMRES”. But in the include files petscksp.h and petsc/finclude/petscksp.h there is no definition for KSPAGMRES, just KSPDGMRES. I wonder if the definition KSPAGMRES is simply missing, or do I have to call DGMRES and set another option for AGMRES ?! The standard GMRES has the problem that MPI_Allreduce gets expensive for 2048 cores. Therefore I wanted to see, if AGMRES has a bit less communication, as this is mentioned in the description of the method. Greetings, Frank Bramkamp
