Ok so, i went back and erased the old libpetsc.so.3 i think it was the one causing problems, i had --with-shared-libraries=0 and the installation complained of not having that file, then reinstalled with --with-shared-libraries=1 and it is finally recognizing my system installation with only CUDA, but now it gives a 11 SEGV violation error:
[valera@node50 alone]$ ./linsolve laplacian.petsc ! TrivSoln loaded, size: 125 / 125 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range [0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors [0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run [0]PETSC ERROR: to get more information on the crash. [0]PETSC ERROR: --------------------- Error Message -------------------------------------------------------------- [0]PETSC ERROR: Signal received [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting. [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-2027-g045eeab GIT Date: 2018-03-12 13:30:25 -0500 [0]PETSC ERROR: ./linsolve on a named node50 by valera Wed Mar 14 16:17:34 2018 [0]PETSC ERROR: Configure options --prefix=/usr/local/petsc.cod/petsc-install --with-mpi-dir=/usr/lib64/openmpi --with-blaslapack-dir=/usr/lib64 COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=1 --with-debugging=0 --with-cuda=1 --with-cuda-arch=sm_60 [0]PETSC ERROR: #1 User provided function() line 0 in unknown file -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 59. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- [valera@node50 alone]$ Thanks, On Wed, Mar 14, 2018 at 1:52 PM, Matthew Knepley <knep...@gmail.com> wrote: > On Thu, Mar 15, 2018 at 4:01 AM, Manuel Valera <mvaler...@mail.sdsu.edu> > wrote: > >> Ok well, it turns out the $PETSC_DIR points to the testpetsc directory, >> and it makes, install and tests without problems (only a problem on ex5f) >> but trying to reconfigure on valera/petsc directory asks me to change the >> $PETSC_DIR variable, >> > > Yes, you must set PETSC_DIR to point to the installation you want to use. > > Matt > > >> Meanwhile the system installation still points to the valera/petsc/cuda >> build, >> >> should i just delete the petsc installation folder and start over? >> >> Thanks, >> >> On Wed, Mar 14, 2018 at 11:36 AM, Matthew Knepley <knep...@gmail.com> >> wrote: >> >>> On Thu, Mar 15, 2018 at 3:25 AM, Manuel Valera <mvaler...@mail.sdsu.edu> >>> wrote: >>> >>>> yeah that worked, >>>> >>>> [valera@node50 tutorials]$ ./ex19 -dm_vec_type seqcuda -dm_mat_type >>>> seqaijcusparse >>>> lid velocity = 0.0625, prandtl # = 1., grashof # = 1. >>>> Number of SNES iterations = 2 >>>> [valera@node50 tutorials]$ >>>> >>>> How do i make sure the other program refer to this installation? using >>>> the same arguments there i get: >>>> >>>> [valera@node50 alone]$ ./linsolve -vec_type seqcuda -mat_type >>>> seqaijcusparse >>>> laplacian.petsc ! >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing >>>> package: http://www.mcs.anl.gov/petsc/documentation/installation.html >>>> #external >>>> [0]PETSC ERROR: Unknown vector type: seqcuda >>>> >>> >>> This PETSc has not been configured with CUDA. It is located in >>> /home/valera/petsc. >>> The other >>> one you used is located in /home/valera/testpetsc. It does not make >>> much sense to me that >>> it does not understand CUDA since it says the configure arguments had >>> --with-cuda=1. There >>> must have been a build problem. Rebuild >>> >>> cd $PETSC_DIR >>> make all >>> >>> If you still have a problem, reconfigure >>> >>> cd $PETSC_DIR >>> ./cuda/lib/petsc/conf/reconfigure-cuda.py >>> make all >>> >>> If that still fails, then something very bizarre is happening on your >>> machine and we will have >>> to exchange more mail. >>> >>> Matt >>> >>> >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed Mar 14 >>>> 11:25:11 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>> [0]PETSC ERROR: #1 VecSetType() line 42 in >>>> /home/valera/petsc/src/vec/vec/interface/vecreg.c >>>> [0]PETSC ERROR: #2 VecSetTypeFromOptions_Private() line 1241 in >>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>> [0]PETSC ERROR: #3 VecSetFromOptions() line 1276 in >>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Vec object's type is not set: Argument # 1 >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed Mar 14 >>>> 11:25:11 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>> [0]PETSC ERROR: #4 VecDuplicate() line 375 in >>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>> [0]PETSC ERROR: --------------------- Error Message >>>> -------------------------------------------------------------- >>>> [0]PETSC ERROR: Object is in wrong state >>>> [0]PETSC ERROR: Vec object's type is not set: Argument # 1 >>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>> for trouble shooting. >>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed Mar 14 >>>> 11:25:11 2018 >>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda --with-cc=mpicc >>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 --CXXOPTFLAGS=-O3 >>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 --with-cuda=1 >>>> --with-cuda-arch=sm_60 --with-cusp=1 --with-cusp-dir=/home/valera/cusp >>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>> [0]PETSC ERROR: #5 VecDuplicate() line 375 in >>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>> [0]PETSC ERROR: #6 User provided function() line 0 in User file >>>> [valera@node50 alone]$ >>>> >>>> I made sure there is a call for Vec/MatSetFromOptions() there, i am >>>> loading the matrix from a petsc binary in this case, >>>> >>>> Thanks, >>>> >>>> >>>> On Wed, Mar 14, 2018 at 11:22 AM, Matthew Knepley <knep...@gmail.com> >>>> wrote: >>>> >>>>> On Thu, Mar 15, 2018 at 3:19 AM, Manuel Valera < >>>>> mvaler...@mail.sdsu.edu> wrote: >>>>> >>>>>> Yes, this is the system installation that is being correctly linked >>>>>> (the linear solver and model are not linking the correct installation idk >>>>>> why yet) i configured with only CUDA this time because of the message >>>>>> Karl >>>>>> Rupp posted on my installation thread, where he says only one type of >>>>>> library will work at a time, anyway this is what i got: >>>>>> >>>>>> [valera@node50 tutorials]$ ./ex19 -dm_vec_type seqcuda -dm_mat_type >>>>>> seqaijcusp >>>>>> lid velocity = 0.0625, prandtl # = 1., grashof # = 1. >>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>> -------------------------------------------------------------- >>>>>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing >>>>>> package: http://www.mcs.anl.gov/petsc/documentation/installation.html >>>>>> #external >>>>>> [0]PETSC ERROR: Unknown Mat type given: seqaijcusp >>>>>> >>>>> >>>>> It is telling you the problem. Use >>>>> >>>>> -dm_mat_type seqaijcusparse >>>>> >>>>> Matt >>>>> >>>>> >>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>> ocumentation/faq.html for trouble shooting. >>>>>> [0]PETSC ERROR: Petsc Development GIT revision: v3.8.3-2027-g045eeab >>>>>> GIT Date: 2018-03-12 13:30:25 -0500 >>>>>> [0]PETSC ERROR: ./ex19 on a named node50 by valera Wed Mar 14 >>>>>> 11:17:25 2018 >>>>>> [0]PETSC ERROR: Configure options >>>>>> --prefix=/usr/local/petsc.cod/petsc-install >>>>>> --with-mpi-dir=/usr/lib64/openmpi --with-blaslapack-dir=/usr/lib64 >>>>>> COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=0 >>>>>> --download-hypre --with-debugging=0 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>> [0]PETSC ERROR: #1 MatSetType() line 61 in >>>>>> /home/valera/testpetsc/src/mat/interface/matreg.c >>>>>> [0]PETSC ERROR: #2 DMCreateMatrix_DA() line 693 in >>>>>> /home/valera/testpetsc/src/dm/impls/da/fdda.c >>>>>> [0]PETSC ERROR: #3 DMCreateMatrix() line 1199 in >>>>>> /home/valera/testpetsc/src/dm/interface/dm.c >>>>>> [0]PETSC ERROR: #4 SNESSetUpMatrices() line 646 in >>>>>> /home/valera/testpetsc/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #5 SNESSetUp_NEWTONLS() line 296 in >>>>>> /home/valera/testpetsc/src/snes/impls/ls/ls.c >>>>>> [0]PETSC ERROR: #6 SNESSetUp() line 2795 in >>>>>> /home/valera/testpetsc/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #7 SNESSolve() line 4187 in >>>>>> /home/valera/testpetsc/src/snes/interface/snes.c >>>>>> [0]PETSC ERROR: #8 main() line 161 in /home/valera/testpetsc/src/sne >>>>>> s/examples/tutorials/ex19.c >>>>>> [0]PETSC ERROR: PETSc Option Table entries: >>>>>> [0]PETSC ERROR: -dm_mat_type seqaijcusp >>>>>> [0]PETSC ERROR: -dm_vec_type seqcuda >>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>>> entire error message to petsc-ma...@mcs.anl.gov---------- >>>>>> ------------------------------------------------------------ >>>>>> -------------- >>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>> with errorcode 86. >>>>>> >>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>> You may or may not see output from other processes, depending on >>>>>> exactly when Open MPI kills them. >>>>>> ------------------------------------------------------------ >>>>>> -------------- >>>>>> >>>>>> >>>>>> On Wed, Mar 14, 2018 at 11:16 AM, Matthew Knepley <knep...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> On Thu, Mar 15, 2018 at 3:12 AM, Manuel Valera < >>>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>>> >>>>>>>> Thanks, got this error: >>>>>>>> >>>>>>> >>>>>>> Did you not configure with CUSP? It looks like you have CUDA, so use >>>>>>> >>>>>>> -dm_vec_type seqcuda >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Matt >>>>>>> >>>>>>> >>>>>>>> [valera@node50 testpetsc]$ cd src/snes/examples/tutorials/ >>>>>>>> [valera@node50 tutorials]$ PETSC_ARCH="" make ex19 >>>>>>>> /usr/lib64/openmpi/bin/mpicc -o ex19.o -c -Wall -Wwrite-strings >>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>>>>> -fvisibility=hidden -O2 -I/home/valera/testpetsc/include >>>>>>>> -I/home/valera/testpetsc/arch-linux2-c-opt/include >>>>>>>> -I/usr/local/petsc.cod/petsc-install/include >>>>>>>> -I/usr/local/cuda/include -I/usr/lib64/openmpi/include `pwd`/ex19.c >>>>>>>> /usr/lib64/openmpi/bin/mpicc -Wall -Wwrite-strings >>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>>>>> -fvisibility=hidden -O2 -o ex19 ex19.o >>>>>>>> -L/home/valera/testpetsc/arch-linux2-c-opt/lib >>>>>>>> -Wl,-rpath,/usr/local/petsc.cod/petsc-install/lib >>>>>>>> -L/usr/local/petsc.cod/petsc-install/lib -Wl,-rpath,/usr/lib64 >>>>>>>> -L/usr/lib64 -Wl,-rpath,/usr/local/cuda/lib64 >>>>>>>> -L/usr/local/cuda/lib64 -L/usr/lib64/openmpi/lib >>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 >>>>>>>> -Wl,-rpath,/usr/lib64/openmpi/lib -lpetsc -lHYPRE -llapack -lblas >>>>>>>> -lm -lcufft -lcublas -lcudart -lcusparse -lX11 -lstdc++ -ldl >>>>>>>> -lmpi_usempi >>>>>>>> -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath >>>>>>>> -lpthread -lstdc++ -ldl >>>>>>>> /usr/bin/rm -f ex19.o >>>>>>>> [valera@node50 tutorials]$ ./ex19 -dm_vec_type seqcusp >>>>>>>> -dm_mat_type seqaijcusp >>>>>>>> lid velocity = 0.0625, prandtl # = 1., grashof # = 1. >>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>> -------------------------------------------------------------- >>>>>>>> [0]PETSC ERROR: Unknown type. Check for miss-spelling or missing >>>>>>>> package: http://www.mcs.anl.gov/petsc/d >>>>>>>> ocumentation/installation.html#external >>>>>>>> [0]PETSC ERROR: Unknown vector type: seqcusp >>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>> v3.8.3-2027-g045eeab GIT Date: 2018-03-12 13:30:25 -0500 >>>>>>>> [0]PETSC ERROR: ./ex19 on a named node50 by valera Wed Mar 14 >>>>>>>> 11:12:11 2018 >>>>>>>> [0]PETSC ERROR: Configure options >>>>>>>> --prefix=/usr/local/petsc.cod/petsc-install >>>>>>>> --with-mpi-dir=/usr/lib64/openmpi --with-blaslapack-dir=/usr/lib64 >>>>>>>> COPTFLAGS=-O2 CXXOPTFLAGS=-O2 FOPTFLAGS=-O2 --with-shared-libraries=0 >>>>>>>> --download-hypre --with-debugging=0 --with-cuda=1 >>>>>>>> --with-cuda-arch=sm_60 >>>>>>>> [0]PETSC ERROR: #1 VecSetType() line 42 in >>>>>>>> /home/valera/testpetsc/src/vec/vec/interface/vecreg.c >>>>>>>> [0]PETSC ERROR: #2 DMCreateGlobalVector_DA() line 39 in >>>>>>>> /home/valera/testpetsc/src/dm/impls/da/dadist.c >>>>>>>> [0]PETSC ERROR: #3 DMCreateGlobalVector() line 865 in >>>>>>>> /home/valera/testpetsc/src/dm/interface/dm.c >>>>>>>> [0]PETSC ERROR: #4 main() line 158 in /home/valera/testpetsc/src/sne >>>>>>>> s/examples/tutorials/ex19.c >>>>>>>> [0]PETSC ERROR: PETSc Option Table entries: >>>>>>>> [0]PETSC ERROR: -dm_mat_type seqaijcusp >>>>>>>> [0]PETSC ERROR: -dm_vec_type seqcusp >>>>>>>> [0]PETSC ERROR: ----------------End of Error Message -------send >>>>>>>> entire error message to petsc-ma...@mcs.anl.gov---------- >>>>>>>> ------------------------------------------------------------ >>>>>>>> -------------- >>>>>>>> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD >>>>>>>> with errorcode 86. >>>>>>>> >>>>>>>> NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. >>>>>>>> You may or may not see output from other processes, depending on >>>>>>>> exactly when Open MPI kills them. >>>>>>>> ------------------------------------------------------------ >>>>>>>> -------------- >>>>>>>> [valera@node50 tutorials]$ >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Mar 14, 2018 at 11:10 AM, Matthew Knepley < >>>>>>>> knep...@gmail.com> wrote: >>>>>>>> >>>>>>>>> On Thu, Mar 15, 2018 at 2:46 AM, Manuel Valera < >>>>>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>>>>> >>>>>>>>>> Ok lets try that, if i go to /home/valera/testpetsc/arch >>>>>>>>>> -linux2-c-opt/tests/src/snes/examples/tutorials there is >>>>>>>>>> runex19.sh and a lot of other ex19 variantes, but if i run that i >>>>>>>>>> get: >>>>>>>>>> >>>>>>>>> >>>>>>>>> knepley/feature-plex-functionals *$:/PETSc3/petsc/petsc-dev$ >>>>>>>>> pushd src/snes/examples/tutorials/ >>>>>>>>> knepley/feature-plex-functionals >>>>>>>>> *$:/PETSc3/petsc/petsc-dev/src/snes/examples/tutorials$ >>>>>>>>> PETSC_ARCH=arch-master-debug make ex19 >>>>>>>>> knepley/feature-plex-functionals >>>>>>>>> *$:/PETSc3/petsc/petsc-dev/src/snes/examples/tutorials$ >>>>>>>>> ./ex19 -dm_vec_type seqcusp -dm_mat_type seqaijcusp >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Matt >>>>>>>>> >>>>>>>>> >>>>>>>>>> [valera@node50 tutorials]$ ./runex19.sh >>>>>>>>>> not ok snes_tutorials-ex19_1 >>>>>>>>>> # ------------------------------------------------------------ >>>>>>>>>> -------------- >>>>>>>>>> # mpiexec was unable to launch the specified application as it >>>>>>>>>> could not access >>>>>>>>>> # or execute an executable: >>>>>>>>>> # >>>>>>>>>> # Executable: ../ex19 >>>>>>>>>> # Node: node50 >>>>>>>>>> # >>>>>>>>>> # while attempting to start process rank 0. >>>>>>>>>> # ------------------------------------------------------------ >>>>>>>>>> -------------- >>>>>>>>>> # 2 total processes failed to start >>>>>>>>>> ok snes_tutorials-ex19_1 # SKIP Command failed so no diff >>>>>>>>>> >>>>>>>>>> is this the one i should be running ? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Mar 14, 2018 at 10:39 AM, Matthew Knepley < >>>>>>>>>> knep...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Mar 15, 2018 at 2:27 AM, Manuel Valera < >>>>>>>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>>>>>>> >>>>>>>>>>>> Ok thanks Matt, i made a smaller case with only the linear >>>>>>>>>>>> solver and a 25x25 matrix, the error i have in this case is: >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Ah, it appears that not all parts of your problem are taking the >>>>>>>>>>> type options. If you want the >>>>>>>>>>> linear algebra objects to change type, you need to have >>>>>>>>>>> >>>>>>>>>>> VecSetFromOptions() and MatSetFromOptions() >>>>>>>>>>> >>>>>>>>>>> called after you create them, but before sizes are set and data >>>>>>>>>>> is entered. However, it should >>>>>>>>>>> not be possible to have a seq Vec with the seqcusp AXPY routine >>>>>>>>>>> set. Something else is wrong... >>>>>>>>>>> Did you try a PETSc example, such as SNES ex19, with this? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Matt >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> [valera@node50 alone]$ mpirun -n 1 ./linsolve -vec_type cusp >>>>>>>>>>>> -mat_type aijcusparse >>>>>>>>>>>> laplacian.petsc ! >>>>>>>>>>>> TrivSoln loaded, size: 125 / 125 >>>>>>>>>>>> RHS loaded, size: 125 / 125 >>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>> [0]PETSC ERROR: Null argument, when expecting valid pointer >>>>>>>>>>>> [0]PETSC ERROR: Null Pointer: Parameter # 4 >>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed >>>>>>>>>>>> Mar 14 10:24:35 2018 >>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 >>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>> [0]PETSC ERROR: #1 VecSetValues() line 851 in >>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/rvector.c >>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>> -------------------------------------------------------------- >>>>>>>>>>>> [0]PETSC ERROR: Invalid argument >>>>>>>>>>>> [0]PETSC ERROR: Object (seq) is not seqcusp or mpicusp >>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>> [0]PETSC ERROR: ./linsolve on a cuda named node50 by valera Wed >>>>>>>>>>>> Mar 14 10:24:35 2018 >>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 >>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>> [0]PETSC ERROR: #2 VecCUSPGetArrayRead() line 1792 in >>>>>>>>>>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >>>>>>>>>>>> [0]PETSC ERROR: #3 VecAXPY_SeqCUSP() line 314 in >>>>>>>>>>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >>>>>>>>>>>> [0]PETSC ERROR: #4 VecAXPY() line 612 in >>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/rvector.c >>>>>>>>>>>> [0]PETSC ERROR: #5 KSPSolve_GCR_cycle() line 60 in >>>>>>>>>>>> /home/valera/petsc/src/ksp/ksp/impls/gcr/gcr.c >>>>>>>>>>>> [0]PETSC ERROR: #6 KSPSolve_GCR() line 114 in >>>>>>>>>>>> /home/valera/petsc/src/ksp/ksp/impls/gcr/gcr.c >>>>>>>>>>>> [0]PETSC ERROR: #7 KSPSolve() line 669 in >>>>>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>> soln maxval: 0.0000000000000000 >>>>>>>>>>>> soln minval: 0.0000000000000000 >>>>>>>>>>>> Norm: 11.180339887498949 >>>>>>>>>>>> Its: 0 >>>>>>>>>>>> WARNING! There are options you set that were not used! >>>>>>>>>>>> WARNING! could be spelling mistake, etc! >>>>>>>>>>>> Option left: name:-mat_type value: aijcusparse >>>>>>>>>>>> [valera@node50 alone]$ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I also see the configure options are not correct, so i guess is >>>>>>>>>>>> still linking a different petsc installation, but maybe we can try >>>>>>>>>>>> to make >>>>>>>>>>>> it work as it is, i will let you know if i am able to link the >>>>>>>>>>>> correct >>>>>>>>>>>> petsc installation here, >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Mar 11, 2018 at 9:00 AM, Matthew Knepley < >>>>>>>>>>>> knep...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On Fri, Mar 9, 2018 at 3:05 AM, Manuel Valera < >>>>>>>>>>>>> mvaler...@mail.sdsu.edu> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I am working on porting a linear solver into GPUs for timing >>>>>>>>>>>>>> purposes, so far i've been able to compile and run the CUSP >>>>>>>>>>>>>> libraries and >>>>>>>>>>>>>> compile PETSc to be used with CUSP and ViennaCL, after the >>>>>>>>>>>>>> initial runs i >>>>>>>>>>>>>> noticed some errors, they are different for different flags and >>>>>>>>>>>>>> i would >>>>>>>>>>>>>> appreciate any help interpreting them, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The only elements in this program that use PETSc are the >>>>>>>>>>>>>> laplacian matrix (sparse), the RHS and X vectors and a scatter >>>>>>>>>>>>>> petsc >>>>>>>>>>>>>> object, so i would say it's safe to pass the command line >>>>>>>>>>>>>> arguments for the >>>>>>>>>>>>>> Mat/VecSetType()s instead of changing the source code, >>>>>>>>>>>>>> >>>>>>>>>>>>>> If i use *-vec_type cuda -mat_type aijcusparse* or *-vec_type >>>>>>>>>>>>>> viennacl -mat_type aijviennacl *i get the following: >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> These systems do not properly propagate errors. My only advice >>>>>>>>>>>>> is to run a smaller problem and see. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> [0]PETSC ERROR: ------------------------------ >>>>>>>>>>>>>> ------------------------------------------ >>>>>>>>>>>>>> [0]PETSC ERROR: Caught signal number 11 SEGV: Segmentation >>>>>>>>>>>>>> Violation, probably memory access out of range >>>>>>>>>>>>>> [0]PETSC ERROR: Try option -start_in_debugger or >>>>>>>>>>>>>> -on_error_attach_debugger >>>>>>>>>>>>>> [0]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>>> ocumentation/faq.html#valgrind >>>>>>>>>>>>>> [0]PETSC ERROR: or try http://valgrind.org on GNU/linux and >>>>>>>>>>>>>> Apple Mac OS X to find memory corruption errors >>>>>>>>>>>>>> [0]PETSC ERROR: likely location of problem given in stack >>>>>>>>>>>>>> below >>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Stack Frames >>>>>>>>>>>>>> ------------------------------------ >>>>>>>>>>>>>> [0]PETSC ERROR: Note: The EXACT line numbers in the stack are >>>>>>>>>>>>>> not available, >>>>>>>>>>>>>> [0]PETSC ERROR: INSTEAD the line number of the start of >>>>>>>>>>>>>> the function >>>>>>>>>>>>>> [0]PETSC ERROR: is given. >>>>>>>>>>>>>> [0]PETSC ERROR: [0] VecSetValues line 847 >>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/rvector.c >>>>>>>>>>>>>> [0]PETSC ERROR: [0] VecSetType line 36 >>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/vecreg.c >>>>>>>>>>>>>> [0]PETSC ERROR: [0] VecSetTypeFromOptions_Private line 1230 >>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>>>>>>>>>>>> [0]PETSC ERROR: [0] VecSetFromOptions line 1271 >>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/interface/vector.c >>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> [0]PETSC ERROR: Signal received >>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by >>>>>>>>>>>>>> valera Thu Mar 8 09:50:51 2018 >>>>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort >>>>>>>>>>>>>> --COPTFLAGS=-O3 >>>>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>>> [0]PETSC ERROR: #5 User provided function() line 0 in >>>>>>>>>>>>>> unknown file >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> -------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> This seems to be a memory out of range, maybe my vector is >>>>>>>>>>>>>> too big for my CUDA system? how do i assess that? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Next, if i use *-vec_type cusp -mat_type aijcusparse *i get >>>>>>>>>>>>>> something different and more interesting: >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> We need to see the entire error message, since it has the >>>>>>>>>>>>> stack. >>>>>>>>>>>>> >>>>>>>>>>>>> This seems like a logic error, but could definitely be on our >>>>>>>>>>>>> end. Here is how I think about these: >>>>>>>>>>>>> >>>>>>>>>>>>> 1) We have nightly test solves, so at least some solver >>>>>>>>>>>>> configuration works >>>>>>>>>>>>> >>>>>>>>>>>>> 2) Some vector which is marked read-only (happens for input >>>>>>>>>>>>> to solvers), but someone is trying to update it. >>>>>>>>>>>>> The stack will tell me where this is happening. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> >>>>>>>>>>>>> Matt >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> [0]PETSC ERROR: Object is in wrong state >>>>>>>>>>>>>> [0]PETSC ERROR: Vec is locked read only, argument # 3 >>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by >>>>>>>>>>>>>> valera Thu Mar 8 10:02:19 2018 >>>>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort >>>>>>>>>>>>>> --COPTFLAGS=-O3 >>>>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>>> [0]PETSC ERROR: #48 KSPSolve() line 615 in >>>>>>>>>>>>>> /home/valera/petsc/src/ksp/ksp/interface/itfunc.c >>>>>>>>>>>>>> PETSC_SOLVER_ONLY 6.8672990892082453E-005 s >>>>>>>>>>>>>> [0]PETSC ERROR: --------------------- Error Message >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> [0]PETSC ERROR: Invalid argument >>>>>>>>>>>>>> [0]PETSC ERROR: Object (seq) is not seqcusp or mpicusp >>>>>>>>>>>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/d >>>>>>>>>>>>>> ocumentation/faq.html for trouble shooting. >>>>>>>>>>>>>> [0]PETSC ERROR: Petsc Development GIT revision: >>>>>>>>>>>>>> v3.8.3-1817-g96b6f8a GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>>> [0]PETSC ERROR: ./gcmSeamount on a cuda named node50 by >>>>>>>>>>>>>> valera Thu Mar 8 10:02:19 2018 >>>>>>>>>>>>>> [0]PETSC ERROR: Configure options PETSC_ARCH=cuda >>>>>>>>>>>>>> --with-cc=mpicc --with-cxx=mpic++ --with-fc=mpifort >>>>>>>>>>>>>> --COPTFLAGS=-O3 >>>>>>>>>>>>>> --CXXOPTFLAGS=-O3 --FOPTFLAGS=-O3 --with-shared-libraries=1 >>>>>>>>>>>>>> --with-debugging=1 --with-cuda=1 --with-cuda-arch=sm_60 >>>>>>>>>>>>>> --with-cusp=1 >>>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp --with-vienacl=1 >>>>>>>>>>>>>> --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>>> [0]PETSC ERROR: #49 VecCUSPGetArrayReadWrite() line 1718 in >>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/veccusp2.cu >>>>>>>>>>>>>> [0]PETSC ERROR: #50 VecScatterCUSP_StoS() line 269 in >>>>>>>>>>>>>> /home/valera/petsc/src/vec/vec/impls/seq/seqcusp/vecscatterc >>>>>>>>>>>>>> usp.cu >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> And it yields a "solution" to the system and also a log at >>>>>>>>>>>>>> the end: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ./gcmSeamount on a cuda named node50 with 1 processor, by >>>>>>>>>>>>>> valera Thu Mar 8 10:02:24 2018 >>>>>>>>>>>>>> Using Petsc Development GIT revision: v3.8.3-1817-g96b6f8a >>>>>>>>>>>>>> GIT Date: 2018-02-28 10:19:08 -0600 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Max Max/Min Avg >>>>>>>>>>>>>> Total >>>>>>>>>>>>>> Time (sec): 4.573e+00 1.00000 4.573e+00 >>>>>>>>>>>>>> Objects: 8.100e+01 1.00000 8.100e+01 >>>>>>>>>>>>>> Flop: 3.492e+07 1.00000 3.492e+07 >>>>>>>>>>>>>> 3.492e+07 >>>>>>>>>>>>>> Flop/sec: 7.637e+06 1.00000 7.637e+06 >>>>>>>>>>>>>> 7.637e+06 >>>>>>>>>>>>>> Memory: 2.157e+08 1.00000 >>>>>>>>>>>>>> 2.157e+08 >>>>>>>>>>>>>> MPI Messages: 0.000e+00 0.00000 0.000e+00 >>>>>>>>>>>>>> 0.000e+00 >>>>>>>>>>>>>> MPI Message Lengths: 0.000e+00 0.00000 0.000e+00 >>>>>>>>>>>>>> 0.000e+00 >>>>>>>>>>>>>> MPI Reductions: 0.000e+00 0.00000 >>>>>>>>>>>>>> >>>>>>>>>>>>>> Flop counting convention: 1 flop = 1 real number operation of >>>>>>>>>>>>>> type (multiply/divide/add/subtract) >>>>>>>>>>>>>> e.g., VecAXPY() for real vectors >>>>>>>>>>>>>> of length N --> 2N flop >>>>>>>>>>>>>> and VecAXPY() for complex vectors >>>>>>>>>>>>>> of length N --> 8N flop >>>>>>>>>>>>>> >>>>>>>>>>>>>> Summary of Stages: ----- Time ------ ----- Flop ----- --- >>>>>>>>>>>>>> Messages --- -- Message Lengths -- -- Reductions -- >>>>>>>>>>>>>> Avg %Total Avg %Total >>>>>>>>>>>>>> counts %Total Avg %Total counts %Total >>>>>>>>>>>>>> 0: Main Stage: 4.5729e+00 100.0% 3.4924e+07 100.0% >>>>>>>>>>>>>> 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% >>>>>>>>>>>>>> >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> See the 'Profiling' chapter of the users' manual for details >>>>>>>>>>>>>> on interpreting output. >>>>>>>>>>>>>> Phase summary info: >>>>>>>>>>>>>> Count: number of times phase was executed >>>>>>>>>>>>>> Time and Flop: Max - maximum over all processors >>>>>>>>>>>>>> Ratio - ratio of maximum to minimum over >>>>>>>>>>>>>> all processors >>>>>>>>>>>>>> Mess: number of messages sent >>>>>>>>>>>>>> Avg. len: average message length (bytes) >>>>>>>>>>>>>> Reduct: number of global reductions >>>>>>>>>>>>>> Global: entire computation >>>>>>>>>>>>>> Stage: stages of a computation. Set stages with >>>>>>>>>>>>>> PetscLogStagePush() and PetscLogStagePop(). >>>>>>>>>>>>>> %T - percent time in this phase %F - percent >>>>>>>>>>>>>> flop in this phase >>>>>>>>>>>>>> %M - percent messages in this phase %L - percent >>>>>>>>>>>>>> message lengths in this phase >>>>>>>>>>>>>> %R - percent reductions in this phase >>>>>>>>>>>>>> Total Mflop/s: 10e-6 * (sum of flop over all >>>>>>>>>>>>>> processors)/(max time over all processors) >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ############################## >>>>>>>>>>>>>> ############################ >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # WARNING!!! >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # This code was compiled with a debugging option, >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # To get timing results run ./configure >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # using --with-debugging=no, the performance will >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # be generally two or three times faster. >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # >>>>>>>>>>>>>> # >>>>>>>>>>>>>> ############################## >>>>>>>>>>>>>> ############################ >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Event Count Time (sec) Flop >>>>>>>>>>>>>> --- Global --- --- Stage --- Total >>>>>>>>>>>>>> Max Ratio Max Ratio Max Ratio >>>>>>>>>>>>>> Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> >>>>>>>>>>>>>> --- Event Stage 0: Main Stage >>>>>>>>>>>>>> >>>>>>>>>>>>>> MatLUFactorNum 1 1.0 4.9502e-02 1.0 3.49e+07 1.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 1100 0 0 0 1100 0 0 0 706 >>>>>>>>>>>>>> MatILUFactorSym 1 1.0 1.9642e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>> MatAssemblyBegin 2 1.0 6.9141e-06 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>> MatAssemblyEnd 2 1.0 2.6612e-01 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 6 0 0 0 0 6 0 0 0 0 0 >>>>>>>>>>>>>> MatGetRowIJ 1 1.0 5.0068e-06 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>> MatGetOrdering 1 1.0 1.7186e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>> MatLoad 1 1.0 1.1575e-01 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 3 0 0 0 0 3 0 0 0 0 0 >>>>>>>>>>>>>> MatView 1 1.0 8.0877e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0 >>>>>>>>>>>>>> MatCUSPCopyTo 1 1.0 2.4664e-01 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 5 0 0 0 0 5 0 0 0 0 0 >>>>>>>>>>>>>> VecSet 68 1.0 5.1665e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>>>>>>>>>>>> VecAssemblyBegin 17 1.0 5.2691e-05 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>> VecAssemblyEnd 17 1.0 4.3631e-05 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>> VecScatterBegin 15 1.0 1.5345e-03 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>> VecCUSPCopyFrom 1 1.0 1.1199e-03 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0 >>>>>>>>>>>>>> KSPSetUp 1 1.0 5.1929e-02 1.0 0.00e+00 0.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 1 0 0 0 0 1 0 0 0 0 0 >>>>>>>>>>>>>> PCSetUp 2 1.0 8.6590e-02 1.0 3.49e+07 1.0 >>>>>>>>>>>>>> 0.0e+00 0.0e+00 0.0e+00 2100 0 0 0 2100 0 0 0 403 >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> ------------------------------------------------------------ >>>>>>>>>>>>>> >>>>>>>>>>>>>> Memory usage is given in bytes: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Object Type Creations Destructions Memory >>>>>>>>>>>>>> Descendants' Mem. >>>>>>>>>>>>>> Reports information only for process 0. >>>>>>>>>>>>>> >>>>>>>>>>>>>> --- Event Stage 0: Main Stage >>>>>>>>>>>>>> >>>>>>>>>>>>>> Matrix 3 1 52856972 0. >>>>>>>>>>>>>> Matrix Null Space 1 1 608 0. >>>>>>>>>>>>>> Vector 66 3 3414600 0. >>>>>>>>>>>>>> Vector Scatter 1 1 680 0. >>>>>>>>>>>>>> Viewer 3 2 1680 0. >>>>>>>>>>>>>> Krylov Solver 1 0 0 0. >>>>>>>>>>>>>> Preconditioner 2 1 864 0. >>>>>>>>>>>>>> Index Set 4 1 800 0. >>>>>>>>>>>>>> ============================================================ >>>>>>>>>>>>>> ============================================================ >>>>>>>>>>>>>> Average time to get PetscTime(): 9.53674e-08 >>>>>>>>>>>>>> #PETSc Option Table entries: >>>>>>>>>>>>>> -ksp_view >>>>>>>>>>>>>> -log_view >>>>>>>>>>>>>> -mat_type aijcusparse >>>>>>>>>>>>>> -matload_block_size 1 >>>>>>>>>>>>>> -vec_type cusp >>>>>>>>>>>>>> #End of PETSc Option Table entries >>>>>>>>>>>>>> Compiled without FORTRAN kernels >>>>>>>>>>>>>> Compiled with full precision matrices (default) >>>>>>>>>>>>>> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 >>>>>>>>>>>>>> sizeof(PetscScalar) 8 sizeof(PetscInt) 4 >>>>>>>>>>>>>> Configure options: PETSC_ARCH=cuda --with-cc=mpicc >>>>>>>>>>>>>> --with-cxx=mpic++ --with-fc=mpifort --COPTFLAGS=-O3 >>>>>>>>>>>>>> --CXXOPTFLAGS=-O3 >>>>>>>>>>>>>> --FOPTFLAGS=-O3 --with-shared-libraries=1 --with-debugging=1 >>>>>>>>>>>>>> --with-cuda=1 >>>>>>>>>>>>>> --with-cuda-arch=sm_60 --with-cusp=1 >>>>>>>>>>>>>> --with-cusp-dir=/home/valera/cusp >>>>>>>>>>>>>> --with-vienacl=1 --download-fblaslapack=1 --download-hypre >>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>> Libraries compiled on Mon Mar 5 16:37:18 2018 on node50 >>>>>>>>>>>>>> Machine characteristics: Linux-3.10.0-693.17.1.el7.x86_ >>>>>>>>>>>>>> 64-x86_64-with-centos-7.2.1511-Core >>>>>>>>>>>>>> Using PETSc directory: /home/valera/petsc >>>>>>>>>>>>>> Using PETSc arch: cuda >>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Using C compiler: mpicc -fPIC -Wall -Wwrite-strings >>>>>>>>>>>>>> -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector >>>>>>>>>>>>>> -fvisibility=hidden -O3 >>>>>>>>>>>>>> Using Fortran compiler: mpifort -fPIC -Wall >>>>>>>>>>>>>> -ffree-line-length-0 -Wno-unused-dummy-argument -O3 >>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Using include paths: -I/home/valera/petsc/cuda/include >>>>>>>>>>>>>> -I/home/valera/petsc/include -I/home/valera/petsc/include >>>>>>>>>>>>>> -I/home/valera/petsc/cuda/include -I/home/valera/cusp/ >>>>>>>>>>>>>> -I/usr/local/cuda/include >>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> Using C linker: mpicc >>>>>>>>>>>>>> Using Fortran linker: mpifort >>>>>>>>>>>>>> Using libraries: -Wl,-rpath,/home/valera/petsc/cuda/lib >>>>>>>>>>>>>> -L/home/valera/petsc/cuda/lib -lpetsc >>>>>>>>>>>>>> -Wl,-rpath,/home/valera/petsc/cuda/lib >>>>>>>>>>>>>> -L/home/valera/petsc/cuda/lib -Wl,-rpath,/usr/local/cuda/lib64 >>>>>>>>>>>>>> -L/usr/local/cuda/lib64 -Wl,-rpath,/usr/lib64/openmpi/lib >>>>>>>>>>>>>> -L/usr/lib64/openmpi/lib >>>>>>>>>>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 >>>>>>>>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -lHYPRE -lflapack >>>>>>>>>>>>>> -lfblas -lm -lcufft -lcublas -lcudart -lcusparse -lX11 -lstdc++ >>>>>>>>>>>>>> -ldl >>>>>>>>>>>>>> -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm >>>>>>>>>>>>>> -lgcc_s >>>>>>>>>>>>>> -lquadmath -lpthread -lstdc++ -ldl >>>>>>>>>>>>>> ----------------------------------------- >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for your help, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Manuel >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> What most experimenters take for granted before they begin >>>>>>>>>>>>> their experiments is infinitely more interesting than any results >>>>>>>>>>>>> to which >>>>>>>>>>>>> their experiments lead. >>>>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>>>> >>>>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>>>> experiments is infinitely more interesting than any results to >>>>>>>>>>> which their >>>>>>>>>>> experiments lead. >>>>>>>>>>> -- Norbert Wiener >>>>>>>>>>> >>>>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> What most experimenters take for granted before they begin their >>>>>>>>> experiments is infinitely more interesting than any results to which >>>>>>>>> their >>>>>>>>> experiments lead. >>>>>>>>> -- Norbert Wiener >>>>>>>>> >>>>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> What most experimenters take for granted before they begin their >>>>>>> experiments is infinitely more interesting than any results to which >>>>>>> their >>>>>>> experiments lead. >>>>>>> -- Norbert Wiener >>>>>>> >>>>>>> https://www.cse.buffalo.edu/~knepley/ >>>>>>> <http://www.caam.rice.edu/~mk51/> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> What most experimenters take for granted before they begin their >>>>> experiments is infinitely more interesting than any results to which their >>>>> experiments lead. >>>>> -- Norbert Wiener >>>>> >>>>> https://www.cse.buffalo.edu/~knepley/ >>>>> <http://www.caam.rice.edu/~mk51/> >>>>> >>>> >>>> >>> >>> >>> -- >>> What most experimenters take for granted before they begin their >>> experiments is infinitely more interesting than any results to which their >>> experiments lead. >>> -- Norbert Wiener >>> >>> https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >>> >> >> > > > -- > What most experimenters take for granted before they begin their > experiments is infinitely more interesting than any results to which their > experiments lead. > -- Norbert Wiener > > https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/> >