Hi Jed, You are right, it doesn't if you use just one PETSc. In our case, we use 2 PETSc libraries (compiled separately, one each for complex and double). And in that case, when the second PETSc library is initialized, cudaSetDevice throws an error. I can't think of any way in which both the PETSc libraries could be initialized with CUDA on the same process.
Do you have any idea? I can only think of turning CUDA off on one PETSc as a way to avoid this. Harshad ----- Original Message ----- From: "Jed Brown" <[email protected]> To: "Harshad Sahasrabudhe" <[email protected]>, "Matthew Knepley" <[email protected]> Cc: [email protected] Sent: Thursday, September 26, 2013 8:37:36 PM Subject: Re: [petsc-users] cudaSetDevice Harshad Sahasrabudhe <[email protected]> writes: > Basically, PetscInitialize is called 2 times, the second time from > libmesh. It should not be. $ git grep -C3 PetscInitialize src/base/libmesh.C src/base/libmesh.C- // Check whether the calling program has already initialized src/base/libmesh.C- // PETSc, and avoid duplicate Initialize/Finalize src/base/libmesh.C- PetscBool petsc_already_initialized; src/base/libmesh.C: ierr = PetscInitialized(&petsc_already_initialized); src/base/libmesh.C- CHKERRABORT(libMesh::COMM_WORLD,ierr); src/base/libmesh.C- if (petsc_already_initialized != PETSC_TRUE) src/base/libmesh.C- libmesh_initialized_petsc = true; -- src/base/libmesh.C-# else src/base/libmesh.C- if (libmesh_initialized_petsc) src/base/libmesh.C- { src/base/libmesh.C: ierr = PetscInitialize (&argc, const_cast<char***>(&argv), NULL, NULL); src/base/libmesh.C- CHKERRABORT(libMesh::COMM_WORLD,ierr); src/base/libmesh.C- } src/base/libmesh.C-# endif
