Basically, PetscInitialize is called 2 times, the second time from libmesh. PetscInitialize calls cudaSetDevice(), which fails the second time.
----- Original Message ----- From: "Matthew Knepley" <[email protected]> To: "Harshad Sahasrabudhe" <[email protected]> Cc: [email protected] Sent: Thursday, September 26, 2013 8:23:14 PM Subject: Re: [petsc-users] cudaSetDevice On Thu, Sep 26, 2013 at 3:40 PM, Harshad Sahasrabudhe < [email protected] > wrote: Hi, In our code, PetscInitialize() is called initially. Then, depending on whether we need libmesh or not, LibMeshInit() is called. This works fine when we are not using CUDA in PETSc. But when we are using CUDA, cudaSetDevice() gets called twice which throws an error: So you are saying cudaSetDevice() gets called before PetscInit? What would you propose for us? The problem is that we have no abstraction (like the communicator) which can tell us that you have called it. Matt [0]PETSC ERROR: --------------------- Error Message ------------------------------------ [0]PETSC ERROR: Error in system call! [0]PETSC ERROR: error in cudaSetDevice cannot set while device is active in this process! [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: Petsc Development GIT revision: GIT Date: [0]PETSC ERROR: See docs/changes/index.html for recent updates. [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting. [0]PETSC ERROR: See docs/index.html for manual pages. [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: ./nemo on a arch-linux2-cxx-opt named nid20743 by sahasrab Thu Sep 26 17:26:43 2013 [0]PETSC ERROR: Libraries linked from /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/arch-linux2-cxx-opt/lib [0]PETSC ERROR: Configure run at Mon Sep 23 18:59:51 2013 [0]PETSC ERROR: Configure options --with-x=0 --with-hdf5 --with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.11/GNU/48 --with-scalar-type=real --with-single-library=0 --with-pic=1 --with-shared-libraries=0 --with-mpiexec=aprun -lsci --with-clanguage=C++ --with-fortran --with-debugging=1 --with-cc="cc " --with-fc="ftn " --with-cxx="CC " COPTFLAGS=-O3 CXXOPTFLAGS=-O3 FOPTFLAGS=-O3 --download-metis=1 --download-parmetis=1 --download-superlu_dist=1 --download-mumps=1 --download-scalapack=1 --download-blacs=1 --download-magma=1 --with-cuda-dir=/opt/nvidia/cudatoolkit/5.0.35 --with-cusp-dir=/u/sciteam/sahasrab/NEMO5/libs/petsc/cusplibrary-master --with-cuda=1 --with-thrust=1 --with-mpi-dir=/opt/cray/mpt/6.0.1/gni/mpich2-cray/81 --with-mpi=1 [0]PETSC ERROR: ------------------------------------------------------------------------ [0]PETSC ERROR: PetscOptionsCheckInitial_Private() line 575 in /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/init.c [0]PETSC ERROR: PetscInitialize() line 788 in /u/sciteam/sahasrab/NEMO5/libs/petsc/build-real/src/sys/objects/pinit.c [0]PETSC ERROR: LibMeshInit() line 457 in "unknowndirectory/"src/base/libmesh.C Can we tell PETSc when to run cudaSetDevice()? Or is there any other way to avoid this? Thanks, Harshad -- What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead. -- Norbert Wiener
