Thanks a lot. On Sat, Jun 20, 2020 at 11:40 AM Junchao Zhang <[email protected]> wrote:
> Sam, > There are more problems. I am working on a fix. Please wait an hour. > Thanks. > > --Junchao Zhang > > > On Sat, Jun 20, 2020 at 1:12 PM Sam Guo <[email protected]> wrote: > >> Junchao, >> I debugged: MPI_Finalize is not called for serial. >> >> Barry, >> I tried your patch and it seems better but eventually got following >> error: >> >> [0]PETSC ERROR: #1 PetscCommDuplicate() line 160 in >> ../../../petsc/src/sys/objects/tagm.c >> [0]PETSC ERROR: #2 PetscHeaderCreate_Private() line 64 in >> ../../../petsc/src/sys/objects/inherit.c >> [0]PETSC ERROR: #3 MatCreate() line 91 in >> ../../../petsc/src/mat/utils/gcreate.c >> [0]PETSC ERROR: #4 MatCreateShell() line 787 in >> ../../../petsc/src/mat/impls/shell/shell.c >> [0]PETSC ERROR: --------------------- Error Message >> -------------------------------------------------------------- >> [0]PETSC ERROR: Null argument, when expecting valid pointer >> [0]PETSC ERROR: Null Object: Parameter # 1 >> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html >> for trouble shooting. >> [0]PETSC ERROR: Petsc Release Version 3.11.3, Jun, 26, 2019 >> [0]PETSC ERROR: #6 MatShellSetOperation() line 1052 in >> ../../../petsc/src/mat/impls/shell/shell.c >> >> On Sat, Jun 20, 2020 at 10:24 AM Barry Smith <[email protected]> wrote: >> >>> >>> Junchao, >>> >>> This is a good bug fix. It solves the problem when PETSc initialize >>> is called many times. >>> >>> There is another fix you can do to limit PETSc mpiuni running out >>> of attributes inside a single PETSc run: >>> >>> >>> int MPI_Comm_create_keyval(MPI_Copy_function >>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>> { >>> >>> if (num_attr >= MAX_ATTR){ >>> for (i=0; i<num_attr; i++) { >>> if (!attr_keyval[i].extra_state) { >>> /* reuse this slot */ >>> attr_keyval[i].extra_state = extra_state; >>> attr_keyval[i.]del = delete_fn; >>> *keyval = i; >>> return MPI_SUCCESS; >>> } >>> } >>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>> } >>> return MPIUni_Abort(MPI_COMM_WORLD,1); >>> attr_keyval[num_attr].extra_state = extra_state; >>> attr_keyval[num_attr].del = delete_fn; >>> *keyval = num_attr++; >>> return MPI_SUCCESS; >>> } >>> >>> This will work if the user creates tons of attributes but is >>> constantly deleting some as they new ones. So long as the number >>> outstanding at one time is < MAX_ATTR) >>> >>> Barry >>> >>> >>> >>> >>> >>> On Jun 20, 2020, at 10:54 AM, Junchao Zhang <[email protected]> >>> wrote: >>> >>> I don't understand what your session means. Let's try this patch >>> >>> diff --git a/src/sys/mpiuni/mpi.c b/src/sys/mpiuni/mpi.c >>> index d559a513..c058265d 100644 >>> --- a/src/sys/mpiuni/mpi.c >>> +++ b/src/sys/mpiuni/mpi.c >>> @@ -283,6 +283,7 @@ int MPI_Finalize(void) >>> MPI_Comm_free(&comm); >>> comm = MPI_COMM_SELF; >>> MPI_Comm_free(&comm); >>> + num_attr = 1; /* reset the counter */ >>> MPI_was_finalized = 1; >>> return MPI_SUCCESS; >>> } >>> >>> >>> --Junchao Zhang >>> >>> >>> On Sat, Jun 20, 2020 at 10:48 AM Sam Guo <[email protected]> wrote: >>> >>>> Typo: I mean “Assuming initializer is only needed once for entire >>>> session” >>>> >>>> On Saturday, June 20, 2020, Sam Guo <[email protected]> wrote: >>>> >>>>> Assuming finalizer is only needed once for entire session(?), I can >>>>> put initializer into the static block to call it once but where do I call >>>>> finalizer? >>>>> >>>>> >>>>> On Saturday, June 20, 2020, Junchao Zhang <[email protected]> >>>>> wrote: >>>>> >>>>>> The counter num_attr should be recycled. But first try to call PETSc >>>>>> initialize/Finalize only once to see it fixes the error. >>>>>> --Junchao Zhang >>>>>> >>>>>> >>>>>> On Sat, Jun 20, 2020 at 12:48 AM Sam Guo <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> To clarify, I call PETSc initialize and PETSc finalize everytime I >>>>>>> call SLEPc: >>>>>>> >>>>>>> PetscInitializeNoPointers(argc,args,nullptr,nullptr); >>>>>>> >>>>>>> SlepcInitialize(&argc,&args,static_cast<char*>(nullptr),help); >>>>>>> >>>>>>> //calling slepc >>>>>>> >>>>>>> SlepcFinalize(); >>>>>>> >>>>>>> PetscFinalize(); >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, Jun 19, 2020 at 10:32 PM Sam Guo <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Dear PETSc team, >>>>>>>> When I called SLEPc multiple time, I eventually got following >>>>>>>> error: >>>>>>>> >>>>>>>> MPI operation not supported by PETSc's sequential MPI wrappers >>>>>>>> [0]PETSC ERROR: #1 PetscInitialize() line 967 in >>>>>>>> ../../../petsc/src/sys/objects/pinit.c >>>>>>>> [0]PETSC ERROR: #2 SlepcInitialize() line 262 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> [0]PETSC ERROR: #3 SlepcInitializeNoPointers() line 359 in >>>>>>>> ../../../slepc/src/sys/slepcinit.c >>>>>>>> PETSC ERROR: Logging has not been enabled. >>>>>>>> You might have forgotten to call PetscInitialize(). >>>>>>>> >>>>>>>> I debugged: it is because of following in >>>>>>>> petsc/src/sys/mpiuni/mpi.c >>>>>>>> >>>>>>>> if (num_attr >= MAX_ATTR) >>>>>>>> >>>>>>>> in function int MPI_Comm_create_keyval(MPI_Copy_function >>>>>>>> *copy_fn,MPI_Delete_function *delete_fn,int *keyval,void *extra_state) >>>>>>>> >>>>>>>> num_attr is declared static and keeps increasing every >>>>>>>> time MPI_Comm_create_keyval is called. >>>>>>>> >>>>>>>> I am using petsc 3.11.3 but found 3.13.2 has the same logic. >>>>>>>> >>>>>>>> Is this a bug or I didn't use it correctly? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Sam >>>>>>>> >>>>>>> >>>
