Hong, You need to do similar for your elemental grid to avoid memory leaks when using mpiuni. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/169c19e9ea54
Shri On Jul 16, 2012, at 8:08 AM, Shri wrote: > > On Jul 15, 2012, at 5:26 PM, Shri wrote: > >> >> On Jul 15, 2012, at 4:57 PM, Barry Smith wrote: >> >>> >>> On Jul 15, 2012, at 4:48 PM, Shri wrote: >>> >>>> >>>> On Jul 15, 2012, at 3:27 PM, Barry Smith wrote: >>>> >>>>> >>>>> Shri, >>>>> >>>>> You need to test your code WITHOUT pthreads OR openmp configure. >>>> That is what i am doing, no openmp or pthreads. >>>> >>>>> You have the call to PetscThreadCommFinalize() protected by >>>>> PTHREADCLASSES >>>> >>>> There is no PetscThreadCommFinalize(). There is a 'PetscThreadsFinalize()' >>>> call in PetscFinalize() that is protected by PTHREADCLASSES. >>>> PetscThreadsFinalize() is for finalizing the previous pthread pool model >>>> that i had added on which vec_type pthread and mat_type seqaijpthread are >>>> based. It has nothing to do with threadcomm. Once i add new vector and >>>> matrix code using threadcomm , i'll remove the old pthread code and the >>>> pthread vector and matrix class. >>> >>> Why not just remove all the old pthread code now; it will make things >>> clearer. >> >> Don't want to hurl people who are using/testing it into darkness. >> >>> >>>>> but not PetscThreadCommInitializePackage() projected! Also why do you >>>>> call PetscThreadCommPackageInitialize() at the beginning but >>>>> PetscThreadCommFinalize() at the end and not >>>>> PetscThreadCommFinalizePackage()? >>>> >>>> PetscThreadCommFinalizePackage is registerfinalized in >>>> PetscThreadCommInitializePackage() >>>> >>>> ierr = PetscRegisterFinalize(PetscThreadCommFinalizePackage);CHKERRQ(ierr); >>> >>> Sorry my mistake. >>>> >>>> So it is called on PetscRegisterFinalizeAll(). >>>>> >>>>> BTW: If I #if def out the PetscThreadCommPackageInitialize() then nothing >>>>> works (I have built without pthread classes and without openmp). So >>>>> there is some bad stuff going on. >>>> >>>> PetscThreadCommPackageInitialize() attaches the threadcomm attribute on >>>> the MPI communicator and there are several places where the threadcomm >>>> attribute is requested (sys/objects/subcomm.c, src/sys/tagm.c). Hence, the >>>> errors. >>> >>> Try a build with mpiuni. >> >> Aah, i can see the memory leak now. Thanks. It looks like the threadcomm >> object is not being freed. Something to do with destroying the MPIUNI >> attribute perhaps? Will check. > > Fixed. http://petsc.cs.iit.edu/petsc/petsc-dev/rev/169c19e9ea54 > The memory leak was due to the COMM_SELF and COMM_WORLD communicators being > same for mpiuni. This was leaving an extra reference for the innercomm. > > Shri > >> >> Shri >>> Maybe the memory leak is because that isn't handling stuff possible. My >>> build with memory leaks is with mpiuni. >>> >>> Thanks >>> >>> >>> Barry >>> >>>> >>>> Thanks, >>>> Shri >>>>> >>>>> Barry >>>>> >>>>> On Jul 15, 2012, at 11:54 AM, Shri wrote: >>>>> >>>>>> >>>>>> On Jul 14, 2012, at 7:55 PM, Satish Balay wrote: >>>>>> >>>>>>> I see this with 32bit linux build - but not 64bit linux build. >>>>>> >>>>>> I tried a fresh clone on my Mac and a linux cluster (both x86_64) but >>>>>> still did not get any memory leaks. Perhaps this is a 32 bit build >>>>>> issue as Satish mentions. What configure options do i need to have a 32 >>>>>> bit build on x86_64? CC="gcc -m32" FC="gfortran -m32"? >>>>>> >>>>>> Thanks, >>>>>> Shri >>>>>> >>>>>>> >>>>>>> also 32bit linux build gives errors with -fp_trap in ddot(). Haven't >>>>>>> checked why. >>>>>>> >>>>>>> Satish >>>>>>> >>>>>>> >>>>>>> On Sat, 14 Jul 2012, Barry Smith wrote: >>>>>>> >>>>>>>> >>>>>>>> Many of them with make test >>>>>>>> >>>>>>>> Running test examples to verify correct installation >>>>>>>> Using PETSC_DIR=/Users/barrysmith/Src/petsc-dev and >>>>>>>> PETSC_ARCH=arch-float128-opt >>>>>>>> Possible error running C/C++ src/snes/examples/tutorials/ex19 with 1 >>>>>>>> MPI process >>>>>>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html >>>>>>>> lid velocity = Qg, prandtl # = Qg, grashof # = Qg >>>>>>>> Number of SNES iterations = 2 >>>>>>>> [0]Total space allocated 6880 bytes >>>>>>>> [ 0]16 bytes PetscThreadCommReductionCreate() line 260 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c >>>>>>>> [ 0]16 bytes PetscThreadCommReductionCreate() line 259 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c >>>>>>>> [ 0]48 bytes PetscThreadCommReductionCreate() line 258 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcommred.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommInitialize() line 698 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]176 bytes PetscThreadCommInitialize() line 694 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]272 bytes PetscThreadCommInitialize() line 692 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]16 bytes PetscThreadCommSetAffinities() line 331 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]48 bytes PetscThreadCommCreate() line 122 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> [ 0]320 bytes PetscThreadCommCreate() line 118 in >>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>> Completed test examples >>>>>>>> barry-smiths-macbook-pro:petsc-dev barrysmith$ >>>>>>>> >>>>>>>> >>>>>>>> On Jul 14, 2012, at 6:36 PM, Shri wrote: >>>>>>>> >>>>>>>>> Which example are you running? I don't see any memory leak for the >>>>>>>>> examples i've tested (threadcomm examples >>>>>>>>> sys/threadcomm/examples/tutorials/ and >>>>>>>>> snes/examples/tutorials/ex19.c,ex5.c) >>>>>>>>> On Jul 14, 2012, at 4:22 PM, Barry Smith wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please always test for memory leaks before pushing >>>>>>>>>> >>>>>>>>>> [ 0]48 bytes PetscThreadCommCreate() line 122 in >>>>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>>>> [0] PetscThreadCommInitialize() line 685 in >>>>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>>>> [0] PetscThreadCommInitializePackage() line 53 in >>>>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c >>>>>>>>>> [ 0]320 bytes PetscThreadCommCreate() line 118 in >>>>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>>>> [0] PetscThreadCommInitialize() line 685 in >>>>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/threadcomm.c >>>>>>>>>> [0] PetscThreadCommInitializePackage() line 53 in >>>>>>>>>> /Users/barrysmith/Src/petsc-dev/src/sys/threadcomm/interface/dlregisthreadcomm.c >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
