Hi Trevor,

Thanks for looking into this bug. It isn't holding back my research, but tracking it down is a fun distraction from applying to post-doc positions.


I am building Trilinos with Trilinos_ENABLE_ALL_PACKAGES=ON and only a few parts of SEACAS are automatically disabled. I disabled the TPLs for Matio, GLM, and Netcdf because I don't need them. I think it's safe to say that I have all the modules.

I've been pulling from the GitHub repository, and I noticed that Trilinos 11.10.2 isn't available there, only on the Trilinos site: https://trilinos.org/download/previous-releases/download-11-10/ But at this point, I doubt that this is related to the issue I am experiencing.


As a side note, If anyone wants help compiling Trilinos, I am getting very good at it.


Best,

Mike


On 10/12/2016 10:49 AM, Keller, Trevor (Fed) wrote:

Hi Mike,


Thanks for testing all those combinations! Sorry none worked though. For me, the specific combination of Trilinos 11.10.2 with swig 2.0.8 closed the memory leak back in March. I'm surprised the latest versions (6 months later) still create a leak. It makes me suspect we're doing something wrong, perhaps neglecting some important module in Trilinos? I'll try building the latest release and, if the leak persists on my machine as well, try a bit harder to track it down. If nothing else, I'll file a bug report and try to get more developers' eyes on this.


My software environment is very similar to yours:Boost v1.55, OpenMPI 1.6.5, and gcc 4.9.2.


With regards,

Trevor



Trevor Keller, Ph.D.
Materials Science and Engineering Division
National Institute of Standards and Technology
100 Bureau Dr. MS 8550; Gaithersburg, MD 20899
Office: 223/A131 or (301) 975-2889


------------------------------------------------------------------------
*From:* fipy-boun...@nist.gov <fipy-boun...@nist.gov> on behalf of Michael Waters <waters.mik...@gmail.com>
*Sent:* Tuesday, October 11, 2016 4:55:51 PM
*To:* FIPY
*Subject:* Re: Memory Leaks with Trilinos

Hi Trevor,

My mystery deepens. Today, I've compiled and tried the following combinations of Trilinos/Swig:

12.8.1/3.0.2

11.14.3/3.0.2

11.12.1/3.0.2

11.10.1/2.0.8

11.2.5/2.0.8

All of these combinations leak memory at about the same rate. I am using Boost 1.54, Openmpi 1.6.5, and GCC 4.8.4. Are you using similar?

Thanks,

-Mike


On 03/30/2016 05:06 PM, Keller, Trevor (Fed) wrote:
Mike & Jon,

Running 25 steps on Debian wheezy using PyTrilinos 11.10.2 and swig 2.0.8, 
monitoring with memory-profiler, I do not see a memory leak (see attachment): 
the simulation is fairly steady at 8GB RAM. The same code, but using PyTrilinos 
12.1.0, ramps up to 16GB in the same simulation time.

Later revisions of Trilinos 12 remain defiant (though, compiling swig-3.0.8 has 
improved matters). If one of them compiles and works, I will let you know. If 
not, if possible, Mike, can you test the next incremental version (11.10.2)?

Trevor


________________________________________
From:fipy-boun...@nist.gov  <fipy-boun...@nist.gov>  on behalf of Michael 
Waters<waters.mik...@gmail.com>
Sent: Wednesday, March 30, 2016 4:36 PM
To: FIPY
Subject: Re: Memory Leaks with Trilinos

Hi Jon,

I just compiled an old version of swig (2.0.8) and compiled Trilinos
(11.10.1) against that. Sadly, I am still having the leak.

I am out of ideas for the day... and should be looking for a post doc
anyway.

Thanks,
-mike


On 3/30/16 3:32 PM, Guyer, Jonathan E. Dr. (Fed) wrote:
No worries. If building trilinos doesn't blindside you with something 
unexpected and unpleasant, you're not doing it right.

I have a conda recipe 
athttps://github.com/guyer/conda-recipes/tree/trilinos_upgrade_11_10_2/trilinos 
 that has worked for me to build 11.10.2 on both OS X and Docker (Debian?). I 
haven't tried to adjust it to 12.x, yet.


On Mar 30, 2016, at 2:42 PM, Michael Waters<waters.mik...@gmail.com>  wrote:

Hi Jon,

I was just reviewing my version of Trilinos 11.10 and discovered that there is 
no way that I compiled it last night after exercising. It has unsatisfied 
dependencies on my machine. So I must apologize, I must have been more tired 
than I thought.

Sorry for the error!
-Mike Waters

On 3/30/16 11:52 AM, Guyer, Jonathan E. Dr. (Fed) wrote:
It looked to me like steps and accuracy were the way to do it, but my runs 
finish in one step, so I was confused. When I change to accuracy = 10.0**-6, it 
takes 15 steps, but still no leak (note, the hiccup in RSS and in ELAPSED time 
is because I put my laptop to sleep for awhile, but VSIZE is rock-steady).

The fact that things never (or slowly) converge for you and Trevor, in addition 
to the leak, makes me wonder if Trilinos seriously broke something between 11.x 
and 12.x. Trevor's been struggling to build 12.4. I'll try to find time to do 
the same.

In case it matters, I'm running on OS X. What's your system?

- Jon

On Mar 29, 2016, at 3:59 PM, Michael Waters<waters.mik...@gmail.com>  wrote:

When I did my testing and made those graphs, I ran Trilinos in serial.
Syrupy didn't seem to track the other processes memory. I watched in
real time as the parallel version ate all my ram though.

To make the program run longer while not changing the memory:

steps = 100  # increase this, (limits the number of self-consistent
iterations)
accuracy = 10.0**-5 # make this number smaller, (relative energy
eigenvalue change for being considered converged )
initial_solver_iterations_per_step = 7 # reduce this to 1,  (number of
solver iterations per self-consistent iteration, to small and it's slow,
to high and the solutions are not stable)

I did those tests on a machine with 128 GB of ram so I wasn't expecting
any swapping.

Thanks,
-mike


On 3/29/16 3:38 PM, Guyer, Jonathan E. Dr. (Fed) wrote:
I guess I spoke too soon. FWIW, I'm running Trilinos version: 11.10.2.


On Mar 29, 2016, at 3:34 PM, Guyer, Jonathan E. Dr. 
(Fed)<jonathan.gu...@nist.gov>  wrote:

I'm not seeing a leak. The below is for trilinos. VSIZE grows to about 11 MiB 
and saturates and RSS saturates at around 5 MiB. VSIZE is more relevant for 
tracking leaks, as RSS is deeply tied to your system's swapping architecture 
and what else is running; either way, neither seems to be leaking, but this 
problem does use a lot of memory.

What do I need to do to get it to run longer?



On Mar 25, 2016, at 7:16 PM, Michael Waters<waters.mik...@gmail.com>  wrote:

Hello,

I still have a large memory leak when using Trilinos. I am not sure where to 
start looking so I made an example code that produces my problem in hopes that 
someone can help me.

But! my example is cool. I implemented Density Functional Theory in FiPy!

My code is slow, but runs in parallel and is simple (relative to most DFT 
codes). The example I have attached is just a lithium and hydrogen atom. The 
electrostatic boundary conditions are goofy but work well enough for 
demonstration purposes. If you set use_trilinos to True, the code will slowly 
use more memory. If not, it will try to use Pysparse.

Thanks,
-Michael Waters
<input.xyz><fipy-dft.py>_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
   [ NIST internal ONLY:https://email.nist.gov/mailman/listinfo/fipy  ]
<attachment.png>
_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
   [ NIST internal ONLY:https://email.nist.gov/mailman/listinfo/fipy  ]
_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
    [ NIST internal ONLY:https://email.nist.gov/mailman/listinfo/fipy  ]
_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
   [ NIST internal ONLY:https://email.nist.gov/mailman/listinfo/fipy  ]
<Mail Attachment.png>


_______________________________________________
fipy mailing list

fipy@nist.gov
http://www.ctcms.nist.gov/fipy

    [ NIST internal ONLY:
https://email.nist.gov/mailman/listinfo/fipy
   ]

_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
   [ NIST internal ONLY:https://email.nist.gov/mailman/listinfo/fipy  ]
_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
    [ NIST internal ONLY:https://email.nist.gov/mailman/listinfo/fipy  ]
_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
   [ NIST internal ONLY:https://email.nist.gov/mailman/listinfo/fipy  ]


_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
   [ NIST internal ONLY:https://email.nist.gov/mailman/listinfo/fipy  ]



_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
   [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]

_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
  [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]

Reply via email to