Dear All,

Here is an update to my e-mail from yesterday which described the exponential 
slow-down of my simulation with increasing CPU time. I have a applied a higher 
level of under-relaxation (i.e. smaller values for the underRelaxation term in 
the .sweep method), and by 200 seconds of simulation time, the number of sweeps 
and CPU time per sweep is remaining relatively constant. I’m running a longer 
simulation now to verify this, although I am hesitant to say the issue is 
solved until I investigate further.

Generally, the stability and rate of convergence are highly sensitive to the 
fixed under-relaxation factors used for each of the six .sweep methods within 
each time-step. Varying some relaxation factors by more than 0.01 can cause the 
simulation to either not converge at all, or to fail immediately with a 
StagnatedSolverWarning.

Best regards,
 
-          Ian

-----Original Message-----
From: fipy-boun...@nist.gov [mailto:fipy-boun...@nist.gov] On Behalf Of 
Campbell, Ian
Sent: 11 October 2016 18:21
To: fipy@nist.gov
Subject: RE: Memory Leakage & Object Build-up with FiPy Sweeps

Dear Jonathan, Daniel,

Thank you for your responses. Just yesterday, I discovered and solved this 
problem (another remains). It wasn't a result of calls to .sweep. The 
time-varying boundary condition for one of the PDEs was being re-defined within 
the time-stepping loop using the PDE.faceGrad.constrain() method. This led to a 
net creation of objects with every timestep, irrespective of garbage collection 
call frequency, and that in turn caused the slow-down of the simulation.

The solution was to instead update the value of the boundary condition by the 
.setValue method within the time-stepping loop, as below.

# Outside the loop, declare a FaceVariable for the value of the BC:
species_flux_neg_particle_surf = FaceVariable(mesh=p2d_mesh) # Next, apply that 
value to the BC at the top of that mesh:
Cs_p2d.faceGrad.constrain(species_flux_neg_particle_surf, 
where=p2d_mesh.facesTop) .....
# Within the time-stepping loop, update the boundary condition using .setValue 
as follows:
species_flux_neg_particle_surf.setValue(my_new_BC_value)
# Enjoy not creating new objects

I haven't yet had time to finish producing a new vprof memory consumption plot 
for comparison. However, it's clear from Pympler’s 
SummaryTracker().print_diff() function that this change to the way the BC is 
updated solved the memory leak issue.

For comparison, here are the number of objects and CPU time per timestep 
plotted against three seconds of simulation time, firstly using the 
.faceGrad.constrian() method and, secondly, using the .setValue() method for 
updating. The now-stable number of objects illustrates the fixed leak.
faceGrad.constrain, leaking: https://goo.gl/3LqSm7 setValue, memory leak fixed: 
https://goo.gl/6kQMjH

There is a new issue which also slows the simulation to an unusable level, 
described below.

With the memory leak solved, I was able to run the simulation well beyond three 
seconds, and discovered that the number of sweeps required per timestep begins 
to exponentially increase after around 120s of simulation time. It seems that 
this in turn pulls up the CPU time required per timestep. The plot at the 
following link illustrates the problem: https://goo.gl/G9DD5r

I do not know why this is. It's clear that a memory leak is no longer the cause 
- the number of objects is relatively constant (varying only slightly between 
garbage collector cycles). Plotting, at 1 & 150 seconds into the simulation, 
the residuals returned by the .sweep function for each of the six PDEs being 
solved in the time-stepping loop provides some insight into the stability of 
convergence. Each subplot in a figure is for one of the six PDEs: 
https://goo.gl/Nnm7Si

At 150s, the residuals are still decreasing with sweeping, but at a much slower 
rate, towards the tolerance (1e-4). Do you know why this might be happening?

With best regards,

 - Ian

-----Original Message-----
From: fipy-boun...@nist.gov [mailto:fipy-boun...@nist.gov] On Behalf Of Guyer, 
Jonathan E. Dr. (Fed)
Sent: 11 October 2016 16:37
To: FIPY <FIPY@nist.gov>
Subject: Re: Memory Leakage & Object Build-up with FiPy Sweeps

I have access to their code. Ian, please provide an explicit recipe for 
demonstrating the leak with the code in your github repo.

- Jon

> On Oct 11, 2016, at 11:15 AM, Daniel Wheeler <daniel.wheel...@gmail.com> 
> wrote:
> 
> Hi Ian,
> 
> Could you possible post your code or a version of the code that demonstrates 
> the problem? Also, do you have the same issue with different solver suites?
> 
> Cheers,
> 
> Daniel
> 
> 
> 
> On Fri, Sep 30, 2016 at 12:41 PM, Campbell, Ian <i.campbel...@imperial.ac.uk> 
> wrote:
> Hi All,
> 
>  
> 
> We are sweeping six PDEs in a time-stepping loop. We’ve noticed that as CPU 
> time progresses, the duration of each time-step increases, although the sweep 
> count remains constant. This is illustrated in the Excel file of data logged 
> from the simulation, which is available at the first hyperlink below.
> 
>  
> 
> Hence, we suspected a memory leak may be occurring. After conducting 
> memory-focused line-profiling with the vprof tool, we observed a linear 
> increase in total memory consumption at a rate of approximately 3 MB per 
> timestep loop. This is evident in the graph at the second link below, which 
> illustrates the memory increase over three seconds of simulation.
> 
>  
> 
> As a further step, we used Pympler to investigate the source of RAM 
> consumption increase for each timestep. The table below is an output from 
> Pympler’s SummaryTracker().print_diff(), which describe the additional 
> objects created within every time-step. Clearly, there are ~3.2 MB of 
> additional data being generated with every loop – this correlates perfectly 
> with the total rate of increase of memory consumption reported by vprof. 
> Although we are not yet sure, we suspect that the increasing time spent per 
> loop is the result of this apparent memory leak.
> 
>  
> 
> We suspect this is the result of the calls to .sweep, since we are not 
> explicitly creating these objects. Can the origin of these objects be traced, 
> and furthermore, is there a way to avoid re-creating them and consuming more 
> memory with every loop?  Without some method of unloading or preventing this 
> object build-up, it isn’t feasible to run our simulation for long durations.
> 
> 
> dict
> 
> 2684
> 
> 927.95
> 
> KB
> 
> type
> 
> 1716
> 
> 757.45
> 
> KB
> 
> tuple
> 
> 9504
> 
> 351.31
> 
> KB
> 
> list
> 
> 4781
> 
> 227.09
> 
> KB
> 
> str
> 
> 2582
> 
> 210.7
> 
> KB
> 
> numpy.ndarray
> 
> 396
> 
> 146.78
> 
> KB
> 
> cell
> 
> 3916
> 
> 107.08
> 
> KB
> 
> property
> 
> 2288
> 
> 98.31
> 
> KB
> 
> weakref
> 
> 2287
> 
> 98.27
> 
> KB
> 
> function (getName)
> 
> 1144
> 
> 67.03
> 
> KB
> 
> function (getRank)
> 
> 1144
> 
> 67.03
> 
> KB
> 
> function (_calcValue_)
> 
> 1144
> 
> 67.03
> 
> KB
> 
> function (__init__)
> 
> 1144
> 
> 67.03
> 
> KB
> 
> function (_getRepresentation)
> 
> 1012
> 
> 59.3
> 
> KB
> 
> function (__setitem__)
> 
> 572
> 
> 33.52
> 
> KB
> 
> SUM
> 
> 3285.88
> 
> KB
> 
>  
> 
>  
> 
> https://imperialcollegelondon.box.com/s/zp9jj67du3mxdcfgbc4el8cqpxwnv0
> y4
> 
>  
> 
> https://imperialcollegelondon.box.com/s/ict9tnswqk9z57ovx8r3ll5po5ccri
> b9
> 
>  
> 
> With best regards,
> 
>  
> 
> -          Ian & Krishna
> 
>  
> 
> P.S. Daniel, thank you very much for the excellent example solution you 
> provided in response to our question on obtaining the sharp discontinuity.
> 
>  
> 
> Ian Campbell | PhD Candidate
> 
> Electrochemical Science & Engineering Group
> 
> Imperial College London, SW7 2AZ, United Kingdom
> 
>  
> 
> 
> _______________________________________________
> fipy mailing list
> fipy@nist.gov
> http://www.ctcms.nist.gov/fipy
>   [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]
> 
> 
> 
> 
> --
> Daniel Wheeler
> _______________________________________________
> fipy mailing list
> fipy@nist.gov
> http://www.ctcms.nist.gov/fipy
>  [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]


_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
  [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]

_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
  [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]

_______________________________________________
fipy mailing list
fipy@nist.gov
http://www.ctcms.nist.gov/fipy
  [ NIST internal ONLY: https://email.nist.gov/mailman/listinfo/fipy ]

Reply via email to