On Tuesday, March 13, 2018 at 9:38:05 AM UTC, Dima Pasechnik wrote:
>> I can imagine a small function constructing a matrix (for which it needs
>> to construct the MatrixSpace) to do some computation but not keeping
>> that matrix in memory. For example, the output of that function could be
>> the determinant of the matrix. If this function is called from a loop,
>> you end up in the scenario that I described.
> this example, assuming the matrix size is a parameter, would easily become
> a memory leak,
> just imagine calling this function millions of times, for 10 different
> fields, with random matrix sizes.
Indeed. The strong LRU/FIFO cache needs to be placed very carefully. It
produces the kind of optimization that a memory pool gives. In the latter
there's a natural granularity: block size. Here it's just time since use.
The random matrix sizes give a good example where this caching is
counterproductive: by keeping parents around that aren't used anymore,
you're preventing quick memory reuse so one would actually be deteriorating
runtime (even before leaking becomes a real problem) because you'd be
making life harder for the memory cache of the processor.
Note that, with the missing LRU/FIFO there's a viable strategy to
counteracting the overhead: profile the code, note that some parent
construction is a bottleneck, identify which parent (that would be clear
from the call graph), and construct that parent before the loop and keep a
Counterproductive behaviour of the LRU/FIFO would be much harder to detect
and even harder to work around.
I recall that for exactly the reason of avoiding expensive parent creation,
we used to have deferred full parent initialization of matrix spaces;
exactly for the scenario where a matrix only gets created as a temporary
thing, for instance for taking its determinant. The advantage of that
approach is that you even gain on the first ephemeral use. Did we lose that
for a reason? Perhaps full parent initialization isn't as expensive as it
used to be? (I have trouble believing that, with the introduction of the
So: the implementation of the CachedWeakValueDictionary on the ticket is
fine, but I doubt that it's actually the appropriate solution for the
problems sketched. We'd need to have actual instances of the problem to
see. Probably some modular forms code somewhere will be quite linear
algebra intensive. The code on https://trac.sagemath.org/ticket/15113 would
be another example.
You received this message because you are subscribed to the Google Groups
To unsubscribe from this group and stop receiving emails from it, send an email
To post to this group, send email to email@example.com.
Visit this group at https://groups.google.com/group/sage-devel.
For more options, visit https://groups.google.com/d/optout.