The problem is real.
I have an application which generates one line of about 300 KB :-(
In this case, Apache eats up all the memory (about 1 GB), and never
gives it back to the OS - game over - reboot !
How could this be solved?
Thanks,
Nick
Dan Poirier wrote:
Nick Gearls <[email protected]> writes:
There seems to be a memory problem when substituing something on very
long lines (several hundreds KB). this problem is different from bug
44948 (I applied this patch).
When using something like "Substitute s/string1/string2/" on a 300 KB
line with 30 times "string1" on the line, there are 2 problems:
1. It uses a huge amount of memory, as - I think - it reallocates a
new buffer (300 KB) before each substitute without freeing the
previous one
I think you're right - most of the working memory is not released until
the request is over. This is kind of a worst-case for mod_substitute's
temporary memory usage.
Is this a problem in real cases? Or just in artificial testing?
If these are real requests, maybe if you tell a little more about them,
we can find a better solution.
2. The memory is not freeed at the end of the HTTP request.
Maybe is it due to Keep-alive?
If by "the memory is not freed", you mean the memory usage for the
http process as reported by the OS doesn't go down, that's right.
But at the end of the request, the memory should be released back
to Apache to re-use. You can test this by repeating your test several
times - after the first time, the process's memory usage should not keep
going up the way it does the first time.
If that's a problem in your case, you can set MaxMemFree to ask Apache
to release memory by calling free() above a certain threshold. Even
then, whether the process releases the memory back to the OS depends on
the OS, so it might not gain you anything.
But if this is going to happen very often, you might as well let Apache
hold on to the memory; it's just going to grab it back again the next
time this happens.
If using the "q" switch to not flatten the buckets, it uses almost no
memory. Btw, it correctly handles more 60 substitutions on the same
line (some shorter, some longer) without the q flag !?! When exactly
is this flag needed?
Don't know about this one.
In macro SEDSCAT, we have
s2 = apr_pstrmemdup(pool, buff, blen);
s1 = apr_pstrcat(pool, s1, s2, NULL);
Shouldn't we free the previous s1 buffer?
Pools don't provide a way of freeing individual allocations; you have
to destroy the whole pool.
Anyway, I do not understand why the memory is not released, as the
pool is supposed to be destroyed.
See above.
