Really interesting and instructive. Let me see if I understood properly:
Considering the below explanation, an idle (not going to receive more
messages the next 7 days) imrelp rsyslog...shouldn't memory be freed
from current 512MB usage in the near future?
You mentioned forcing each 100.000 messages, but that's never gonna
happen if we are still at 20k, and no more messages coming.
Regards
El 19/01/17 a las 11:16, Rainer Gerhards escribió:
2017-01-19 10:19 GMT+01:00 mostolog--- via rsyslog
<[email protected] <mailto:[email protected]>>:
with very few exceptions, rsyslog releases the memory as it
goes, there should not be any significant amount of memory
freed by rsyslog after it's been idle for a while.
But being idle for 15m should release memory from 512MB to a few
KB if they aren't used, isnt it?
Memory alloc is not as simple as it seems ;-)
First, clib does not always do a proper cleanup. I guess it seems not
to consolidate free space in all cases. There is a clib call to force
this, and rsyslog does it from time to time (I think every 100,000
messages). We do not do it very frequently, because it is an expensive
operation. Also note that memory is reusable internally, so even
though it is not returned to the OS, further alloc requests inside
rsyslog can use this memory and do so. Returning memory to the OS and
re-claiming it is expensive. Thus you do want to keep some memory
allocated but internally unused to avoid doing this operation too
frequently.
Secondly, memory alloc from the OS is done by sbrk[1] IIRC. The
important point is that we need to alloc and free memory in sequence.
This means if you alloc 100MB, than alloc 1MB, you have the following
memory layout
BASE
100MB
1MB
with the break at BASE+101MB. If you now free the 100MB chunk, you
have this layout:
BASE
100MB free
1MB
if you want to return to the OS, you'd need to copy down the 1MB to
immediatly after BASE, because otherwise you cannot reset the break to
BASE+1MB. The allocator does not do this, it would totally wrek
performance (note: that is not rsyslog specific, that is how the *C
runtime* works). More importantly, it would mean all pointers to it
would need to be updated. And the runtime does not know where these
pointers are located. So it would not only costly, compaction is
simply impossible. Let's assume we now alloc another 10 MB. Then we have
BASE
10MB
90MB free
1MB
because the allocator uses the free, but still allocated mem. Now,
let's assume we free the 1MB chunk, we get:
BASE
10MB
91MB free
Now the free space is at the end of the data segment. So the alloc
subsystem has the choice to reduce memory alloc from the OS. It may or
may not dealloc. I don't know the exact rules, but the important thing
is that the alloc system uses some heuristic (plus the call I
mentioned) to decide if to dealloc. Let's assume it does. Then it
reduceds the data segement size and we get to
BASE
10MB
effectively reducing rss by 91 MB.
IMHO the alloc system strongly works on the assumption that memory
allocated (from the OS) but free internally does not really hurt, as
it is just virtual address space, which, if actually unused, is paged
out to disk once and then doesn't matter at all until it get's reused
again.
OF COURSE if we have constantly growing memory, the app seems not to
free some chunks, so the alloc system doesn't know they are free. This
has nothing to do with what I explained. What I explained just means
that an app may do proper free()'s, but the rss size doesn't reflect
this. The typical (and often visible) effect of that is that the app
grows to a certain size and then remains at it (no growth, no
decrease). This is where, via the clib call, we force free.
There is one important point, though: if we have a memory block at the
far end of the data segment, we cannot return mem to the OS until this
block has been freed. In rsyslog, you typical see this if
a) old-style "last message repeated n times" is active
b) an infrequently written-to output stores a message object when
queues are very full
c) no different messages are written to that output for a long time
What here is very probably that in b) a high memory address is used
for that memory block. As due to c) we do not have any traffic, that
high address is kept in use for hours, maybe days. Once a different
message arrives at the output, the old object is processed and freed.
This can result in a very sharp decrease of rss size, especially if
the system has low load during such times. Let's assume this layout:
BASE
10MB used
1.5GB free, but allocated from OS
5k msg object
If we now free those 5k, most probably the alloc system heuristic will
immediately return 1.5GB+5K to the OS, reducing the rss size accordingly.
Bottom line: not everything that looks like a memory leak actually is one.
HTH
[1] https://linux.die.net/man/2/sbrk
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
