forgotten reply for this one: 2017-01-19 18:28 GMT+01:00 mostolog--- via rsyslog < [email protected]>:
> Really interesting and instructive. Let me see if I understood properly: > > Considering the below explanation, an idle (not going to receive more > messages the next 7 days) imrelp rsyslog...shouldn't memory be freed from > current 512MB usage in the near future? > For the scenario I have described: no, as long as the high address memory block is allocated, nothing is freed. Not even after 70 days of idleness ;-) > > You mentioned forcing each 100.000 messages, but that's never gonna happen > if we are still at 20k, and no more messages coming. > right Rainer > > Regards > > > El 19/01/17 a las 11:16, Rainer Gerhards escribió: > >> 2017-01-19 10:19 GMT+01:00 mostolog--- via rsyslog < >> [email protected] <mailto:[email protected]>>: >> >> >> with very few exceptions, rsyslog releases the memory as it >> goes, there should not be any significant amount of memory >> freed by rsyslog after it's been idle for a while. >> >> But being idle for 15m should release memory from 512MB to a few >> KB if they aren't used, isnt it? >> >> >> Memory alloc is not as simple as it seems ;-) >> >> First, clib does not always do a proper cleanup. I guess it seems not to >> consolidate free space in all cases. There is a clib call to force this, >> and rsyslog does it from time to time (I think every 100,000 messages). We >> do not do it very frequently, because it is an expensive operation. Also >> note that memory is reusable internally, so even though it is not returned >> to the OS, further alloc requests inside rsyslog can use this memory and do >> so. Returning memory to the OS and re-claiming it is expensive. Thus you do >> want to keep some memory allocated but internally unused to avoid doing >> this operation too frequently. >> >> Secondly, memory alloc from the OS is done by sbrk[1] IIRC. The important >> point is that we need to alloc and free memory in sequence. This means if >> you alloc 100MB, than alloc 1MB, you have the following memory layout >> >> BASE >> 100MB >> 1MB >> >> with the break at BASE+101MB. If you now free the 100MB chunk, you have >> this layout: >> >> BASE >> 100MB free >> 1MB >> >> if you want to return to the OS, you'd need to copy down the 1MB to >> immediatly after BASE, because otherwise you cannot reset the break to >> BASE+1MB. The allocator does not do this, it would totally wrek performance >> (note: that is not rsyslog specific, that is how the *C runtime* works). >> More importantly, it would mean all pointers to it would need to be >> updated. And the runtime does not know where these pointers are located. So >> it would not only costly, compaction is simply impossible. Let's assume we >> now alloc another 10 MB. Then we have >> >> BASE >> 10MB >> 90MB free >> 1MB >> >> because the allocator uses the free, but still allocated mem. Now, let's >> assume we free the 1MB chunk, we get: >> >> BASE >> 10MB >> 91MB free >> >> Now the free space is at the end of the data segment. So the alloc >> subsystem has the choice to reduce memory alloc from the OS. It may or may >> not dealloc. I don't know the exact rules, but the important thing is that >> the alloc system uses some heuristic (plus the call I mentioned) to decide >> if to dealloc. Let's assume it does. Then it reduceds the data segement >> size and we get to >> >> BASE >> 10MB >> >> effectively reducing rss by 91 MB. >> >> IMHO the alloc system strongly works on the assumption that memory >> allocated (from the OS) but free internally does not really hurt, as it is >> just virtual address space, which, if actually unused, is paged out to disk >> once and then doesn't matter at all until it get's reused again. >> >> OF COURSE if we have constantly growing memory, the app seems not to free >> some chunks, so the alloc system doesn't know they are free. This has >> nothing to do with what I explained. What I explained just means that an >> app may do proper free()'s, but the rss size doesn't reflect this. The >> typical (and often visible) effect of that is that the app grows to a >> certain size and then remains at it (no growth, no decrease). This is >> where, via the clib call, we force free. >> >> There is one important point, though: if we have a memory block at the >> far end of the data segment, we cannot return mem to the OS until this >> block has been freed. In rsyslog, you typical see this if >> >> a) old-style "last message repeated n times" is active >> b) an infrequently written-to output stores a message object when queues >> are very full >> c) no different messages are written to that output for a long time >> >> What here is very probably that in b) a high memory address is used for >> that memory block. As due to c) we do not have any traffic, that high >> address is kept in use for hours, maybe days. Once a different message >> arrives at the output, the old object is processed and freed. This can >> result in a very sharp decrease of rss size, especially if the system has >> low load during such times. Let's assume this layout: >> >> BASE >> 10MB used >> 1.5GB free, but allocated from OS >> 5k msg object >> >> If we now free those 5k, most probably the alloc system heuristic will >> immediately return 1.5GB+5K to the OS, reducing the rss size accordingly. >> >> Bottom line: not everything that looks like a memory leak actually is one. >> >> HTH >> [1] https://linux.die.net/man/2/sbrk >> > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
