2017-01-19 10:19 GMT+01:00 mostolog--- via rsyslog < [email protected]>:
> with very few exceptions, rsyslog releases the memory as it goes, there >> should not be any significant amount of memory freed by rsyslog after it's >> been idle for a while. >> > But being idle for 15m should release memory from 512MB to a few KB if > they aren't used, isnt it? > > Memory alloc is not as simple as it seems ;-) First, clib does not always do a proper cleanup. I guess it seems not to consolidate free space in all cases. There is a clib call to force this, and rsyslog does it from time to time (I think every 100,000 messages). We do not do it very frequently, because it is an expensive operation. Also note that memory is reusable internally, so even though it is not returned to the OS, further alloc requests inside rsyslog can use this memory and do so. Returning memory to the OS and re-claiming it is expensive. Thus you do want to keep some memory allocated but internally unused to avoid doing this operation too frequently. Secondly, memory alloc from the OS is done by sbrk[1] IIRC. The important point is that we need to alloc and free memory in sequence. This means if you alloc 100MB, than alloc 1MB, you have the following memory layout BASE 100MB 1MB with the break at BASE+101MB. If you now free the 100MB chunk, you have this layout: BASE 100MB free 1MB if you want to return to the OS, you'd need to copy down the 1MB to immediatly after BASE, because otherwise you cannot reset the break to BASE+1MB. The allocator does not do this, it would totally wrek performance (note: that is not rsyslog specific, that is how the *C runtime* works). More importantly, it would mean all pointers to it would need to be updated. And the runtime does not know where these pointers are located. So it would not only costly, compaction is simply impossible. Let's assume we now alloc another 10 MB. Then we have BASE 10MB 90MB free 1MB because the allocator uses the free, but still allocated mem. Now, let's assume we free the 1MB chunk, we get: BASE 10MB 91MB free Now the free space is at the end of the data segment. So the alloc subsystem has the choice to reduce memory alloc from the OS. It may or may not dealloc. I don't know the exact rules, but the important thing is that the alloc system uses some heuristic (plus the call I mentioned) to decide if to dealloc. Let's assume it does. Then it reduceds the data segement size and we get to BASE 10MB effectively reducing rss by 91 MB. IMHO the alloc system strongly works on the assumption that memory allocated (from the OS) but free internally does not really hurt, as it is just virtual address space, which, if actually unused, is paged out to disk once and then doesn't matter at all until it get's reused again. OF COURSE if we have constantly growing memory, the app seems not to free some chunks, so the alloc system doesn't know they are free. This has nothing to do with what I explained. What I explained just means that an app may do proper free()'s, but the rss size doesn't reflect this. The typical (and often visible) effect of that is that the app grows to a certain size and then remains at it (no growth, no decrease). This is where, via the clib call, we force free. There is one important point, though: if we have a memory block at the far end of the data segment, we cannot return mem to the OS until this block has been freed. In rsyslog, you typical see this if a) old-style "last message repeated n times" is active b) an infrequently written-to output stores a message object when queues are very full c) no different messages are written to that output for a long time What here is very probably that in b) a high memory address is used for that memory block. As due to c) we do not have any traffic, that high address is kept in use for hours, maybe days. Once a different message arrives at the output, the old object is processed and freed. This can result in a very sharp decrease of rss size, especially if the system has low load during such times. Let's assume this layout: BASE 10MB used 1.5GB free, but allocated from OS 5k msg object If we now free those 5k, most probably the alloc system heuristic will immediately return 1.5GB+5K to the OS, reducing the rss size accordingly. Bottom line: not everything that looks like a memory leak actually is one. HTH [1] https://linux.die.net/man/2/sbrk _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
