On Thu, 2009-01-29 at 21:13 -0800, [email protected] wrote: > On Thu, 29 Jan 2009, [email protected] wrote: > interesting note on memory useage. > > I'm using the default fixed array queue type on this box with a 1K max > message length. if I hammer the box with a steady ~120K messages/sec > (while it can write 93K/sec) the queue builds up to where it takes ~12G of > ram. at this point the throughput takes a nose dive (not just dropping > inbound packets, but also the number of packets written is much less) > > if I kill the sender, it starts emptying it's queue (interestingly, not > quite as fast as if it is also recieving some messages), but the memory > isn't freed up until I start sending it messages again.
This actually is expected behavior - and it has lots to do with "last message repeated n time". In order to implement that functionality, I need to hold on the the last message until a new one comes in (so that I can compare new to old). As such, a message that is fully processed can not immediately be freed. This happens, when the next message comes in - whenever this be. Note that each output has separate "last message..." status, so each action keeps a copy of the previous message until a new one arrives. What now happens is that when the queue builds up, malloc extends the data segment size. It is fair to assume that the last message received - on a very busy system will probably end up at a high location in the data segment (but note it is just a probability - it may even receive a very low location, if that was just freed immediately before). When the queue is now drained, we free everything but this message. As the message is still referenced for "last m...", it can not be freed. As it has a high address, the data segment size can not be reduced. As such, rsyslog still holds the whole data segement, with it containing almost no actually allocated memory. I do not know if the runtime system has a way to tell the OS it now uses a "sparse data segement", but I guess it doesn't do that. When the next message comes in (hours later?), the previous message can be freed, and the runtime can then reduce the data segment size (which should result in a sharp decrease of memory usage seen). This is one of the reasons I don't like "last message...". I hope this clarifies. Rainer _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
