On Tue, 15 Dec 2015, Rainer Gerhards wrote:
2015-12-15 8:24 GMT+01:00 David Lang <[email protected]>:
Ok, this shows that the queues at this point in time are pretty empty, but
at some point they had filled up. If they all filled up at the same time (a
reasonable possibility), you had ~23K messages queued at one point.
do you use fixedarray or linkedlist as your queue type?
do you do much parsing of the logs? I don't know exactly how the memory is
accounted for, but if you have a message size of 10K, and have it parsed
into many messages, your $! variables are going to be more than 10K by
themselves. I don't know if $msg and $rawmsg are separate copies in ram or
just pointers into the original message. The ES module would create a large
string of the entire batch it's trying to process before sending it to the
ES server (what's your batch size? you would have hit it right after the
outage where you were unable to deliver to ES)?
it still seems a stretch for 23K messages to use 6G of ram. that's
~250K/message when you have a max message size of 10K (if I've done my math
right). If we were talking a factor of 3x or 4x I would shrug and put it
down to plain overhead, but 25x seems a bit much :-)
Let's see if Rainer has any other thoughts when he wakes up in the next few
hours.
I would prefer if we look into 8.15, simply because I *know* it has
some fixes for memleaks that very seldom show up. If the problem
persists, it makes sense to debug, but I'd prefer not to hunt
potentially fixed bugs...
I was looking for your thoughts on how much memory a message should take as a
multiple of maxmessagesize (see my thoughts above as to extra copies of the
data into variables, output modules templates, etc) as opposed to looking for a
specific leak.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
