Agreed. Will wait for the 8.15 release and get back to you if this reproduces.
@David: Just want to respond also to your questions, in case it helps someday: - message size is small, usually around 200 chars. - queue type is FixedArray, but I plan to change to LinkedList. Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Dec 15, 2015 at 9:29 AM, Rainer Gerhards <[email protected]> wrote: > 2015-12-15 8:24 GMT+01:00 David Lang <[email protected]>: > > Ok, this shows that the queues at this point in time are pretty empty, > but > > at some point they had filled up. If they all filled up at the same time > (a > > reasonable possibility), you had ~23K messages queued at one point. > > > > do you use fixedarray or linkedlist as your queue type? > > > > do you do much parsing of the logs? I don't know exactly how the memory > is > > accounted for, but if you have a message size of 10K, and have it parsed > > into many messages, your $! variables are going to be more than 10K by > > themselves. I don't know if $msg and $rawmsg are separate copies in ram > or > > just pointers into the original message. The ES module would create a > large > > string of the entire batch it's trying to process before sending it to > the > > ES server (what's your batch size? you would have hit it right after the > > outage where you were unable to deliver to ES)? > > > > it still seems a stretch for 23K messages to use 6G of ram. that's > > ~250K/message when you have a max message size of 10K (if I've done my > math > > right). If we were talking a factor of 3x or 4x I would shrug and put it > > down to plain overhead, but 25x seems a bit much :-) > > > > Let's see if Rainer has any other thoughts when he wakes up in the next > few > > hours. > > I would prefer if we look into 8.15, simply because I *know* it has > some fixes for memleaks that very seldom show up. If the problem > persists, it makes sense to debug, but I'd prefer not to hunt > potentially fixed bugs... > > Rainer > > > > David Lang > > > > On Tue, 15 Dec 2015, Ciprian Hacman wrote: > > > >> Date: Tue, 15 Dec 2015 08:43:52 +0200 > >> From: Ciprian Hacman <[email protected]> > >> Reply-To: rsyslog-users <[email protected]> > >> To: rsyslog-users <[email protected]> > >> Subject: Re: [rsyslog] Rsyslog using huge amounts of memory > >> > >> > >> Hi Peter, > >> > >> Just checked and impstats were actually enabled. Here is the sample > before > >> restart: > >> > >> Dec 14 09:25:09 host1 rsyslogd-pstats: omelasticsearch: > >> origin=omelasticsearch submitted=198115631 failed.http=260 > >> failed.httprequests=2 failed.es=58720 > >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-norm-direct: > >> origin=core.action > >> processed=197076777 failed=0 suspended=0 suspended.duration=0 resumed=0 > >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-es-direct: > origin=core.action > >> processed=197076777 failed=0 suspended=0 suspended.duration=0 resumed=0 > >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-norm: origin=core.action > >> processed=1040122 failed=0 suspended=0 suspended.duration=0 resumed=0 > >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-es: origin=core.action > >> processed=1039926 failed=0 suspended=0 suspended.duration=0 resumed=0 > >> Dec 14 09:25:09 host1 rsyslogd-pstats: resource-usage: origin=impstats > >> utime=5859695897 stime=206002475 maxrss=6587196 minflt=15802757 > >> majflt=19279 inblock=3838232 oublock=2238104 nvcsw=5667006 nivcsw=596991 > >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-es-direct queue: > >> origin=core.queue size=75 enqueued=197076777 full=1742 > discarded.full=1423 > >> discarded.nf=0 maxqsize=10000 > >> Dec 14 09:25:09 host1 rsyslogd-pstats: test-es queue: origin=core.queue > >> size=0 enqueued=1039926 full=0 discarded.full=0 discarded.nf=0 > >> maxqsize=2888 > >> Dec 14 09:25:09 host1 rsyslogd-pstats: main Q: origin=core.queue size=27 > >> enqueued=198705203 full=0 discarded.full=0 discarded.nf=0 maxqsize=9765 > >> > >> > >> Ciprian > >> > >> -- > >> Performance Monitoring * Log Analytics * Search Analytics > >> Solr & Elasticsearch Support * http://sematext.com/ > >> > >> On Tue, Dec 15, 2015 at 7:57 AM, Peter Portante > >> <[email protected]> > >> wrote: > >> > >>> On Tue, Dec 15, 2015 at 12:30 AM, Ciprian Hacman < > >>> [email protected]> wrote: > >>> > >>>> Hi David, > >>>> > >>>> maxMessageSize="10000" > >>>> queue.size="10000" - main queue > >>>> queue.size="10000" - elasticsearch queue > >>>> > >>>> Based on my calculations this brings me to a max of 200MB of memory, > >>> > >>> maybe > >>>> > >>>> a little more depending on how maxMessageSize is calculated. > >>>> > >>>> I read logs from a file and push them to Elasticsearch (on the same > >>>> network), so TCP is the only possibility. This server has a very > simple > >>>> setup. > >>>> > >>>> If I don't find the reason for this issue, I might have to go > implement > >>> > >>> the > >>>> > >>>> forwarding to a central location and push to Elasticsearch from there. > >>>> > >>>> Thanks, > >>>> Ciprian > >>>> > >>>> -- > >>>> Performance Monitoring * Log Analytics * Search Analytics > >>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>> > >>>> On Tue, Dec 15, 2015 at 12:52 AM, David Lang <[email protected]> wrote: > >>>> > >>>>> what is your maxmessagesize and your max queue size? rsyslog will use > >>> > >>> up > >>>>> > >>>>> to maxmessagesize*maxqueuesize ram to buffer messages if they can't > be > >>>>> delivered. > >>>>> > >>>>> you probably want to set these values smaller rather than setting > >>>>> something up to kill rsyslog when it gets large. > >>>>> > >>>>> What is the transport you use to deliver the logs from these systems? > >>>>> > >>>>> I like to setup log redundant log relay servers in each datacenter > and > >>>>> then have all the systems log to these relays via UDP. UDP is > reliable > >>>> > >>>> over > >>>>> > >>>>> a local network, but if there is a problem with the receiving system, > >>> > >>> it > >>>>> > >>>>> will go ahead and loose logs rather than affecting the sending > system. > >>>>> > >>>>> David Lang > >>>>> > >>>>> > >>>>> > >>>>> On Mon, 14 Dec 2015, Ciprian Hacman wrote: > >>>>> > >>>>> Hi David, > >>>>>> > >>>>>> > >>>>>> Yes, killing Rsyslog is a last resort, but for most people I think > >>>>>> shipping > >>>>>> logs is a secondary function on a server. Would prefer that Rsyslog > >>>>>> doesn't > >>>>>> interfere with other apps. > >>>>>> > >>>>>> I usually enable impstats, though on these particular server the > >>> > >>> queues > >>>>>> > >>>>>> are > >>>>>> really tiny so that it doesn't use that much memory. I would expect > >>> > >>> some > >>>>>> > >>>>>> memory usage fluctuations when Elasticsearch doesn't respond, but > >>>> > >>>> nothing > >>>>>> > >>>>>> as extreme as using 6GB of memory. > >>>>>> > >>>>>> If changes in 8.15 don't help, I think I have to spend a few hours > >>>> > >>>> trying > >>>>>> > >>>>>> to reproduce this. > >>>>>> > >>>>>> Thanks, > >>>>>> Ciprian > >>>>>> > >>>>>> -- > >>>>>> Performance Monitoring * Log Analytics * Search Analytics > >>>>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>>>> > >>>>>> On Mon, Dec 14, 2015 at 8:17 PM, David Lang <[email protected]> wrote: > >>>>>> > >>>>>> On Mon, 14 Dec 2015, Ciprian Hacman wrote: > >>>>>>> > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>>> > >>>>>>>> We are noticing some Rsyslog instances that push about 500MB of > logs > >>>>>>>> daily > >>>>>>>> to an Elasticsearch instance, so not that much. We noticed one of > >>> > >>> the > >>>>>>>> > >>>>>>>> Rsyslog processes using about 6GB of RAM. Usually this is less > than > >>>> > >>>> 1MB. > >>>>>>>> > >>>>>>>> > >>>>>>>> I plan on debugging this in the next few days, but wanted to see > >>> > >>> also > >>>> > >>>> if > >>>>>>>> > >>>>>>>> there is a good idea to add a RSS limit (doable in Upstart and > >>>> > >>>> Systemd) > >>>>>>>> > >>>>>>>> and > >>>>>>>> kill / restart Rsyslog when it gets into such a situation. > >>>>>>>> > >>>>>>>> > >>>>>>> killing/restarting rsyslog is a last resort. large memory usage > >>> > >>> usually > >>>>>>> > >>>>>>> means that you have lots of logs that aren't delivered and are > >>> > >>> sitting > >>>>>>> > >>>>>>> in a > >>>>>>> queue somewhere. > >>>>>>> > >>>>>>> do you have impstats configured? > >>>> > >>>> > >>> > >>> Ciprian, are you going to enable impstats? I'd be curious to know > what I > >>> shows. > >>> > >>> Thanks, -peter > >>> > >>> > >>> > >>>> if not, it's a _really_ good idea to > >>>>>>> > >>>>>>> configure it and have it write either directly to a file (log > >>> > >>> rotation > >>>> > >>>> of > >>>>>>> > >>>>>>> this file is a bit of an issue) or to it's own ruleset. either way > >>>> > >>>> means > >>>>>>> > >>>>>>> that a blockage in normal log processing will not affect the pstats > >>>> > >>>> logs. > >>>>>>> > >>>>>>> These logs will show you if you have queues building up and where. > >>>>>>> > >>>>>>> David Lang > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> > >>>> myriad > >>>>>>> > >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>> > >>> you > >>>>>>> > >>>>>>> DON'T LIKE THAT. > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>> > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com/professional-services/ > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>> > >>> myriad > >>>>>> > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>>> DON'T LIKE THAT. > >>>>>> > >>>>>> _______________________________________________ > >>>>> > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>> > >>> myriad > >>>>> > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>> DON'T LIKE THAT. > >>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >> LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
