Hi, I created an issue as recommended by Rainer: https://github.com/rsyslog/rsyslog/issues/709
@David, please find the full config here, but probably this can be reproduced with a much simpler config: https://gist.github.com/hakman/44afddaf4eb67cda28c6 Thanks, Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Dec 16, 2015 at 8:08 PM, David Lang <[email protected]> wrote: > what modules do you use? I've been running 8.15-pre git builds for a while > (tracking down the segfaults that Rainer just traced down to json-c) and > have had no memory issues. > > But I'm not using imfile or omelasticsearch. > > David Lang > > On Wed, 16 Dec 2015, Ciprian Hacman wrote: > > Date: Wed, 16 Dec 2015 18:00:48 +0200 >> From: Ciprian Hacman <[email protected]> >> Reply-To: rsyslog-users <[email protected]> >> To: rsyslog-users <[email protected]> >> Subject: Re: [rsyslog] Rsyslog using huge amounts of memory >> >> Not sure if you saw, but provided output from Valgrind with bout inotify >> and polling. >> To me it seems that in both cases the leak can be seen. >> >> Thanks, >> Ciprian >> >> -- >> Performance Monitoring * Log Analytics * Search Analytics >> Solr & Elasticsearch Support * http://sematext.com/ >> >> On Wed, Dec 16, 2015 at 5:23 PM, Rainer Gerhards < >> [email protected]> >> wrote: >> >> 2015-12-16 16:21 GMT+01:00 Ciprian Hacman <[email protected]>: >>> >>>> Not sure how easy is to reproduce on your side. >>>> If you need help narrowing the leak to a certain commit or release, I >>>> >>> could >>> >>>> help. >>>> >>> >>> OK, I thought you could just switch the imfile mode. So I suggest to >>> open a bug tracker, I guess I can look into it after the holiday >>> period. >>> >>> Rainer >>> >>>> >>>> Ciprian >>>> >>>> -- >>>> Performance Monitoring * Log Analytics * Search Analytics >>>> Solr & Elasticsearch Support * http://sematext.com/ >>>> >>>> On Wed, Dec 16, 2015 at 4:22 PM, Rainer Gerhards < >>>> >>> [email protected]> >>> >>>> wrote: >>>> >>>> 2015-12-16 15:10 GMT+01:00 Ciprian Hacman <[email protected] >>>>> >>>> : >>>> >>>>> Done. Can you check the gist again? >>>>>> >>>>> >>>>> yup. It now contains better info, but I wonder where these leaks stem >>>>> from. One suspect is inotify polling mode, which we didn't really test >>>>> well for quite a while... >>>>> >>>>> Rainer >>>>> >>>>>> >>>>>> Thanks, >>>>>> Ciprian >>>>>> >>>>>> -- >>>>>> Performance Monitoring * Log Analytics * Search Analytics >>>>>> Solr & Elasticsearch Support * http://sematext.com/ >>>>>> >>>>>> On Wed, Dec 16, 2015 at 3:40 PM, Rainer Gerhards < >>>>>> >>>>> [email protected]> >>>>> >>>>>> wrote: >>>>>> >>>>>> 2015-12-16 14:16 GMT+01:00 Ciprian Hacman < >>>>>>> >>>>>> [email protected] >>> >>>> : >>>>>> >>>>>>> Done! I updated the output in the same place: >>>>>>>> https://gist.github.com/hakman/44afddaf4eb67cda28c6 >>>>>>>> >>>>>>> >>>>>>> mhhh... less clear than I had hoped... >>>>>>> >>>>>>> Can you switch imfile to use inotify mode and retry? >>>>>>> >>>>>>> Rainer >>>>>>> >>>>>>>> >>>>>>>> Thanks for looking into this so fast! >>>>>>>> >>>>>>>> Ciprian >>>>>>>> >>>>>>>> -- >>>>>>>> Performance Monitoring * Log Analytics * Search Analytics >>>>>>>> Solr & Elasticsearch Support * http://sematext.com/ >>>>>>>> >>>>>>>> On Wed, Dec 16, 2015 at 2:53 PM, Rainer Gerhards < >>>>>>>> >>>>>>> [email protected]> >>>>>>> >>>>>>>> wrote: >>>>>>>> >>>>>>>> 2015-12-16 13:48 GMT+01:00 Ciprian Hacman < >>>>>>>>> >>>>>>>> [email protected] >>>>> >>>>>> : >>>>>>>> >>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I upgraded a server to Rsyslog 8.15 last night and today the >>>>>>>>>> >>>>>>>>> process >>>>> >>>>>> was >>>>>>> >>>>>>>> using almost 200MB of RAM (raising steadily). >>>>>>>>>> Tried running the process in Valgrind to see if I get an idea >>>>>>>>>> >>>>>>>>> about >>> >>>> what >>>>>>> >>>>>>>> happens, but wasn't that much help for me. >>>>>>>>>> >>>>>>>>> >>>>>>>>> That's because debug symbols are unloaded at module unload time. >>>>>>>>> >>>>>>>> This >>> >>>> makes valgrind stacktraces unusable. Nevertheless, the information >>>>>>>>> looks very promising. >>>>>>>>> >>>>>>>>> Can you build rsyslog yourself for that box? All we need is the >>>>>>>>> --enable-valgrind option, which will essentially remove the module >>>>>>>>> unloads and make the stacktrace usable. >>>>>>>>> >>>>>>>>> Rainer >>>>>>>>> >>>>>>>>>> >>>>>>>>>> If someone has better debugging skills, I pasted the output >>>>>>>>>> >>>>>>>>> here. >>> >>>> Not >>>>> >>>>>> sure >>>>>>>>> >>>>>>>>>> if I let it run enough or leave it running longer. >>>>>>>>>> https://gist.github.com/hakman/44afddaf4eb67cda28c6 >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Ciprian >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Performance Monitoring * Log Analytics * Search Analytics >>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/ >>>>>>>>>> >>>>>>>>>> On Tue, Dec 15, 2015 at 2:41 PM, David Lang <[email protected]> >>>>>>>>>> >>>>>>>>> wrote: >>> >>>> >>>>>>>>>> On Tue, 15 Dec 2015, Ciprian Hacman wrote: >>>>>>>>>>> >>>>>>>>>>> Hi David, >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> maxMessageSize="10000" >>>>>>>>>>>> queue.size="10000" - main queue >>>>>>>>>>>> queue.size="10000" - elasticsearch queue >>>>>>>>>>>> >>>>>>>>>>>> Based on my calculations this brings me to a max of 200MB of >>>>>>>>>>>> >>>>>>>>>>> memory, >>>>> >>>>>> maybe >>>>>>>>> >>>>>>>>>> a little more depending on how maxMessageSize is calculated. >>>>>>>>>>>> >>>>>>>>>>>> I read logs from a file and push them to Elasticsearch (on the >>>>>>>>>>>> >>>>>>>>>>> same >>>>> >>>>>> network), so TCP is the only possibility. This server has a >>>>>>>>>>>> >>>>>>>>>>> very >>> >>>> simple >>>>>>> >>>>>>>> setup. >>>>>>>>>>>> >>>>>>>>>>>> If I don't find the reason for this issue, I might have to go >>>>>>>>>>>> >>>>>>>>>>> implement >>>>>>> >>>>>>>> the >>>>>>>>>>>> forwarding to a central location and push to Elasticsearch >>>>>>>>>>>> >>>>>>>>>>> from >>> >>>> there. >>>>>>> >>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> There are advantages to sending things to a central server. >>>>>>>>>>> >>>>>>>>>>> it's one place to queue data, so you can either allocate more >>>>>>>>>>> >>>>>>>>>> ram, >>> >>>> or go >>>>>>> >>>>>>>> to disk as needed without impacting other workloads. >>>>>>>>>>> >>>>>>>>>>> it's more efficient, the central server is more likely to have >>>>>>>>>>> >>>>>>>>>> larger >>>>> >>>>>> batches of data to feed to ES, and ES only needs to be running >>>>>>>>>>> >>>>>>>>>> one >>> >>>> thread >>>>>>>>> >>>>>>>>>> processing inbound data >>>>>>>>>>> >>>>>>>>>>> while it is one more point to have to look at, I think it >>>>>>>>>>> >>>>>>>>>> simplifies >>>>> >>>>>> troubleshooting as all the communication to ES (and therefor >>>>>>>>>>> >>>>>>>>>> all >>> >>>> the >>>>> >>>>>> errors >>>>>>>>> >>>>>>>>>> for such communication) happen in one place instead of >>>>>>>>>>> >>>>>>>>>> distributed. >>>>> >>>>>> >>>>>>>>>>> anyway, let's see how things look with 8.15 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> David Lang >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED >>>>>>>>>>> >>>>>>>>>> by a >>> >>>> myriad >>>>>>> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT >>>>>>>>>>> >>>>>>>>>> POST if >>> >>>> you >>>>>>> >>>>>>>> DON'T LIKE THAT. >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>> rsyslog mailing list >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by >>>>>>>>>> >>>>>>>>> a >>> >>>> myriad >>>>>>> >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>> >>>>>>>> you >>>>> >>>>>> DON'T LIKE THAT. >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> >>>>>>>> myriad >>>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>> >>>>>>>> you >>>>> >>>>>> DON'T LIKE THAT. >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>> rsyslog mailing list >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>> >>>>>>> myriad >>>>> >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> >>>>>> you >>> >>>> DON'T LIKE THAT. >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> >>>>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>> >>>>>> you >>> >>>> DON'T LIKE THAT. >>>>>>> >>>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> >>>>> myriad >>> >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>> myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
