Not sure if you saw, but provided output from Valgrind with bout inotify and polling. To me it seems that in both cases the leak can be seen.
Thanks, Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Wed, Dec 16, 2015 at 5:23 PM, Rainer Gerhards <[email protected]> wrote: > 2015-12-16 16:21 GMT+01:00 Ciprian Hacman <[email protected]>: > > Not sure how easy is to reproduce on your side. > > If you need help narrowing the leak to a certain commit or release, I > could > > help. > > OK, I thought you could just switch the imfile mode. So I suggest to > open a bug tracker, I guess I can look into it after the holiday > period. > > Rainer > > > > Ciprian > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > On Wed, Dec 16, 2015 at 4:22 PM, Rainer Gerhards < > [email protected]> > > wrote: > > > >> 2015-12-16 15:10 GMT+01:00 Ciprian Hacman <[email protected] > >: > >> > Done. Can you check the gist again? > >> > >> yup. It now contains better info, but I wonder where these leaks stem > >> from. One suspect is inotify polling mode, which we didn't really test > >> well for quite a while... > >> > >> Rainer > >> > > >> > Thanks, > >> > Ciprian > >> > > >> > -- > >> > Performance Monitoring * Log Analytics * Search Analytics > >> > Solr & Elasticsearch Support * http://sematext.com/ > >> > > >> > On Wed, Dec 16, 2015 at 3:40 PM, Rainer Gerhards < > >> [email protected]> > >> > wrote: > >> > > >> >> 2015-12-16 14:16 GMT+01:00 Ciprian Hacman < > [email protected] > >> >: > >> >> > Done! I updated the output in the same place: > >> >> > https://gist.github.com/hakman/44afddaf4eb67cda28c6 > >> >> > >> >> mhhh... less clear than I had hoped... > >> >> > >> >> Can you switch imfile to use inotify mode and retry? > >> >> > >> >> Rainer > >> >> > > >> >> > Thanks for looking into this so fast! > >> >> > > >> >> > Ciprian > >> >> > > >> >> > -- > >> >> > Performance Monitoring * Log Analytics * Search Analytics > >> >> > Solr & Elasticsearch Support * http://sematext.com/ > >> >> > > >> >> > On Wed, Dec 16, 2015 at 2:53 PM, Rainer Gerhards < > >> >> [email protected]> > >> >> > wrote: > >> >> > > >> >> >> 2015-12-16 13:48 GMT+01:00 Ciprian Hacman < > >> [email protected] > >> >> >: > >> >> >> > Hi, > >> >> >> > > >> >> >> > I upgraded a server to Rsyslog 8.15 last night and today the > >> process > >> >> was > >> >> >> > using almost 200MB of RAM (raising steadily). > >> >> >> > Tried running the process in Valgrind to see if I get an idea > about > >> >> what > >> >> >> > happens, but wasn't that much help for me. > >> >> >> > >> >> >> That's because debug symbols are unloaded at module unload time. > This > >> >> >> makes valgrind stacktraces unusable. Nevertheless, the information > >> >> >> looks very promising. > >> >> >> > >> >> >> Can you build rsyslog yourself for that box? All we need is the > >> >> >> --enable-valgrind option, which will essentially remove the module > >> >> >> unloads and make the stacktrace usable. > >> >> >> > >> >> >> Rainer > >> >> >> > > >> >> >> > If someone has better debugging skills, I pasted the output > here. > >> Not > >> >> >> sure > >> >> >> > if I let it run enough or leave it running longer. > >> >> >> > https://gist.github.com/hakman/44afddaf4eb67cda28c6 > >> >> >> > > >> >> >> > Thanks, > >> >> >> > Ciprian > >> >> >> > > >> >> >> > > >> >> >> > -- > >> >> >> > Performance Monitoring * Log Analytics * Search Analytics > >> >> >> > Solr & Elasticsearch Support * http://sematext.com/ > >> >> >> > > >> >> >> > On Tue, Dec 15, 2015 at 2:41 PM, David Lang <[email protected]> > wrote: > >> >> >> > > >> >> >> >> On Tue, 15 Dec 2015, Ciprian Hacman wrote: > >> >> >> >> > >> >> >> >> Hi David, > >> >> >> >>> > >> >> >> >>> maxMessageSize="10000" > >> >> >> >>> queue.size="10000" - main queue > >> >> >> >>> queue.size="10000" - elasticsearch queue > >> >> >> >>> > >> >> >> >>> Based on my calculations this brings me to a max of 200MB of > >> memory, > >> >> >> maybe > >> >> >> >>> a little more depending on how maxMessageSize is calculated. > >> >> >> >>> > >> >> >> >>> I read logs from a file and push them to Elasticsearch (on the > >> same > >> >> >> >>> network), so TCP is the only possibility. This server has a > very > >> >> simple > >> >> >> >>> setup. > >> >> >> >>> > >> >> >> >>> If I don't find the reason for this issue, I might have to go > >> >> implement > >> >> >> >>> the > >> >> >> >>> forwarding to a central location and push to Elasticsearch > from > >> >> there. > >> >> >> >>> > >> >> >> >> > >> >> >> >> There are advantages to sending things to a central server. > >> >> >> >> > >> >> >> >> it's one place to queue data, so you can either allocate more > ram, > >> >> or go > >> >> >> >> to disk as needed without impacting other workloads. > >> >> >> >> > >> >> >> >> it's more efficient, the central server is more likely to have > >> larger > >> >> >> >> batches of data to feed to ES, and ES only needs to be running > one > >> >> >> thread > >> >> >> >> processing inbound data > >> >> >> >> > >> >> >> >> while it is one more point to have to look at, I think it > >> simplifies > >> >> >> >> troubleshooting as all the communication to ES (and therefor > all > >> the > >> >> >> errors > >> >> >> >> for such communication) happen in one place instead of > >> distributed. > >> >> >> >> > >> >> >> >> anyway, let's see how things look with 8.15 > >> >> >> >> > >> >> >> >> > >> >> >> >> David Lang > >> >> >> >> _______________________________________________ > >> >> >> >> rsyslog mailing list > >> >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >> >> http://www.rsyslog.com/professional-services/ > >> >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by a > >> >> myriad > >> >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST if > >> >> you > >> >> >> >> DON'T LIKE THAT. > >> >> >> >> > >> >> >> > _______________________________________________ > >> >> >> > rsyslog mailing list > >> >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >> > http://www.rsyslog.com/professional-services/ > >> >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >> >> myriad > >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >> you > >> >> >> DON'T LIKE THAT. > >> >> >> _______________________________________________ > >> >> >> rsyslog mailing list > >> >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> >> http://www.rsyslog.com/professional-services/ > >> >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >> >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >> you > >> >> >> DON'T LIKE THAT. > >> >> >> > >> >> > _______________________________________________ > >> >> > rsyslog mailing list > >> >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> > http://www.rsyslog.com/professional-services/ > >> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad > >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> >> DON'T LIKE THAT. > >> >> _______________________________________________ > >> >> rsyslog mailing list > >> >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> >> http://www.rsyslog.com/professional-services/ > >> >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> >> DON'T LIKE THAT. > >> >> > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
