Hi Peter, Just checked and impstats were actually enabled. Here is the sample before restart:
Dec 14 09:25:09 host1 rsyslogd-pstats: omelasticsearch: origin=omelasticsearch submitted=198115631 failed.http=260 failed.httprequests=2 failed.es=58720 Dec 14 09:25:09 host1 rsyslogd-pstats: test-norm-direct: origin=core.action processed=197076777 failed=0 suspended=0 suspended.duration=0 resumed=0 Dec 14 09:25:09 host1 rsyslogd-pstats: test-es-direct: origin=core.action processed=197076777 failed=0 suspended=0 suspended.duration=0 resumed=0 Dec 14 09:25:09 host1 rsyslogd-pstats: test-norm: origin=core.action processed=1040122 failed=0 suspended=0 suspended.duration=0 resumed=0 Dec 14 09:25:09 host1 rsyslogd-pstats: test-es: origin=core.action processed=1039926 failed=0 suspended=0 suspended.duration=0 resumed=0 Dec 14 09:25:09 host1 rsyslogd-pstats: resource-usage: origin=impstats utime=5859695897 stime=206002475 maxrss=6587196 minflt=15802757 majflt=19279 inblock=3838232 oublock=2238104 nvcsw=5667006 nivcsw=596991 Dec 14 09:25:09 host1 rsyslogd-pstats: test-es-direct queue: origin=core.queue size=75 enqueued=197076777 full=1742 discarded.full=1423 discarded.nf=0 maxqsize=10000 Dec 14 09:25:09 host1 rsyslogd-pstats: test-es queue: origin=core.queue size=0 enqueued=1039926 full=0 discarded.full=0 discarded.nf=0 maxqsize=2888 Dec 14 09:25:09 host1 rsyslogd-pstats: main Q: origin=core.queue size=27 enqueued=198705203 full=0 discarded.full=0 discarded.nf=0 maxqsize=9765 Ciprian -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Tue, Dec 15, 2015 at 7:57 AM, Peter Portante <[email protected]> wrote: > On Tue, Dec 15, 2015 at 12:30 AM, Ciprian Hacman < > [email protected]> wrote: > > > Hi David, > > > > maxMessageSize="10000" > > queue.size="10000" - main queue > > queue.size="10000" - elasticsearch queue > > > > Based on my calculations this brings me to a max of 200MB of memory, > maybe > > a little more depending on how maxMessageSize is calculated. > > > > I read logs from a file and push them to Elasticsearch (on the same > > network), so TCP is the only possibility. This server has a very simple > > setup. > > > > If I don't find the reason for this issue, I might have to go implement > the > > forwarding to a central location and push to Elasticsearch from there. > > > > Thanks, > > Ciprian > > > > -- > > Performance Monitoring * Log Analytics * Search Analytics > > Solr & Elasticsearch Support * http://sematext.com/ > > > > On Tue, Dec 15, 2015 at 12:52 AM, David Lang <[email protected]> wrote: > > > > > what is your maxmessagesize and your max queue size? rsyslog will use > up > > > to maxmessagesize*maxqueuesize ram to buffer messages if they can't be > > > delivered. > > > > > > you probably want to set these values smaller rather than setting > > > something up to kill rsyslog when it gets large. > > > > > > What is the transport you use to deliver the logs from these systems? > > > > > > I like to setup log redundant log relay servers in each datacenter and > > > then have all the systems log to these relays via UDP. UDP is reliable > > over > > > a local network, but if there is a problem with the receiving system, > it > > > will go ahead and loose logs rather than affecting the sending system. > > > > > > David Lang > > > > > > > > > > > > On Mon, 14 Dec 2015, Ciprian Hacman wrote: > > > > > > Hi David, > > >> > > >> Yes, killing Rsyslog is a last resort, but for most people I think > > >> shipping > > >> logs is a secondary function on a server. Would prefer that Rsyslog > > >> doesn't > > >> interfere with other apps. > > >> > > >> I usually enable impstats, though on these particular server the > queues > > >> are > > >> really tiny so that it doesn't use that much memory. I would expect > some > > >> memory usage fluctuations when Elasticsearch doesn't respond, but > > nothing > > >> as extreme as using 6GB of memory. > > >> > > >> If changes in 8.15 don't help, I think I have to spend a few hours > > trying > > >> to reproduce this. > > >> > > >> Thanks, > > >> Ciprian > > >> > > >> -- > > >> Performance Monitoring * Log Analytics * Search Analytics > > >> Solr & Elasticsearch Support * http://sematext.com/ > > >> > > >> On Mon, Dec 14, 2015 at 8:17 PM, David Lang <[email protected]> wrote: > > >> > > >> On Mon, 14 Dec 2015, Ciprian Hacman wrote: > > >>> > > >>> Hi, > > >>> > > >>>> > > >>>> We are noticing some Rsyslog instances that push about 500MB of logs > > >>>> daily > > >>>> to an Elasticsearch instance, so not that much. We noticed one of > the > > >>>> Rsyslog processes using about 6GB of RAM. Usually this is less than > > 1MB. > > >>>> > > >>>> I plan on debugging this in the next few days, but wanted to see > also > > if > > >>>> there is a good idea to add a RSS limit (doable in Upstart and > > Systemd) > > >>>> and > > >>>> kill / restart Rsyslog when it gets into such a situation. > > >>>> > > >>>> > > >>> killing/restarting rsyslog is a last resort. large memory usage > usually > > >>> means that you have lots of logs that aren't delivered and are > sitting > > >>> in a > > >>> queue somewhere. > > >>> > > >>> do you have impstats configured? > > > > Ciprian, are you going to enable impstats? I'd be curious to know what I > shows. > > Thanks, -peter > > > > > if not, it's a _really_ good idea to > > >>> configure it and have it write either directly to a file (log > rotation > > of > > >>> this file is a bit of an issue) or to it's own ruleset. either way > > means > > >>> that a blockage in normal log processing will not affect the pstats > > logs. > > >>> These logs will show you if you have queues building up and where. > > >>> > > >>> David Lang > > >>> > > >>> _______________________________________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com/professional-services/ > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad > > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > > >>> DON'T LIKE THAT. > > >>> > > >>> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com/professional-services/ > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > >> DON'T LIKE THAT. > > >> > > >> _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > > DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
