On Tue, Dec 15, 2015 at 12:30 AM, Ciprian Hacman < [email protected]> wrote:
> Hi David, > > maxMessageSize="10000" > queue.size="10000" - main queue > queue.size="10000" - elasticsearch queue > > Based on my calculations this brings me to a max of 200MB of memory, maybe > a little more depending on how maxMessageSize is calculated. > > I read logs from a file and push them to Elasticsearch (on the same > network), so TCP is the only possibility. This server has a very simple > setup. > > If I don't find the reason for this issue, I might have to go implement the > forwarding to a central location and push to Elasticsearch from there. > > Thanks, > Ciprian > > -- > Performance Monitoring * Log Analytics * Search Analytics > Solr & Elasticsearch Support * http://sematext.com/ > > On Tue, Dec 15, 2015 at 12:52 AM, David Lang <[email protected]> wrote: > > > what is your maxmessagesize and your max queue size? rsyslog will use up > > to maxmessagesize*maxqueuesize ram to buffer messages if they can't be > > delivered. > > > > you probably want to set these values smaller rather than setting > > something up to kill rsyslog when it gets large. > > > > What is the transport you use to deliver the logs from these systems? > > > > I like to setup log redundant log relay servers in each datacenter and > > then have all the systems log to these relays via UDP. UDP is reliable > over > > a local network, but if there is a problem with the receiving system, it > > will go ahead and loose logs rather than affecting the sending system. > > > > David Lang > > > > > > > > On Mon, 14 Dec 2015, Ciprian Hacman wrote: > > > > Hi David, > >> > >> Yes, killing Rsyslog is a last resort, but for most people I think > >> shipping > >> logs is a secondary function on a server. Would prefer that Rsyslog > >> doesn't > >> interfere with other apps. > >> > >> I usually enable impstats, though on these particular server the queues > >> are > >> really tiny so that it doesn't use that much memory. I would expect some > >> memory usage fluctuations when Elasticsearch doesn't respond, but > nothing > >> as extreme as using 6GB of memory. > >> > >> If changes in 8.15 don't help, I think I have to spend a few hours > trying > >> to reproduce this. > >> > >> Thanks, > >> Ciprian > >> > >> -- > >> Performance Monitoring * Log Analytics * Search Analytics > >> Solr & Elasticsearch Support * http://sematext.com/ > >> > >> On Mon, Dec 14, 2015 at 8:17 PM, David Lang <[email protected]> wrote: > >> > >> On Mon, 14 Dec 2015, Ciprian Hacman wrote: > >>> > >>> Hi, > >>> > >>>> > >>>> We are noticing some Rsyslog instances that push about 500MB of logs > >>>> daily > >>>> to an Elasticsearch instance, so not that much. We noticed one of the > >>>> Rsyslog processes using about 6GB of RAM. Usually this is less than > 1MB. > >>>> > >>>> I plan on debugging this in the next few days, but wanted to see also > if > >>>> there is a good idea to add a RSS limit (doable in Upstart and > Systemd) > >>>> and > >>>> kill / restart Rsyslog when it gets into such a situation. > >>>> > >>>> > >>> killing/restarting rsyslog is a last resort. large memory usage usually > >>> means that you have lots of logs that aren't delivered and are sitting > >>> in a > >>> queue somewhere. > >>> > >>> do you have impstats configured? > Ciprian, are you going to enable impstats? I'd be curious to know what I shows. Thanks, -peter > if not, it's a _really_ good idea to > >>> configure it and have it write either directly to a file (log rotation > of > >>> this file is a bit of an issue) or to it's own ruleset. either way > means > >>> that a blockage in normal log processing will not affect the pstats > logs. > >>> These logs will show you if you have queues building up and where. > >>> > >>> David Lang > >>> > >>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
