what rainer says
2G of memory isn't _that_ much nowdays. If you have a 1K log message that then
gets parsed into a bunch of variables, it's going to take substantially more ram
(2-3K would not be unreasonable, more is possible)
As the messages are then sent out, they remain in the queue while a copy is made
and formatted for delivery. That copy can then get compressed/encrypted which
makes an additional copy.
If each message takes 4K of ram, and you have 100K messages in a queue, you now
have used 400M of ram, a pretty significant fraction of your 2G total (and this
doesn't count for code space.
If your queues back up, you can end up eating a LOT of ram, even if you don't
set omfile buffers large
each outbound connection you have will have it's own buffer, each inbound
connection will have it's own buffer (amoung other things, this is part of why
we were trying to discourage you from having separate ports for each log type
:-)
but the impstats output will show you how large the queues are and what the
overall resource usage of rsyslog is.
you may also want to look at the output of top (with the 'H' option to see
threads) and see if you have an explosion of threads happening. I tripped over
this with dynafiles and async writing when I was trying to improve compressed
file output performance.
David Lang
On Thu, 12 Jan 2017, Rainer Gerhards wrote:
Have a look what impstats tells you. I guess some queue is defined very
large and fills up.
Rainer
2017-01-12 12:01 GMT+01:00 mostolog--- via rsyslog <
[email protected]>:
Hi
At this moment, our testing rsyslog is receiving messages from relay
through RELP in RFC3164 format with message in JSON like:
<PRI>DATE RELAYHOST TAG: {"field":"foo",..."data":"orig
inal_forwarded_message"}
Once received, mmjsonparse is executed over MSG to check if incoming
messages "comply", before mmnormalize on data (actually doing nothing due
to liblognorm bugs already reported) and setting some other fields.
After that, omelasticsearch action is executed and messages are indexed.
We are receiving ~1 million events/hour and rsyslog has 2GB of memory and
process is killed cause out of memory (every other hour).
What steps could we take to diagnose if that's correct, how to avoid
it...? Any up to date literature?
Thanks. Regards
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
