Just did some analysis of rsyslog stats counters and found the following.
The maxrss counter is increasing accordingly to size of queue.
Seems there is much higher overhead than expected.
I tried the message sizes of 1840/940/640/340 characters.
These are the outcomes:
size avg_mem_footprint_per_message
340 ~1024
640 ~1330
940 ~1630
1840 ~2530
size VmPeak_150000_messages_in_queue
340 859108 kB
640 715740 kB
940 859128 kB (stopped at ~145000 messages in queue)
1840 1053580 kB
>From these numbers it is obvious the static overhead of memory
footprint for one message logged (doesn't matter of the message
length) to file is ~690 Bytes.
Numbers were counted from rsyslog stats module only. With one TCP
forward queue (target not available).
Messages generated by:
~# while true; do echo -n .; logger -t perftest -i "xx(how log the
message is needed)xx"; done
Then stat counters processed by (may be improved):
~# awk '{gsub(/size=|maxrss=/,"")} /resource-usage/{m=$9} /action 1
queue:/{q=$9} {print m"\t"q}' /var/log/rsyslog-stats.300
--
Peter
On Thu, Apr 6, 2017 at 11:02 AM, David Lang <[email protected]> wrote:
> On Thu, 6 Apr 2017, Peter Viskup wrote:
>
>> Hello David,
>> thank you for pointing to the RSYSLOG_DebugFormat.
>> One test message showed me this:
>>
>> Debug line with all properties:
>> FROMHOST: 'server', fromhost-ip: '127.0.0.1', HOSTNAME: 'server', PRI: 46,
>> syslogtag 'rsyslogd:', programname: 'rsyslogd', APP-NAME: 'rsyslogd',
>> PROCID: '-', MSGID: '-',
>> TIMESTAMP: 'Apr 6 10:45:05', STRUCTURED-DATA: '-',
>> msg: ' [origin software="rsyslogd" swVersion="8.15.0" x-pid="2957"
>> x-info="http://www.rsyslog.com";] start'
>> escaped msg: ' [origin software="rsyslogd" swVersion="8.15.0"
>> x-pid="2957" x-info="http://www.rsyslog.com";] start'
>> inputname: rsyslogd rawmsg: ' [origin software="rsyslogd"
>> swVersion="8.15.0" x-pid="2957" x-info="http://www.rsyslog.com";]
>> start'
>> $!:
>> $.:
>> $/:
>>
>> Does this mean the message is in memory in three copies - as 'msg',
>> 'escaped msg' and 'rawmsg'?
>> Does the whole debug line provide the memory footprint or do I have to
>> count the message size overhead with value-name strings only
>> "FROMHOST: , fromhost-ip: , HOSTNAME: , PRI: , syslogtag, programname:
>> , APP-NAME: , PROCID: , MSGID: , TIMESTAMP: , STRUCTURED-DATA: ...."?
>> Or in any other way?
>>
>> In case the whole debug line is equal to the message size in memory,
>> the message of 162B consume 602B in memory. Is that right?
>
>
> not all the variable names need to be counted, but I'm not sure which ones
> can be ignored (it's possible that all of them could, but Rainer would need
> to comment on this)
>
> but with that caviot, I believe you are correct in your interpretation.
>
> David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
