The in-memory queue isn't dropped, only the counter is reset to 0 after a while. Just opened issue in rsyslog regarding the queue stats - seems to be there is a bug:
https://github.com/rsyslog/rsyslog/issues/1585 Thus not able to do proper sizing based on the counters from impstats outputs at the moment. Peter On Mon, Apr 10, 2017 at 10:06 AM, Peter Viskup <[email protected]> wrote: > On Fri, Apr 7, 2017 at 6:32 PM, David Lang <[email protected]> wrote: > > On Fri, 7 Apr 2017, Peter Viskup wrote: > > > >> Just did some analysis of rsyslog stats counters and found the > following. > >> The maxrss counter is increasing accordingly to size of queue. > >> > >> Seems there is much higher overhead than expected. > >> I tried the message sizes of 1840/940/640/340 characters. > >> > >> These are the outcomes: > >> size avg_mem_footprint_per_message > >> 340 ~1024 > >> 640 ~1330 > >> 940 ~1630 > >> 1840 ~2530 > >> > >> size VmPeak_150000_messages_in_queue > >> 340 859108 kB > >> 640 715740 kB > >> 940 859128 kB (stopped at ~145000 messages in queue) > >> 1840 1053580 kB > >> > >> From these numbers it is obvious the static overhead of memory > >> footprint for one message logged (doesn't matter of the message > >> length) to file is ~690 Bytes. > > > > > > I suspect that it's not quite this simple, but close. I would guess that > the > > length of the hostname and syslogtag affect this, and if the timestamp is > > the short traditional timestamp or the longer new format that includes > year > > and timezone will affect this a bit. > > > > but those variations probably don't have that big an effect (tens of > bytes, > > not hundreds) > > > > That's more overhead than I expected. > > > > would you mind posting this to the mailing list? I thing others would > find > > it useful. > > The sizes of messages are counted using the RSYSLOG_TraditionalFileFormat. > > Unfortunately I didn't find the way how to monitor the message memory > footprint with better quality. > This is because after a while the in-memory queue is dropped (even > when the thresholds were not reached). > > Attached are stats outputs from the time the queue was dropped. > > Because of this I had to increase the message rate per second to let > the queue fill (occupate more memory) to be able to get reasonable > numbers. > Is there any way how to stop the queue being dropped? > > This is the queue configuration: > $ActionQueueFileName forward_queue > $ActionQueueType LinkedList > $ActionQueueSaveOnShutdown on > $ActionQueueSize 150000 > $ActionQueueMaxDiskSpace 50m > $ActionQueueDiscardMark 145000 > $ActionQueueHighWaterMark 130000 > $ActionQueueLowWaterMark 100000 > > :syslogtag, contains, "perftest" @@127.0.0.1:10000 > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
