If I write out to /dev/null the only way for me to see what is being processed will be thorugh the pstats? ----- Original Message ----- From: David Lang Sent: 10/09/13 02:49 PM To: Robert Subject: Re: [rsyslog] Tr : Re: perfomance tweaking (fwd)
thanks for including the config, it's hard to keep track of everything. the 'unexpected end of file' error is because rsyslog still has the file open at the time you are doing the zgrep, if you rotate the files (mv the file then hit rsyslog with kill -HUP) rsyslog will write the remainder of the file and you will not get that error. The fact that all the per second numbers are eactly 200000 or 300000 is very odd what did top look like during the test? and just to confirm, this is rsyslog 7.5 right? if you remove ziplevel, it will disable gzip entirely. I expect that when you do that you will find iostat going to 100%, but you can give it a try. could you do a zgrep for a time frame when the queue was full (this will give us an idea of the throughput), say around 14:12 or so David Lang On Wed, 9 Oct 2013, Robert wrote: > Date: Wed, 09 Oct 2013 14:40:13 -0400 > From: Robert <[email protected]> > To: David Lang <[email protected]> > Cc: Rainer Gerhards <[email protected]>, rmkml <[email protected]> > Subject: Re: [rsyslog] Tr : Re: perfomance tweaking (fwd) > > Hello Everyone, > > I have been away for business, but I'm back in action, attached you will find the stats for the rsyslog server, [we are currently throwing 395k/eps to the server ~ 985Mb] and below you will find the current config (just to refreshen the memory) > > > #### MODULES #### > > module(load="imuxsock") # needs to be done just once Roberto 8-9-13 > module(load="imklog") > module(load="impstats" interval="30" log.syslog="off" > log.file="/var/log/pstats") > > # Provides UDP syslog reception > module(load="imudp" > batchSize="1000" > TimeRequery="100" > SchedulingPolicy="fifo" > SchedulingPriority="10") > input(type="imudp" port="514" ) > > #### GLOBAL DIRECTIVES #### > > # Use imprecise-old-style timestamp format > $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > $MainMsgQueueType FixedArray > $MainMsgQueueSize 20000000 > $MainMsgQueueWorkerThreads 2 > $MainMsgQueueDequeu eBatchSize 1000 > $InputUDPMaxSessions 40000000 > > #### RULES #### > > > > action (type="omfile" > File="/var/log/test/f_all" > ziplevel="1" > ASyncWriting="on") > > > Lastly these are the result from the zgrep command: > > > > [rcortiz@simon test]$ sudo zgrep 14:07:53 f_all | wc -l > > gzip: f_all: unexpected end of file > 200001 > [rcortiz@simon test]$ sudo zgrep 14:07:54 f_all | wc -l > > gzip: f_all: unexpected end of file > 200000 > [rcortiz@simon test]$ sudo zgrep 14:07:55 f_all | wc -l > > gzip: f_all: unexpected end of file > 200000 > [rcortiz@simon test]$ sudo zgrep 14:08:00 f_all | wc -l > > gzip: f_all: unexpected end of file > 300000 > [rcortiz@simon test]$ sudo zgrep 14:08:10 f_all | wc -l > > gzip: f_all: unexpected end of file > 200000 > [rcortiz@simon test]$ sudo zgrep 14:08:20 f_all | wc -l > > gzip: f_all: unexpected end of file > 200000 > [rcortiz@simon test]$ sudo zgrep 14:08:30 f_all | wc -l > > gzip: f_all: unexpected end of file > 200000 > [rcortiz@si mon test]$ sudo zgrep 14:08:40 f_all | wc -l > > gzip: f_all: unexpected end of file > 200001 > [rcortiz@simon test]$ sudo zgrep 14:08:50 f_all | wc -l > > gzip: f_all: unexpected end of file > 200000 > [rcortiz@simon test]$ > > > I am curious if we need to remove the zipping part of the action also maybe adding the recvbuffer size? > > Robert. > Robert. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
