since this is being transmitted via UDP, I believe that this could be a DNS slowdown, could you try running ryslog with DNS disabled? (-x option at startup)
what version of rsyslog is this? It would probably be useful to look at top during this timerun top, hit 'H' to see the threads, and then when you send the logs, look for any threads that max out the cpu.
It would help if you would name your actions (add name='name' to each action), this will make both the pstats and top output clearer.
Please more impstats up to the top of the configPlease disable all output except the squid.log, the fact that you have an action that's sending logs via omrelp could make it so that the bottleneck is in the server you are sending it to there, so let's make sure we don't have a problem locally before we have to chase that link.
David Lang On Wed, 18 Jun 2014, Muhammad Asif wrote:
Thanx David and Rainer for your useful help. To avoid load on one NIC, i sent 100000 (2.3 sec) from a desktop machine using tcpflood and 100000 (0.5 sec, why it is so fast?) messages from a server on rsyslog running on same server. Rsyslog version is 8.2.1. I am facing the same issue. Rsyslog receive only 100000 message in one interval of 10 sec. I have attached the pstat file. please help me in this regard. ./tcpflood -t 172.20.16.8 -p 514 -m 100000 -M '2014-05-15T09:21:40.663676+05:00 172.20.8.12 (squid): 1400122878.349 2226 172.20.13.11 TCP_MISS/200 2261 GET http://images04.olx-st.com/ui/8/38/22/s_1400092897_645663622_5-samsung-s3-original-for-sale.jpg - DIRECT/89.149.175.34 image/jpeg' -T udp Highlighted part is message. Rsyslog Configuration --------------------------------------------------------------------------------------------------------------------------- # rsyslog configuration file # note that most of this config file uses old-style format, # because it is well-known AND quite suitable for simple cases # like we have with the default config. For more advanced # things, RainerScript configuration is suggested. # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### module(load="imuxsock") # provides support for local system logging (e.g. via logger command) module(load="imklog") # provides kernel logging support (previously done by rklogd) #module(load"immark") # provides --MARK-- message capability # Provides UDP syslog reception # for parameters see http://www.rsyslog.com/doc/imudp.html module(load="imudp") # needs to be done just once input(type="imudp" port="514") #if ($fromhost-ip == '172.20.8.12' AND $rawmsg contains "squid") then /opt/squid.log #& ~ #if ($fromhost-ip == '172.20.8.3') then /var/log/ciit-dc.log #& ~ #$SystemLogRateLimitInterval 10 #$SystemLogRateLimitBurst 100000 # Provides TCP syslog reception # for parameters see http://www.rsyslog.com/doc/imtcp.html module(load="imtcp") # needs to be done just once input(type="imtcp" port="514") $template msgonly,"%rawmsg%\n" module(load="omrelp") action(type="omrelp" target="127.0.0.1" port="520") module(load="impstats" interval="10" severity="7" resetCounters="on" log.syslog="off" log.file="/var/log/stats.log") module(load="omprog") *.*action(binary="/opt/test.sh") #### GLOBAL DIRECTIVES #### if $rawmsg contains "squid" then # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /var/lib/rsyslog # where to place spool files #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ### end of the forwarding rule ### On Tue, Jun 17, 2014 at 6:09 PM, David Lang <[email protected]> wrote:On Tue, 17 Jun 2014, Muhammad Asif wrote: Hi Geeks,I am using tcpflood for sending burst on rsyslog. when i send 100000 messages in 2.3 second everything is fine but when i increase number of messages like 110000, rsyslog receive only about 101000 and drop all rest of the messages. The impstat interval is 10 sec. My rsyslog and tcpflood is on same machine. But behaviour is same if i use tcpflood on other machine. How can i tune tcpflood for 500000 messages in 5 sec and rsyslog receive the same amount of messages. Please guide me regarding this issue.what are you doing with the messages? what is your configuration? what version of rsyslog are you running? we have people who have tuned rsyslog to handle several hundred thousand messages per second (and reports of someone hadling 1 million messages/sec), but it depends on what version you are running, what you are doing with the messages, what transport you are using, etc. For example, if you are using UDP transport and have a poor DNS server, you could be bottlenecked on the DNS lookups, if you are writing out to dynamic file names, you could be bottlenecked there if you don't have a large enough dynafilecache, if you are sending the data to a database, you could be bottlenecked there. David Lang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
stat_bursts.log
Description: Binary data
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
