Thanx David and Rainer for your useful help. To avoid load on one NIC, i sent 100000 (2.3 sec) from a desktop machine using tcpflood and 100000 (0.5 sec, why it is so fast?) messages from a server on rsyslog running on same server. Rsyslog version is 8.2.1. I am facing the same issue. Rsyslog receive only 100000 message in one interval of 10 sec. I have attached the pstat file. please help me in this regard.
./tcpflood -t 172.20.16.8 -p 514 -m 100000 -M '2014-05-15T09:21:40.663676+05:00 172.20.8.12 (squid): 1400122878.349 2226 172.20.13.11 TCP_MISS/200 2261 GET http://images04.olx-st.com/ui/8/38/22/s_1400092897_645663622_5-samsung-s3-original-for-sale.jpg - DIRECT/89.149.175.34 image/jpeg' -T udp Highlighted part is message. Rsyslog Configuration --------------------------------------------------------------------------------------------------------------------------- # rsyslog configuration file # note that most of this config file uses old-style format, # because it is well-known AND quite suitable for simple cases # like we have with the default config. For more advanced # things, RainerScript configuration is suggested. # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### module(load="imuxsock") # provides support for local system logging (e.g. via logger command) module(load="imklog") # provides kernel logging support (previously done by rklogd) #module(load"immark") # provides --MARK-- message capability # Provides UDP syslog reception # for parameters see http://www.rsyslog.com/doc/imudp.html module(load="imudp") # needs to be done just once input(type="imudp" port="514") #if ($fromhost-ip == '172.20.8.12' AND $rawmsg contains "squid") then /opt/squid.log #& ~ #if ($fromhost-ip == '172.20.8.3') then /var/log/ciit-dc.log #& ~ #$SystemLogRateLimitInterval 10 #$SystemLogRateLimitBurst 100000 # Provides TCP syslog reception # for parameters see http://www.rsyslog.com/doc/imtcp.html module(load="imtcp") # needs to be done just once input(type="imtcp" port="514") $template msgonly,"%rawmsg%\n" module(load="omrelp") action(type="omrelp" target="127.0.0.1" port="520") module(load="impstats" interval="10" severity="7" resetCounters="on" log.syslog="off" log.file="/var/log/stats.log") module(load="omprog") *.*action(binary="/opt/test.sh") #### GLOBAL DIRECTIVES #### if $rawmsg contains "squid" then # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /var/lib/rsyslog # where to place spool files #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ### end of the forwarding rule ### On Tue, Jun 17, 2014 at 6:09 PM, David Lang <[email protected]> wrote: > On Tue, 17 Jun 2014, Muhammad Asif wrote: > > Hi Geeks, >> >> I am using tcpflood for sending burst on rsyslog. >> when i send 100000 messages in 2.3 second everything is fine but when i >> increase number of messages like 110000, rsyslog receive only about 101000 >> and drop all rest of the messages. The impstat interval is 10 sec. My >> rsyslog and tcpflood is on same machine. But behaviour is same if i use >> tcpflood on other machine. How can i tune tcpflood for 500000 messages in >> 5 >> sec and rsyslog receive the same amount of messages. >> >> Please guide me regarding this issue. >> > > what are you doing with the messages? > what is your configuration? > what version of rsyslog are you running? > > we have people who have tuned rsyslog to handle several hundred thousand > messages per second (and reports of someone hadling 1 million > messages/sec), but it depends on what version you are running, what you are > doing with the messages, what transport you are using, etc. > > For example, if you are using UDP transport and have a poor DNS server, > you could be bottlenecked on the DNS lookups, if you are writing out to > dynamic file names, you could be bottlenecked there if you don't have a > large enough dynafilecache, if you are sending the data to a database, you > could be bottlenecked there. > > David Lang > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. >
stat_bursts.log
Description: Binary data
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
