On Fri, Sep 6, 2013 at 1:04 AM, Robert McIntyre <[email protected]>wrote:
> Sorry for all the spam, folks. Final update before I wait for some help, > hopefully. If I'm forwarding via UDP, and I monitor tcpdump long enough, > eventually I'll get a short burst of UDP traffic sent. > > It looks like it's about every 30 seconds or so, and only 30-50 messages. > This is on a system that is receiving 40k+ messages per second. My > dequeue batch size is 500, but it's not doing it in chunks of 500, nor is > there any change if I remove the queuing config completely. > > I'm open to suggestions. :) > > mhhh.. sounds very strange. One thing you can do is enable impstats and gather statistics. maybe this points us to where things go wrong. At least we have processed message counts and know if imudp gets all messages and/or thinks it sends all out. Rainer > Thanks! > Robert > > > > From: [email protected] > > To: [email protected] > > Date: Thu, 5 Sep 2013 15:39:45 -0700 > > Subject: Re: [rsyslog] v7.4.4 and omfwd? > > > > Interestingly, if I switch my v7 action(type=) command to tcp/port > 10514, it forwards fine, even with all the queuing config, etc., which > makes me really stumped, now. :| I've verified that there's no firewall in > the way (disabled via iptables), and the exact same tcpdump commands work > to display the outgoing traffic on my production box running 5.8.10. > > > > Does anyone else have omfwd working in 7.4.4 over UDP? > > > > Thanks! > > Robert > > > > > > > From: [email protected] > > > To: [email protected] > > > Date: Thu, 5 Sep 2013 15:12:45 -0700 > > > Subject: Re: [rsyslog] v7.4.4 and omfwd? > > > > > > No luck. Removed the *.* and all queue config for the action in the > v7 action, and still does the same thing. I've captured a debug log, but > am loathe to send it out since it contains the actual syslog events. Can > you suggest things to look for in it? > > > > > > Thanks! > > > Robert > > > > > > > > > > To: [email protected] > > > > From: [email protected] > > > > Date: Thu, 5 Sep 2013 14:59:30 -0700 > > > > Subject: Re: [rsyslog] v7.4.4 and omfwd? > > > > > > > > Thanks. That particular statement (with the port and *.*) works > fine under 5.8.x, but I'll make this change and try it. > > > > > > > > Thanks! > > > > Robert > > > > ________________________________ > > > > From: David Lang<mailto:[email protected]> > > > > Sent: 9/5/2013 2:50 PM > > > > To: rsyslog-users<mailto:[email protected]> > > > > Subject: Re: [rsyslog] v7.4.4 and omfwd? > > > > > > > > by the way, with v7 you don't have to put in *.* you can just put > the action > > > > > > > > @x.x.x.x > > > > > > > > action(whatever) > > > > > > > > > > > > instead of > > > > > > > > *.* @x.x.x.x > > > > > > > > *.* action(whatever) > > > > > > > > David Lang > > > > > > > > > > > > > > > > On Thu, 5 Sep 2013, Robert McIntyre wrote: > > > > > > > > > Date: Thu, 5 Sep 2013 14:45:47 -0700 > > > > > From: Robert McIntyre <[email protected]> > > > > > Reply-To: rsyslog-users <[email protected]> > > > > > To: rsyslog-users <[email protected]> > > > > > Subject: Re: [rsyslog] v7.4.4 and omfwd? > > > > > > > > > > Thanks. I've stripped out everything but the ruleset queue, and > the new format action, and it's still doing the same thing. It's not > queuing up, but still nothing hitting the wire. I'll pull the queue stuff > as well, just in case. > > > > > > > > > > Thanks! > > > > > Robert > > > > > ________________________________ > > > > > From: David Lang<mailto:[email protected]> > > > > > Sent: ?9/?5/?2013 2:30 PM > > > > > To: rsyslog-users<mailto:[email protected]> > > > > > Subject: Re: [rsyslog] v7.4.4 and omfwd? > > > > > > > > > > for the legacy action you hsould not need to specify the port. Try > that and see > > > > > if it works > > > > > > > > > > Then I would say try the new format , but simplify it, drop all he > queue stuff > > > > > and then work up from there. > > > > > > > > > > David Lang > > > > > > > > > > > > > > > > > > > > On Thu, 5 Sep 2013, Robert McIntyre wrote: > > > > > > > > > >> Date: Thu, 5 Sep 2013 13:15:29 -0700 > > > > >> From: Robert McIntyre <[email protected]> > > > > >> Reply-To: rsyslog-users <[email protected]> > > > > >> To: "[email protected]" <[email protected]> > > > > >> Subject: Re: [rsyslog] v7.4.4 and omfwd? > > > > >> > > > > >> A bit more info. When using the legacy action (*.* > @x.x.x.x:514), it shows as being processed fine, but not traffic on the > wire. When I use the action(type="omfwd"...) with a queue, it shows the > queue as expanding, but nothing going out. > > > > >> > > > > >> The full omfwd action config is: > > > > >> > > > > >> *.* action(name="Action_FwdReceiver1" > > > > >> type="omfwd" > > > > >> target="x.x.x.x" > > > > >> protocol="udp" > > > > >> port="514" > > > > >> action.resumeretrycount="-1" > > > > >> queue.dequeuebatchsize="500" > > > > >> queue.checkpointinterval="20000" > > > > >> queue.type="linkedlist" > > > > >> queue.timeoutenqueue="0" > > > > >> queue.filename="FwdReceiver1ActionQueue" > > > > >> queue.size="8000000" > > > > >> queue.highwatermark="7000000" > > > > >> queue.lowwatermark="500000" > > > > >> queue.maxdiskspace="100G" > > > > >> queue.saveonshutdown="on") > > > > >> > > > > >> Thanks! > > > > >> Robert > > > > >> > > > > >>> From: [email protected] > > > > >>> To: [email protected] > > > > >>> Date: Thu, 5 Sep 2013 12:22:44 -0700 > > > > >>> Subject: [rsyslog] v7.4.4 and omfwd? > > > > >>> > > > > >>> I'm trying to get the last of my v7 migration done, and am > testing the omfwd action, and it doesn't seem to be working at all. > > > > >>> > > > > >>> I've pasted most of my config below. What I see is the ruleset > catches the messages, and hands them off to the queues. The action even > reports that it has processed all the forwards, but TCPDUMP, and monitoring > the outgoing traffic doesn't have any of the outgoing forwards. The other > actions (writes to file shares) seem to be working properly. > > > > >>> > > > > >>> This repros whether I use the old forward action format (shown > below), or the newer action(type="omfwd"...) format. > > > > >>> > > > > >>> Hoping someone can tell me what I'm missing. :) > > > > >>> > > > > >>> Thanks! > > > > >>> Robert > > > > >>> > > > > >>> # rsyslog v7 configuration file > > > > >>> # For more information see > /usr/share/doc/rsyslog-*/rsyslog_conf.html > > > > >>> # If you experience problems, see > http://www.rsyslog.com/doc/troubleshoot.html > > > > >>> # NetSec Syslog Server config file v3.0 : See the end of the > file for revision information > > > > >>> > > > > >>> #### MODULES #### > > > > >>> module(load="impstats" interval="300") # Provides periodic > performance statistics (this must be the first thing in rsyslog.conf) > > > > >>> module(load="imuxsock") # Provides support for local system > logging (e.g. via logger command) > > > > >>> module(load="imklog") # Provides kernel logging support > (previously done by rklogd) > > > > >>> module(load="imudp" timerequery="10000")# Provides UDP syslog > reception > > > > >>> #module(load="imptcp") # Provides TCP syslog reception > > > > >>> #module(load="immark") # Provides --MARK-- message > capability > > > > >>> > > > > >>> #### GLOBAL DIRECTIVES #### > > > > >>> # Use default timestamp format > > > > >>> $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat > > > > >>> # Don't escape control characters > > > > >>> $EscapeControlCharactersOnReceive off > > > > >>> # Include all config files in /etc/rsyslog.d/ > > > > >>> $IncludeConfig /etc/rsyslog.d/*.conf > > > > >>> # Set the working directory for disk buffers > > > > >>> $WorkDirectory /syslogdata/buffer > > > > >>> > > > > >>> #### TEMPLATES #### > > > > >>> # Filenames > > > > >>> template (name="FirstProdFile" type="string" > string="/firstprodshare/test/AP/%msg:F,32:3%-%$YEAR%-%$MONTH%-%$DAY%T%$HOUR%.%$QHOUR%") > > > > >>> template (name="SecondProdFile" type="string" > string="/secondprodshare/test/CS/%msg:F,32:3%-%$YEAR%-%$MONTH%-%$DAY%T%$HOUR%.%$QHOUR%") > > > > >>> template (name="SyslogStats" type="string" > string="/firstprodshare/00/syslogstats/%$YEAR%-%$MONTH%-%$DAY%-%$myhostname%-stats.txt") > > > > >>> > > > > >>> # Messages > > > > >>> template (name="TemplateMessage" type="string" > string="<%PRI%>%syslogtag%%msg%\n") > > > > >>> > > > > >>> #### MAIN MESSAGE QUEUE #### > > > > >>> # Establish the Main Message Queue > > > > >>> $MainMsgQueueFileName MainQueue > > > > >>> $MainMsgQueueSize 8000000 > > > > >>> $MainMsgQueueHighWaterMark 7000000 > > > > >>> $MainMsgQueueLowWaterMark 500000 > > > > >>> $MainMsgQueueMaxFileSize 100G > > > > >>> $MainMsgQueueSaveOnShutdown on > > > > >>> $MainMsgQueueType LinkedList > > > > >>> $MainMsgQueueWorkerThreads 4 > > > > >>> $MainMsgQueueCheckpointInterval 20000 > > > > >>> > > > > >>> #### RULES #### > > > > >>> ### Local logging > > > > >>> ruleset(name="Ruleset_Local"){ > > > > >>> kern.* > action(name="Action_local_kern" type="omfile" file="/var/log/messages") > > > > >>> *.info;mail.none;authpriv.none;cron.none;syslog.none > action(name="Action_local_info" type="omfile" file="/var/log/messages") > > > > >>> authpriv.* > action(name="Action_local_authpriv" type="omfile" file="/var/log/secure") > > > > >>> mail.* > action(name="Action_local_mail" type="omfile" file="/var/log/maillog") > > > > >>> cron.* > action(name="Action_local_cron" type="omfile" file="/var/log/cron") > > > > >>> *.emerg > action(name="Action_local_emerg" type="omusrmsg" users="*") > > > > >>> uucp,news.crit > action(name="Action_local_news" type="omfile" file="/var/log/spooler") > > > > >>> local7.* > action(name="Action_local_local7" type="omfile" file="/var/log/boot.log") > > > > >>> syslog.info action(name="Action_SyslogStats" > type="omfile" DynaFile="SyslogStats") > > > > >>> syslog.info action(name="Action_SyslogStats" > type="omfile" file="/var/log/syslog") > > > > >>> } # End ruleset Local > > > > >>> > > > > >>> # Use ruleset Local as default > > > > >>> $DefaultRuleset Ruleset_Local > > > > >>> ### End local logging > > > > >>> > > > > >>> ### Remote logging > > > > >>> ruleset (name="Ruleset_Remote" > > > > >>> queue.type="linkedlist" > > > > >>> queue.filename="RemoteRuleSetQueue" > > > > >>> queue.size="8000000" > > > > >>> queue.highwatermark="7000000" > > > > >>> queue.lowwatermark="500000" > > > > >>> queue.maxdiskspace="100G" > > > > >>> queue.saveonshutdown="on" > > > > >>> queue.workerthreads="4"){ > > > > >>> > > > > >>> # Action: Write to the first file share > > > > >>> *.* action (name="Action_FirstFileShare" > > > > >>> type="omfile" > > > > >>> DynaFile="FirstProdFile" > > > > >>> template="TemplateMessage" > > > > >>> iobuffersize="262144" > > > > >>> action.resumeretrycount="-1" > > > > >>> queue.dequeuebatchsize="5000" > > > > >>> queue.checkpointinterval="20000" > > > > >>> queue.type="linkedlist" > > > > >>> queue.timeoutenqueue="0" > > > > >>> queue.filename="FirstProdShareActionQueue" > > > > >>> queue.size="8000000" > > > > >>> queue.highwatermark="7000000" > > > > >>> queue.lowwatermark="500000" > > > > >>> queue.maxdiskspace="100G" > > > > >>> queue.saveonshutdown="on" > > > > >>> queue.workerthreads="4" > > > > >>> ) > > > > >>> > > > > >>> *.* @x.x.x.x:514 > > > > >>> > > > > >>> # Action: Write to the second file share > > > > >>> *.* action (name="Action_FileShare2" > > > > >>> type="omfile" > > > > >>> DynaFile="SecondProdFile" > > > > >>> template="TemplateMessage" > > > > >>> iobuffersize="262144" > > > > >>> action.resumeretrycount="-1" > > > > >>> queue.dequeuebatchsize="5000" > > > > >>> queue.checkpointinterval="20000" > > > > >>> queue.type="linkedlist" > > > > >>> queue.timeoutenqueue="0" > > > > >>> queue.filename="SecondProdShareActionQueue" > > > > >>> queue.size="8000000" > > > > >>> queue.highwatermark="7000000" > > > > >>> queue.lowwatermark="500000" > > > > >>> queue.maxdiskspace="100G" > > > > >>> queue.saveonshutdown="on" > > > > >>> queue.workerthreads="4") > > > > >>> > > > > >>> } # End ruleset Remote > > > > >>> > > > > >>> ## Listeners ## > > > > >>> # Bind ruleset to UDP listener > > > > >>> input(inputname="RemoteUDP_514" type="imudp" port="514" > ruleset="Ruleset_Remote") > > > > >>> > > > > >>> > > > > >>> > > > > >>> _______________________________________________ > > > > >>> rsyslog mailing list > > > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > >>> http://www.rsyslog.com/professional-services/ > > > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > > > >> > > > > >> _______________________________________________ > > > > >> rsyslog mailing list > > > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > >> http://www.rsyslog.com/professional-services/ > > > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > > > >> > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com/professional-services/ > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > > > > _______________________________________________ > > > > > rsyslog mailing list > > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > > http://www.rsyslog.com/professional-services/ > > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > > > _______________________________________________ > > > > rsyslog mailing list > > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > > http://www.rsyslog.com/professional-services/ > > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > > > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
