what was it that was causing the problem?

David Lang

On Fri, 6 Sep 2013, Robert McIntyre wrote:

Date: Fri, 6 Sep 2013 16:25:36 -0700
From: Robert McIntyre <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] v7.4.4 and omfwd?

Thanks, everyone.  Appreciate the input.  After a frustrating evening/morning 
of rebuilding and troubleshooting, it turns out that I had some untested 
configuration commands in my /etc/sysctl.conf file that were negatively 
impacting the output throughput.  Going back to my current validated 
configuration allowed everything to work as expected.

Apologies for the spam and distraction for what wound up being a non-rsyslog 
issue!

Thanks!
Robert

Date: Fri, 6 Sep 2013 11:43:44 +0200
From: [email protected]
To: [email protected]
Subject: Re: [rsyslog] v7.4.4 and omfwd?

On Fri, Sep 6, 2013 at 1:04 AM, Robert McIntyre <[email protected]>wrote:

Sorry for all the spam, folks.  Final update before I wait for some help,
hopefully.  If I'm forwarding via UDP, and I monitor tcpdump long enough,
eventually I'll get a short burst of UDP traffic sent.

It looks like it's about every 30 seconds or so, and only 30-50 messages.
 This is on a system that is receiving 40k+ messages per second.  My
dequeue batch size is 500, but it's not doing it in chunks of 500, nor is
there any change if I remove the queuing config completely.

I'm open to suggestions. :)


mhhh.. sounds very strange. One thing you can do is enable impstats and
gather statistics. maybe this points us to where things go wrong. At least
we have processed message counts and know if imudp gets all messages and/or
thinks it sends all out.

Rainer

Thanks!
Robert


From: [email protected]
To: [email protected]
Date: Thu, 5 Sep 2013 15:39:45 -0700
Subject: Re: [rsyslog] v7.4.4 and omfwd?

Interestingly, if I switch my v7 action(type=) command to tcp/port
10514, it forwards fine, even with all the queuing config, etc., which
makes me really stumped, now. :|  I've verified that there's no firewall in
the way (disabled via iptables), and the exact same tcpdump commands work
to display the outgoing traffic on my production box running 5.8.10.

Does anyone else have omfwd working in 7.4.4 over UDP?

Thanks!
Robert


From: [email protected]
To: [email protected]
Date: Thu, 5 Sep 2013 15:12:45 -0700
Subject: Re: [rsyslog] v7.4.4 and omfwd?

No luck.  Removed the *.* and all queue config for the action in the
v7 action, and still does the same thing.  I've captured a debug log, but
am loathe to send it out since it contains the actual syslog events.  Can
you suggest things to look for in it?

Thanks!
Robert


To: [email protected]
From: [email protected]
Date: Thu, 5 Sep 2013 14:59:30 -0700
Subject: Re: [rsyslog] v7.4.4 and omfwd?

Thanks.  That particular statement (with the port and *.*) works
fine under 5.8.x, but I'll make this change and try it.

Thanks!
Robert
________________________________
From: David Lang<mailto:[email protected]>
Sent: 9/5/2013 2:50 PM
To: rsyslog-users<mailto:[email protected]>
Subject: Re: [rsyslog] v7.4.4 and omfwd?

by the way, with v7 you don't have to put in *.* you can just put
the action

@x.x.x.x

action(whatever)


instead of

*.* @x.x.x.x

*.* action(whatever)

David Lang



On Thu, 5 Sep 2013, Robert McIntyre wrote:

Date: Thu, 5 Sep 2013 14:45:47 -0700
From: Robert McIntyre <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: rsyslog-users <[email protected]>
Subject: Re: [rsyslog] v7.4.4 and omfwd?

Thanks.  I've stripped out everything but the ruleset queue, and
the new format action, and it's still doing the same thing.  It's not
queuing up, but still nothing hitting the wire.  I'll pull the queue stuff
as well, just in case.

Thanks!
Robert
________________________________
From: David Lang<mailto:[email protected]>
Sent: ?9/?5/?2013 2:30 PM
To: rsyslog-users<mailto:[email protected]>
Subject: Re: [rsyslog] v7.4.4 and omfwd?

for the legacy action you hsould not need to specify the port. Try
that and see
if it works

Then I would say try the new format , but simplify it, drop all he
queue stuff
and then work up from there.

David Lang



On Thu, 5 Sep 2013, Robert McIntyre wrote:

Date: Thu, 5 Sep 2013 13:15:29 -0700
From: Robert McIntyre <[email protected]>
Reply-To: rsyslog-users <[email protected]>
To: "[email protected]" <[email protected]>
Subject: Re: [rsyslog] v7.4.4 and omfwd?

A bit more info.  When using the legacy action (*.*
@x.x.x.x:514), it shows as being processed fine, but not traffic on the
wire.  When I use the action(type="omfwd"...) with a queue, it shows the
queue as expanding, but nothing going out.

The full omfwd action config is:

*.* action(name="Action_FwdReceiver1"
type="omfwd"
target="x.x.x.x"
protocol="udp"
port="514"
action.resumeretrycount="-1"
queue.dequeuebatchsize="500"
queue.checkpointinterval="20000"
queue.type="linkedlist"
queue.timeoutenqueue="0"
queue.filename="FwdReceiver1ActionQueue"
queue.size="8000000"
queue.highwatermark="7000000"
queue.lowwatermark="500000"
queue.maxdiskspace="100G"
queue.saveonshutdown="on")

Thanks!
Robert

From: [email protected]
To: [email protected]
Date: Thu, 5 Sep 2013 12:22:44 -0700
Subject: [rsyslog] v7.4.4 and omfwd?

I'm trying to get the last of my v7 migration done, and am
testing the omfwd action, and it doesn't seem to be working at all.

I've pasted most of my config below.  What I see is the ruleset
catches the messages, and hands them off to the queues.  The action even
reports that it has processed all the forwards, but TCPDUMP, and monitoring
the outgoing traffic doesn't have any of the outgoing forwards.  The other
actions (writes to file shares) seem to be working properly.

This repros whether I use the old forward action format (shown
below), or the newer action(type="omfwd"...) format.

Hoping someone can tell me what I'm missing. :)

Thanks!
Robert

# rsyslog v7 configuration file
# For more information see
/usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see
http://www.rsyslog.com/doc/troubleshoot.html
# NetSec Syslog Server config file v3.0 : See the end of the
file for revision information

#### MODULES ####
module(load="impstats" interval="300") # Provides periodic
performance statistics (this must be the first thing in rsyslog.conf)
module(load="imuxsock")     # Provides support for local system
logging (e.g. via logger command)
module(load="imklog")       # Provides kernel logging support
(previously done by rklogd)
module(load="imudp" timerequery="10000")# Provides UDP syslog
reception
#module(load="imptcp")     # Provides TCP syslog reception
#module(load="immark")      # Provides --MARK-- message
capability

#### GLOBAL DIRECTIVES ####
# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# Don't escape control characters
$EscapeControlCharactersOnReceive off
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Set the working directory for disk buffers
$WorkDirectory /syslogdata/buffer

#### TEMPLATES ####
# Filenames
template (name="FirstProdFile" type="string"
string="/firstprodshare/test/AP/%msg:F,32:3%-%$YEAR%-%$MONTH%-%$DAY%T%$HOUR%.%$QHOUR%")
template (name="SecondProdFile" type="string"
string="/secondprodshare/test/CS/%msg:F,32:3%-%$YEAR%-%$MONTH%-%$DAY%T%$HOUR%.%$QHOUR%")
template (name="SyslogStats" type="string"
string="/firstprodshare/00/syslogstats/%$YEAR%-%$MONTH%-%$DAY%-%$myhostname%-stats.txt")

# Messages
template (name="TemplateMessage" type="string"
string="<%PRI%>%syslogtag%%msg%\n")

#### MAIN MESSAGE QUEUE ####
# Establish the Main Message Queue
$MainMsgQueueFileName MainQueue
$MainMsgQueueSize 8000000
$MainMsgQueueHighWaterMark 7000000
$MainMsgQueueLowWaterMark 500000
$MainMsgQueueMaxFileSize 100G
$MainMsgQueueSaveOnShutdown on
$MainMsgQueueType LinkedList
$MainMsgQueueWorkerThreads 4
$MainMsgQueueCheckpointInterval 20000

#### RULES ####
### Local logging
ruleset(name="Ruleset_Local"){
kern.*
action(name="Action_local_kern" type="omfile" file="/var/log/messages")
*.info;mail.none;authpriv.none;cron.none;syslog.none
 action(name="Action_local_info" type="omfile" file="/var/log/messages")
authpriv.*
 action(name="Action_local_authpriv" type="omfile" file="/var/log/secure")
mail.*
 action(name="Action_local_mail" type="omfile" file="/var/log/maillog")
cron.*
 action(name="Action_local_cron" type="omfile" file="/var/log/cron")
*.emerg
action(name="Action_local_emerg" type="omusrmsg" users="*")
uucp,news.crit
 action(name="Action_local_news" type="omfile" file="/var/log/spooler")
local7.*
 action(name="Action_local_local7" type="omfile" file="/var/log/boot.log")
syslog.info            action(name="Action_SyslogStats"
type="omfile" DynaFile="SyslogStats")
syslog.info            action(name="Action_SyslogStats"
type="omfile" file="/var/log/syslog")
} # End ruleset Local

# Use ruleset Local as default
$DefaultRuleset Ruleset_Local
### End local logging

### Remote logging
ruleset (name="Ruleset_Remote"
   queue.type="linkedlist"
   queue.filename="RemoteRuleSetQueue"
   queue.size="8000000"
   queue.highwatermark="7000000"
   queue.lowwatermark="500000"
   queue.maxdiskspace="100G"
   queue.saveonshutdown="on"
   queue.workerthreads="4"){

# Action: Write to the first file share
*.*  action (name="Action_FirstFileShare"
    type="omfile"
    DynaFile="FirstProdFile"
    template="TemplateMessage"
    iobuffersize="262144"
    action.resumeretrycount="-1"
    queue.dequeuebatchsize="5000"
    queue.checkpointinterval="20000"
    queue.type="linkedlist"
    queue.timeoutenqueue="0"
    queue.filename="FirstProdShareActionQueue"
    queue.size="8000000"
    queue.highwatermark="7000000"
    queue.lowwatermark="500000"
    queue.maxdiskspace="100G"
    queue.saveonshutdown="on"
    queue.workerthreads="4"
    )

*.*  @x.x.x.x:514

# Action: Write to the second file share
*.*  action (name="Action_FileShare2"
    type="omfile"
    DynaFile="SecondProdFile"
    template="TemplateMessage"
    iobuffersize="262144"
    action.resumeretrycount="-1"
    queue.dequeuebatchsize="5000"
    queue.checkpointinterval="20000"
    queue.type="linkedlist"
    queue.timeoutenqueue="0"
    queue.filename="SecondProdShareActionQueue"
    queue.size="8000000"
    queue.highwatermark="7000000"
    queue.lowwatermark="500000"
    queue.maxdiskspace="100G"
    queue.saveonshutdown="on"
    queue.workerthreads="4")

} # End ruleset Remote

## Listeners ##
# Bind ruleset to UDP listener
input(inputname="RemoteUDP_514" type="imudp" port="514"
ruleset="Ruleset_Remote")



_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to