A couple of things come to mind:
1) enable pstats:
module(load="impstats" interval="600")
(you may want a smaller interval such as 60 seconds instead)
2) modify your output ruleset to send the pstats logs to their own file; this
is one way:
# Out: UDP-spoof to the local and also forward to remote rsyslog
ruleset(name="ruleset_eth0_514"
queue.type="LinkedList") {
if $syslogtag contains 'rsyslogd-pstats' then {
action(type="omfile"
queue.type="LinkedList"
name="pstats"
file="/var/log/pstats"
)
stop
}
call action.local.udp515
call action.fwd.remotebox
stop
}
(This prevents the rsyslog-pstats lines from being sent to your other local UDP
syslog or to the remote server.)
3) Fire up rsyslog and watch /var/log/pstats.
Most likely you'll see the output queue (omfwd.remotebox) grow until it hits
the highwatermark (default 90% of queue.size) which triggers disk-assist mode,
which it sounds like you never leave since it doesn't drop to the lowwatermark
(default 70% of queue.size).
Since your issue seems to be with sending to the remote; make sure that remote
rsyslog receiver is also configured for compression.mode="stream:always" like
this sender. This setting is all-or-nothing; rsyslog cannot dynamically
determine if the sender is using stream compression or not. So if you have
multiple senders all shipping logs to a receiver that is configured for stream
compression, they all must use stream compression. (You could support
compressed and non-compressed inputs on different ports or interfaces if
necessary.)
--
Dave Caplinger, Director, Technical Product Management
Solutionary — An NTT Group Security Company
> On Oct 6, 2015, at 3:35 PM, Randy Baca <[email protected]> wrote:
>
> OK, I got everything updated and we are up and running.
>
> However, there appears to be an issue with disk queuing. When we fire up
> rsyslog the remotebox is only receiving a short burst of events for a few
> seconds and then about 1 or 2 events per second after that. After the few
> seconds when the burst is over the disk queue on the local system then starts
> creating spool files. They don't appear to be flushing. The local system
> receives about 1k events per second. They are all making it to the local UDP
> spoof but not the remotebox, even after playing around with the queue
> settings.
>
> Here is the current conf:
>
> module (load="imklog")
> module (load="imuxsock")
> module (load="imudp")
> module (load="imtcp")
> module (load="omudpspoof")
>
> # Global Configuration
> global (
> parser.escapeControlCharactersCStyle="on"
> workDirectory="/var/spool/rsyslog"
> localHostname="hostname"
> )
>
> # In: 514/TCP (uses the same queue as UDP)
> input(type="imtcp"
> address="10.1.2.3"
> port="514"
> ruleset="ruleset_eth0_514")
>
> # In: 514/UDP (uses the same queue as TCP)
> input(type="imudp"
> address="10.1.2.3"
> port="514"
> ruleset="ruleset_eth0_514")
>
> # Out: UDP-spoof to the local and also forward to remote rsyslog
> ruleset(name="ruleset_eth0_514"
> queue.type="LinkedList") {
> call action.local.udp515
> call action.fwd.remotebox
> stop
> }
>
> # Fwd to localhost:udp/515
> ruleset(name="action.local.udp515") {
> action(type="omudpspoof"
> name="omudpspoof.udp515"
> target="127.0.0.1"
> port="515")
> }
>
> # Fwd to remotebox w/ compression and local disk queueing
> ruleset(name="action.fwd.remotebox") {
> action(type="omfwd"
> name="omfwd.remotebox"
> queue.spoolDirectory="/var/spool/rsyslog_tcp"
> # queue.lowwatermark="2000"
> # queue.highwatermark="1000000"
> queue.workerthreads="2"
> # queue.type="Disk"
> queue.type="LinkedList"
> # queue.filename="omfwd_remotebox"
> # queue.size="100000"
> queue.maxdiskspace="20G"
> queue.saveonshutdown="on"
> action.resumeretrycount="-1"
> target="172.22.22.22"
> port="514"
> protocol="tcp"
> compression.mode="stream:always")
> }
>
>
>
> ________________________________________
> From: [email protected] [[email protected]]
> on behalf of Dave Caplinger [[email protected]]
> Sent: Monday, October 05, 2015 5:40 AM
> To: rsyslog-users
> Subject: Re: [rsyslog] Complex forwarding and spoofing question
>
> David's right, it's a separate package. Sorry about that. Here's a list of
> all the 64-bit rsyslog packages (from 'yum search rsyslog') from a system I
> have access to; please verify you see something similar from yours.
>
> rsyslog-gnutls.x86_64 : TLS protocol support for rsyslog
> rsyslog-gssapi.x86_64 : GSSAPI authentication and encryption support for
> rsyslog
> rsyslog-mysql.x86_64 : MySQL support for rsyslog
> rsyslog-pgsql.x86_64 : PostgresSQL support for rsyslog
> rsyslog-relp.x86_64 : RELP protocol support for rsyslog
> rsyslog-snmp.x86_64 : SNMP protocol support for rsyslog
> rsyslog.x86_64 : Enhanced system logging and kernel message trapping daemons
> rsyslog-elasticsearch.x86_64 : Provides the omelasticsearch module
> rsyslog-mmjsonparse.x86_64 : mmjsonparse support
> rsyslog-mmutf8fix.x86_64 : mmutf8fix support
> rsyslog-udpspoof.x86_64 : Provides the omudpspoof module
>
> Then, of course: 'yum install rsyslog-udpspoof'.
>
> --
> Dave Caplinger, Director, Technical Product Management | 402.361.3063 |
> Solutionary — An NTT Group Security Company
>
>> On Oct 3, 2015, at 1:02 AM, David Lang <[email protected]> wrote:
>>
>> It may be a separate package (I don't know how to query this through yum)
>>
>> in the meantime, try addding the following
>>
>> $template raw,"%rawmsg%"
>>
>> and then change the forwarding via localhost from omudpspoof to udp with the
>> template raw.
>>
>> David Lang
>>
>> On Fri, 2 Oct 2015, Randy Baca wrote:
>>
>>> Date: Fri, 2 Oct 2015 23:46:27 +0000
>>> From: Randy Baca <[email protected]>
>>> Reply-To: rsyslog-users <[email protected]>
>>> To: rsyslog-users <[email protected]>
>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question
>>> Upgraded to 8.13. Looks like now the omudpspoof.so file is missing. Does
>>> it not get installed automatically with the upgrade? I checked in
>>> /lib64/rsyslog/ and it is not there. It also didn't like the number I had
>>> set for the highwatermark so I removed it, and now this is the only error
>>> left.
>>>
>>> [root@host etc]# /sbin/rsyslogd -N2
>>> rsyslogd: version 8.13.0, config validation run (level 2), master config
>>> /etc/rsyslog.conf
>>> rsyslogd: could not load module '/lib64/rsyslog/omudpspoof.so', dlopen:
>>> /lib64/rsyslog/omudpspoof.so: cannot open shared object file: No such file
>>> or directory [v8.13.0 try http://www.rsyslog.com/e/2066 ]
>>> rsyslogd: module name 'omudpspoof' is unknown [v8.13.0 try
>>> http://www.rsyslog.com/e/2209 ]
>>> rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line
>>> 32: errors occured in file '/etc/rsyslog.conf' around line 32 [v8.13.0 try
>>> http://www.rsyslog.com/e/2207 ]
>>>
>>> module (load="imklog")
>>> module (load="imuxsock")
>>> module (load="imudp")
>>> module (load="imtcp")
>>> Module (load="omudpspoof")
>>> # In: 514/TCP (uses the same queue as UDP)
>>> input(type="imtcp"
>>> address="10.1.2.3"
>>> port="514"
>>> ruleset="ruleset_eth0_514")
>>> # In: 514/UDP (uses the same queue as TCP)
>>> input(type="imudp"
>>> address="10.1.2.3"
>>> port="514"
>>> ruleset="ruleset_eth0_514")
>>> # Out: UDP-spoof to the local and also forward to remote rsyslog
>>> ruleset(name="ruleset_eth0_514"
>>> queue.type="LinkedList") {
>>> call action.local.udp515
>>> call action.fwd.remotebox
>>> stop
>>> }
>>> # Fwd to localhost:udp/515
>>> ruleset(name="action.local.udp515") {
>>> action(type="omudpspoof"
>>> name="omudpspoof.udp515"
>>> target="127.0.0.1"
>>> port="515")
>>> }
>>> # Fwd to remotebox w/ compression and local disk queueing
>>> ruleset(name="action.fwd.remotebox") {
>>> action(type="omfwd"
>>> name="omfwd.remotebox"
>>> queue.type="LinkedList"
>>> queue.filename="omfwd_remotebox"
>>> queue.size="10000"
>>> queue.maxdiskspace="2G"
>>> queue.saveonshutdown="on"
>>> action.resumeretrycount="-1"
>>> target="172.22.22.22"
>>> port="514"
>>> protocol="tcp"
>>> compression.mode="stream:always")
>>> }
>>>
>>>
>>> ________________________________________
>>> From: [email protected] [[email protected]]
>>> on behalf of Dave Caplinger [[email protected]]
>>> Sent: Friday, October 02, 2015 2:40 PM
>>> To: rsyslog-users
>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question
>>>
>>> I think stream compression was added to rsyslog around version 7.2 or so.
>>> If i recall correctly, CentOS 6 includes rsyslog v5.8, so you'll definitely
>>> want to upgrade your rsyslog version.
>>>
>>> See http://www.rsyslog.com/rhelcentos-rpms/ for instructions on how to add
>>> the v8-stable repo to your /etc/yum.repos.d/ directory.
>>>
>>> --
>>> Dave Caplinger, Director, Technical Product Management | 402.361.3063 |
>>> Solutionary — An NTT Group Security Company
>>>
>>>> On Oct 2, 2015, at 12:35 PM, Randy Baca <[email protected]> wrote:
>>>>
>>>> Thanks for the conf validation switch. Looks like we have a compatibility
>>>> issue. Most of our existing syslog servers are running on CentOS 6. Will
>>>> an upgrade of rsyslog resolve this, or do we need to convert the conf to
>>>> legacy?
>>>>
>>>> module(load="imklog")
>>>> module(load="imuxsock")
>>>> module(load="imudp")
>>>> module(load="imtcp")
>>>> # In: 514/TCP (uses the same queue as UDP)
>>>> input(type="imtcp"
>>>> address="10.1.2.3"
>>>> port="514"
>>>> ruleset="ruleset_eth0_514")
>>>> # In: 514/UDP (uses the same queue as TCP)
>>>> input(type="imudp"
>>>> address="10.1.2.3"
>>>> port="514"
>>>> ruleset="ruleset_eth0_514")
>>>> # Out: UDP-spoof to the local and also forward to remote rsyslog
>>>> ruleset(name="ruleset_eth0_514"
>>>> queue.highwatermark 50000
>>>> queue.fulldelaymark 20000
>>>> queue.lowwatermark 2000
>>>> queue.type="LinkedList") {
>>>> call action.local.udp515
>>>> call action.fwd.remotebox
>>>> stop
>>>> }
>>>> # Fwd to localhost:udp/515
>>>> ruleset(name="action.local.udp515") {
>>>> action(type="omudpspoof"
>>>> name="omudpspoof.local515"
>>>> target="127.0.0.1"
>>>> port="515")
>>>> }
>>>> # Fwd to remotebox w/ compression and local disk queueing
>>>> ruleset(name="action.fwd.remotebox") {
>>>> action(type="omfwd"
>>>> name="omfwd.remotebox"
>>>> queue.type="LinkedList"
>>>> queue.filename="omfwd_remotebox"
>>>> queue.size="10000"
>>>> queue.maxdiskspace="2G"
>>>> queue.saveonshutdown="on"
>>>> action.resumeretrycount="-1"
>>>> target="172.22.22.22"
>>>> port="514"
>>>> protocol="tcp"
>>>> compression.mode="stream:always")
>>>> }
>>>>
>>>>
>>>> Regards,
>>>>
>>>>
>>>>
>>>> Randy Baca
>>>>
>>>> ________________________________________
>>>> From: [email protected]
>>>> [[email protected]] on behalf of David Lang [[email protected]]
>>>> Sent: Thursday, October 01, 2015 7:53 PM
>>>> To: rsyslog-users
>>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question
>>>>
>>>> I'm not seeing anything obviously wrong, but you left out part of the
>>>> config
>>>> (the module loading if nothing else)
>>>>
>>>> do
>>>> rsyslogd -N2
>>>> to have rsyslog do a syntax check of the config and see if it's happy.
>>>>
>>>> David Lang
>>>>
>>>> On Fri, 2 Oct 2015, Randy Baca wrote:
>>>>
>>>>> Date: Fri, 2 Oct 2015 00:26:11 +0000
>>>>> From: Randy Baca <[email protected]>
>>>>> Reply-To: rsyslog-users <[email protected]>
>>>>> To: rsyslog-users <[email protected]>
>>>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question
>>>>>
>>>>> This is what I am using but nothing in a tcpdump going to local on port
>>>>> 515 or remote on port 514.
>>>>>
>>>>> # In: 514/TCP (uses the same queue as UDP)
>>>>> input(type="imtcp"
>>>>> address="10.1.2.3"
>>>>> port="514"
>>>>> ruleset="ruleset_eth0_514")
>>>>>
>>>>> # In: 514/UDP (uses the same queue as TCP)
>>>>> input(type="imudp"
>>>>> address="10.1.2.3"
>>>>> port="514"
>>>>> ruleset="ruleset_eth0_514")
>>>>>
>>>>> # Queue: UDP-spoof to the local and also forward to remote rsyslog
>>>>> ruleset(name="ruleset_eth0_514"
>>>>> queue.highwatermark 50000
>>>>> queue.fulldelaymark 20000
>>>>> queue.lowwatermark 2000
>>>>> queue.type="LinkedList") {
>>>>> call action.local.udp515
>>>>> call action.fwd.remotebox
>>>>> stop
>>>>> }
>>>>>
>>>>> # Fwd to localhost:udp/515
>>>>> ruleset(name="action.local.udp515") {
>>>>> action(type="omudpspoof"
>>>>> name="omudpspoof.local515"
>>>>> target="127.0.0.1"
>>>>> port="515")
>>>>> }
>>>>>
>>>>> # Fwd to remotebox w/ compression and local disk queueing
>>>>> ruleset(name="action.fwd.remotebox") {
>>>>> action(type="omfwd"
>>>>> name="omfwd.remotebox"
>>>>> queue.type="LinkedList"
>>>>> queue.filename="omfwd_remotebox"
>>>>> queue.size="10000"
>>>>> queue.maxdiskspace="2G"
>>>>> queue.saveonshutdown="on"
>>>>> action.resumeretrycount="-1"
>>>>> target="172.22.22.22"
>>>>> port="514"
>>>>> protocol="tcp"
>>>>> compression.mode="stream:always")
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> From: [email protected]
>>>>> [[email protected]] on behalf of Dave Caplinger
>>>>> [[email protected]]
>>>>> Sent: Thursday, October 01, 2015 2:09 PM
>>>>> To: rsyslog-users
>>>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question
>>>>>
>>>>> Looks like it's "queue.fulldelaymark" (and I presume
>>>>> "queue.lightdelaymark" may be related, but there's no description at
>>>>> http://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html so
>>>>> I'm not sure yet what it does...)
>>>>>
>>>>> Thanks!
>>>>>
>>>>> --
>>>>> Dave Caplinger, Director, Technical Product Management | 402.361.3063 |
>>>>> Solutionary — An NTT Group Security Company
>>>>>
>>>>>> On Oct 1, 2015, at 4:03 PM, David Lang <[email protected]> wrote:
>>>>>>
>>>>>> There is a queue parameter that tells rsyslog that if the queue is
>>>>>> larger than X, stop accepting inputs that can be delayed (like TCP) so
>>>>>> that there is space left for a burst of traffic from inputs that can't
>>>>>> be delayed (like UDP)
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>> _______________________________________________
>>>>> rsyslog mailing list
>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>> http://www.rsyslog.com/professional-services/
>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>> DON'T LIKE THAT.
>>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>>> LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
