We were able to get the stream working but now we have new issues. For one, the streams are coming across garbled due to the compression, which is expected. How do we decompress the streams prior to forwarding them again? Some of the events will be forwarded to our long term storage and some to our SEIM but they are not being recognized.
________________________________________ From: [email protected] [[email protected]] on behalf of David Lang [[email protected]] Sent: Wednesday, October 07, 2015 7:07 AM To: rsyslog-users Subject: Re: [rsyslog] Complex forwarding and spoofing question On Wed, 7 Oct 2015, Dave Caplinger wrote: > A couple of things come to mind: > > 1) enable pstats: > > module(load="impstats" interval="600") > > (you may want a smaller interval such as 60 seconds instead) > > 2) modify your output ruleset to send the pstats logs to their own file; this > is one way: you can give a ruleset in the module load line rather than having to use a filter. David Lang > # Out: UDP-spoof to the local and also forward to remote rsyslog > ruleset(name="ruleset_eth0_514" > queue.type="LinkedList") { > if $syslogtag contains 'rsyslogd-pstats' then { > action(type="omfile" > queue.type="LinkedList" > name="pstats" > file="/var/log/pstats" > ) > stop > } > call action.local.udp515 > call action.fwd.remotebox > stop > } > > (This prevents the rsyslog-pstats lines from being sent to your other local > UDP syslog or to the remote server.) > > 3) Fire up rsyslog and watch /var/log/pstats. > > Most likely you'll see the output queue (omfwd.remotebox) grow until it hits > the highwatermark (default 90% of queue.size) which triggers disk-assist > mode, which it sounds like you never leave since it doesn't drop to the > lowwatermark (default 70% of queue.size). > > Since your issue seems to be with sending to the remote; make sure that > remote rsyslog receiver is also configured for > compression.mode="stream:always" like this sender. This setting is > all-or-nothing; rsyslog cannot dynamically determine if the sender is using > stream compression or not. So if you have multiple senders all shipping logs > to a receiver that is configured for stream compression, they all must use > stream compression. (You could support compressed and non-compressed inputs > on different ports or interfaces if necessary.) > > > -- > Dave Caplinger, Director, Technical Product Management > Solutionary — An NTT Group Security Company > >> On Oct 6, 2015, at 3:35 PM, Randy Baca <[email protected]> wrote: >> >> OK, I got everything updated and we are up and running. >> >> However, there appears to be an issue with disk queuing. When we fire up >> rsyslog the remotebox is only receiving a short burst of events for a few >> seconds and then about 1 or 2 events per second after that. After the few >> seconds when the burst is over the disk queue on the local system then >> starts creating spool files. They don't appear to be flushing. The local >> system receives about 1k events per second. They are all making it to the >> local UDP spoof but not the remotebox, even after playing around with the >> queue settings. >> >> Here is the current conf: >> >> module (load="imklog") >> module (load="imuxsock") >> module (load="imudp") >> module (load="imtcp") >> module (load="omudpspoof") >> >> # Global Configuration >> global ( >> parser.escapeControlCharactersCStyle="on" >> workDirectory="/var/spool/rsyslog" >> localHostname="hostname" >> ) >> >> # In: 514/TCP (uses the same queue as UDP) >> input(type="imtcp" >> address="10.1.2.3" >> port="514" >> ruleset="ruleset_eth0_514") >> >> # In: 514/UDP (uses the same queue as TCP) >> input(type="imudp" >> address="10.1.2.3" >> port="514" >> ruleset="ruleset_eth0_514") >> >> # Out: UDP-spoof to the local and also forward to remote rsyslog >> ruleset(name="ruleset_eth0_514" >> queue.type="LinkedList") { >> call action.local.udp515 >> call action.fwd.remotebox >> stop >> } >> >> # Fwd to localhost:udp/515 >> ruleset(name="action.local.udp515") { >> action(type="omudpspoof" >> name="omudpspoof.udp515" >> target="127.0.0.1" >> port="515") >> } >> >> # Fwd to remotebox w/ compression and local disk queueing >> ruleset(name="action.fwd.remotebox") { >> action(type="omfwd" >> name="omfwd.remotebox" >> queue.spoolDirectory="/var/spool/rsyslog_tcp" >> # queue.lowwatermark="2000" >> # queue.highwatermark="1000000" >> queue.workerthreads="2" >> # queue.type="Disk" >> queue.type="LinkedList" >> # queue.filename="omfwd_remotebox" >> # queue.size="100000" >> queue.maxdiskspace="20G" >> queue.saveonshutdown="on" >> action.resumeretrycount="-1" >> target="172.22.22.22" >> port="514" >> protocol="tcp" >> compression.mode="stream:always") >> } >> >> >> >> ________________________________________ >> From: [email protected] [[email protected]] >> on behalf of Dave Caplinger [[email protected]] >> Sent: Monday, October 05, 2015 5:40 AM >> To: rsyslog-users >> Subject: Re: [rsyslog] Complex forwarding and spoofing question >> >> David's right, it's a separate package. Sorry about that. Here's a list of >> all the 64-bit rsyslog packages (from 'yum search rsyslog') from a system I >> have access to; please verify you see something similar from yours. >> >> rsyslog-gnutls.x86_64 : TLS protocol support for rsyslog >> rsyslog-gssapi.x86_64 : GSSAPI authentication and encryption support for >> rsyslog >> rsyslog-mysql.x86_64 : MySQL support for rsyslog >> rsyslog-pgsql.x86_64 : PostgresSQL support for rsyslog >> rsyslog-relp.x86_64 : RELP protocol support for rsyslog >> rsyslog-snmp.x86_64 : SNMP protocol support for rsyslog >> rsyslog.x86_64 : Enhanced system logging and kernel message trapping daemons >> rsyslog-elasticsearch.x86_64 : Provides the omelasticsearch module >> rsyslog-mmjsonparse.x86_64 : mmjsonparse support >> rsyslog-mmutf8fix.x86_64 : mmutf8fix support >> rsyslog-udpspoof.x86_64 : Provides the omudpspoof module >> >> Then, of course: 'yum install rsyslog-udpspoof'. >> >> -- >> Dave Caplinger, Director, Technical Product Management | 402.361.3063 | >> Solutionary — An NTT Group Security Company >> >>> On Oct 3, 2015, at 1:02 AM, David Lang <[email protected]> wrote: >>> >>> It may be a separate package (I don't know how to query this through yum) >>> >>> in the meantime, try addding the following >>> >>> $template raw,"%rawmsg%" >>> >>> and then change the forwarding via localhost from omudpspoof to udp with >>> the template raw. >>> >>> David Lang >>> >>> On Fri, 2 Oct 2015, Randy Baca wrote: >>> >>>> Date: Fri, 2 Oct 2015 23:46:27 +0000 >>>> From: Randy Baca <[email protected]> >>>> Reply-To: rsyslog-users <[email protected]> >>>> To: rsyslog-users <[email protected]> >>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question >>>> Upgraded to 8.13. Looks like now the omudpspoof.so file is missing. Does >>>> it not get installed automatically with the upgrade? I checked in >>>> /lib64/rsyslog/ and it is not there. It also didn't like the number I had >>>> set for the highwatermark so I removed it, and now this is the only error >>>> left. >>>> >>>> [root@host etc]# /sbin/rsyslogd -N2 >>>> rsyslogd: version 8.13.0, config validation run (level 2), master config >>>> /etc/rsyslog.conf >>>> rsyslogd: could not load module '/lib64/rsyslog/omudpspoof.so', dlopen: >>>> /lib64/rsyslog/omudpspoof.so: cannot open shared object file: No such file >>>> or directory [v8.13.0 try http://www.rsyslog.com/e/2066 ] >>>> rsyslogd: module name 'omudpspoof' is unknown [v8.13.0 try >>>> http://www.rsyslog.com/e/2209 ] >>>> rsyslogd: error during parsing file /etc/rsyslog.conf, on or before line >>>> 32: errors occured in file '/etc/rsyslog.conf' around line 32 [v8.13.0 try >>>> http://www.rsyslog.com/e/2207 ] >>>> >>>> module (load="imklog") >>>> module (load="imuxsock") >>>> module (load="imudp") >>>> module (load="imtcp") >>>> Module (load="omudpspoof") >>>> # In: 514/TCP (uses the same queue as UDP) >>>> input(type="imtcp" >>>> address="10.1.2.3" >>>> port="514" >>>> ruleset="ruleset_eth0_514") >>>> # In: 514/UDP (uses the same queue as TCP) >>>> input(type="imudp" >>>> address="10.1.2.3" >>>> port="514" >>>> ruleset="ruleset_eth0_514") >>>> # Out: UDP-spoof to the local and also forward to remote rsyslog >>>> ruleset(name="ruleset_eth0_514" >>>> queue.type="LinkedList") { >>>> call action.local.udp515 >>>> call action.fwd.remotebox >>>> stop >>>> } >>>> # Fwd to localhost:udp/515 >>>> ruleset(name="action.local.udp515") { >>>> action(type="omudpspoof" >>>> name="omudpspoof.udp515" >>>> target="127.0.0.1" >>>> port="515") >>>> } >>>> # Fwd to remotebox w/ compression and local disk queueing >>>> ruleset(name="action.fwd.remotebox") { >>>> action(type="omfwd" >>>> name="omfwd.remotebox" >>>> queue.type="LinkedList" >>>> queue.filename="omfwd_remotebox" >>>> queue.size="10000" >>>> queue.maxdiskspace="2G" >>>> queue.saveonshutdown="on" >>>> action.resumeretrycount="-1" >>>> target="172.22.22.22" >>>> port="514" >>>> protocol="tcp" >>>> compression.mode="stream:always") >>>> } >>>> >>>> >>>> ________________________________________ >>>> From: [email protected] >>>> [[email protected]] on behalf of Dave Caplinger >>>> [[email protected]] >>>> Sent: Friday, October 02, 2015 2:40 PM >>>> To: rsyslog-users >>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question >>>> >>>> I think stream compression was added to rsyslog around version 7.2 or so. >>>> If i recall correctly, CentOS 6 includes rsyslog v5.8, so you'll >>>> definitely want to upgrade your rsyslog version. >>>> >>>> See http://www.rsyslog.com/rhelcentos-rpms/ for instructions on how to add >>>> the v8-stable repo to your /etc/yum.repos.d/ directory. >>>> >>>> -- >>>> Dave Caplinger, Director, Technical Product Management | 402.361.3063 | >>>> Solutionary — An NTT Group Security Company >>>> >>>>> On Oct 2, 2015, at 12:35 PM, Randy Baca <[email protected]> wrote: >>>>> >>>>> Thanks for the conf validation switch. Looks like we have a >>>>> compatibility issue. Most of our existing syslog servers are running on >>>>> CentOS 6. Will an upgrade of rsyslog resolve this, or do we need to >>>>> convert the conf to legacy? >>>>> >>>>> module(load="imklog") >>>>> module(load="imuxsock") >>>>> module(load="imudp") >>>>> module(load="imtcp") >>>>> # In: 514/TCP (uses the same queue as UDP) >>>>> input(type="imtcp" >>>>> address="10.1.2.3" >>>>> port="514" >>>>> ruleset="ruleset_eth0_514") >>>>> # In: 514/UDP (uses the same queue as TCP) >>>>> input(type="imudp" >>>>> address="10.1.2.3" >>>>> port="514" >>>>> ruleset="ruleset_eth0_514") >>>>> # Out: UDP-spoof to the local and also forward to remote rsyslog >>>>> ruleset(name="ruleset_eth0_514" >>>>> queue.highwatermark 50000 >>>>> queue.fulldelaymark 20000 >>>>> queue.lowwatermark 2000 >>>>> queue.type="LinkedList") { >>>>> call action.local.udp515 >>>>> call action.fwd.remotebox >>>>> stop >>>>> } >>>>> # Fwd to localhost:udp/515 >>>>> ruleset(name="action.local.udp515") { >>>>> action(type="omudpspoof" >>>>> name="omudpspoof.local515" >>>>> target="127.0.0.1" >>>>> port="515") >>>>> } >>>>> # Fwd to remotebox w/ compression and local disk queueing >>>>> ruleset(name="action.fwd.remotebox") { >>>>> action(type="omfwd" >>>>> name="omfwd.remotebox" >>>>> queue.type="LinkedList" >>>>> queue.filename="omfwd_remotebox" >>>>> queue.size="10000" >>>>> queue.maxdiskspace="2G" >>>>> queue.saveonshutdown="on" >>>>> action.resumeretrycount="-1" >>>>> target="172.22.22.22" >>>>> port="514" >>>>> protocol="tcp" >>>>> compression.mode="stream:always") >>>>> } >>>>> >>>>> >>>>> Regards, >>>>> >>>>> >>>>> >>>>> Randy Baca >>>>> >>>>> ________________________________________ >>>>> From: [email protected] >>>>> [[email protected]] on behalf of David Lang >>>>> [[email protected]] >>>>> Sent: Thursday, October 01, 2015 7:53 PM >>>>> To: rsyslog-users >>>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question >>>>> >>>>> I'm not seeing anything obviously wrong, but you left out part of the >>>>> config >>>>> (the module loading if nothing else) >>>>> >>>>> do >>>>> rsyslogd -N2 >>>>> to have rsyslog do a syntax check of the config and see if it's happy. >>>>> >>>>> David Lang >>>>> >>>>> On Fri, 2 Oct 2015, Randy Baca wrote: >>>>> >>>>>> Date: Fri, 2 Oct 2015 00:26:11 +0000 >>>>>> From: Randy Baca <[email protected]> >>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>> To: rsyslog-users <[email protected]> >>>>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question >>>>>> >>>>>> This is what I am using but nothing in a tcpdump going to local on port >>>>>> 515 or remote on port 514. >>>>>> >>>>>> # In: 514/TCP (uses the same queue as UDP) >>>>>> input(type="imtcp" >>>>>> address="10.1.2.3" >>>>>> port="514" >>>>>> ruleset="ruleset_eth0_514") >>>>>> >>>>>> # In: 514/UDP (uses the same queue as TCP) >>>>>> input(type="imudp" >>>>>> address="10.1.2.3" >>>>>> port="514" >>>>>> ruleset="ruleset_eth0_514") >>>>>> >>>>>> # Queue: UDP-spoof to the local and also forward to remote rsyslog >>>>>> ruleset(name="ruleset_eth0_514" >>>>>> queue.highwatermark 50000 >>>>>> queue.fulldelaymark 20000 >>>>>> queue.lowwatermark 2000 >>>>>> queue.type="LinkedList") { >>>>>> call action.local.udp515 >>>>>> call action.fwd.remotebox >>>>>> stop >>>>>> } >>>>>> >>>>>> # Fwd to localhost:udp/515 >>>>>> ruleset(name="action.local.udp515") { >>>>>> action(type="omudpspoof" >>>>>> name="omudpspoof.local515" >>>>>> target="127.0.0.1" >>>>>> port="515") >>>>>> } >>>>>> >>>>>> # Fwd to remotebox w/ compression and local disk queueing >>>>>> ruleset(name="action.fwd.remotebox") { >>>>>> action(type="omfwd" >>>>>> name="omfwd.remotebox" >>>>>> queue.type="LinkedList" >>>>>> queue.filename="omfwd_remotebox" >>>>>> queue.size="10000" >>>>>> queue.maxdiskspace="2G" >>>>>> queue.saveonshutdown="on" >>>>>> action.resumeretrycount="-1" >>>>>> target="172.22.22.22" >>>>>> port="514" >>>>>> protocol="tcp" >>>>>> compression.mode="stream:always") >>>>>> } >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ________________________________________ >>>>>> From: [email protected] >>>>>> [[email protected]] on behalf of Dave Caplinger >>>>>> [[email protected]] >>>>>> Sent: Thursday, October 01, 2015 2:09 PM >>>>>> To: rsyslog-users >>>>>> Subject: Re: [rsyslog] Complex forwarding and spoofing question >>>>>> >>>>>> Looks like it's "queue.fulldelaymark" (and I presume >>>>>> "queue.lightdelaymark" may be related, but there's no description at >>>>>> http://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html so >>>>>> I'm not sure yet what it does...) >>>>>> >>>>>> Thanks! >>>>>> >>>>>> -- >>>>>> Dave Caplinger, Director, Technical Product Management | 402.361.3063 | >>>>>> Solutionary — An NTT Group Security Company >>>>>> >>>>>>> On Oct 1, 2015, at 4:03 PM, David Lang <[email protected]> wrote: >>>>>>> >>>>>>> There is a queue parameter that tells rsyslog that if the queue is >>>>>>> larger than X, stop accepting inputs that can be delayed (like TCP) so >>>>>>> that there is space left for a burst of traffic from inputs that can't >>>>>>> be delayed (like UDP) >>>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>> DON'T LIKE THAT. >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
