Just a quick follow-up: I currently work with Rick on this issue. It looks like there are actually two issues in one:
1) in v5, it can happen that an action in front of an included get's reordered, and this seems to have happened here. This is actually the second time in many years that I see that. I won't fix that in v5, as it would probably lead to something like the v7 engine ;) Rick has moved to v7. 2) omudpspoof discards messages that are larger than the network-supported MTU. I am working to fragment them. Note that I know that some receivers have problems with fragmented UDP, but hey, what else shall we do? Messages over 64K can never be transmitted in a single UDP packet, and as such 7.2.5 will now truncate them to 64K. I will probably change v5 so that >1472 byte messages get truncated, but the message sent instead of being discarded. Will keep you posted when there is notable news (but maybe you just see a new release announcement if everything works out and no new problems pop up). Rainer > -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Rick Brown > Sent: Friday, December 14, 2012 3:24 PM > To: rsyslog-users > Subject: Re: [rsyslog] imfile and omudpspoof > > ----- Original Message ----- > > From: "Rainer Gerhards" <[email protected]> > > To: "rsyslog-users" <[email protected]> > > Sent: Friday, December 14, 2012 9:06:10 AM > > Subject: Re: [rsyslog] imfile and omudpspoof > > > > > -----Original Message----- > > > From: [email protected] [mailto:rsyslog- > > > [email protected]] On Behalf Of Rick Brown > > > Sent: Friday, December 14, 2012 3:04 PM > > > To: rsyslog-users > > > Subject: Re: [rsyslog] imfile and omudpspoof > > > > > > ----- Original Message ----- > > > > From: "Rainer Gerhards" <[email protected]> > > > > To: "rsyslog-users" <[email protected]> > > > > Sent: Friday, December 14, 2012 2:40:05 AM > > > > Subject: Re: [rsyslog] imfile and omudpspoof > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Rick Brown [mailto:[email protected]] > > > > > Sent: Thursday, December 13, 2012 7:06 PM > > > > > To: Rainer Gerhards > > > > > Subject: Re: [rsyslog] imfile and omudpspoof > > > > > > > > > > You're right, it's rather large. I gzipp'd the output and > > > > > posted > > > > > it > > > > > at xxxx > > > > > > > > Mhhh... That log does not include anything from imfile. Did you > > > > accidently capture the wrong system? > > > > > > > > > > I captured from the syslog server in the middle. Let me try to be > > > clearer in stating the problem. > > > > > > Client A scrapes log files with imfile and sends them to my syslog > > > server. The syslog server is configured to forward the messages to > > > a > > > SIEM using imudpspoof. The syslog server successfully writes the > > > logs > > > to disk, however not all messages are being spoofed and sent on. > > > The > > > person who runs the SIEM noted the difference and I then observed > > > tcpdumps on the syslog server. Watching in two terminals: > > > > > > tcpdump host A and host syslog > > > > > > and: > > > tcpdump host syslog and host SIEM > > > > > > I see many more messages in the top terminal than the bottom. On > > > first > > > glance, it looked like none of the messages generated by imfile on > > > client A were making it to the SIEM, but now that I've stared at it > > > long enough, I do occassionally see a couple go by. If I restarted > > > rsyslog while watching the tcpdumps, I see a burst of the imfile > > > messages get spoofed, then they stop for a while, then a few will > > > trickle by. > > > > OK - I misunderstood the initial description. So let's continue with > > that debug log file. Can you tell me which logs lines are > > imfile-originated (and not delivered) and which one are not? > > > > Rainer > > Ahh, good. All of the imfile messages are set to facility news and we > no longer actually run news servers, so it's a safe bet that any news > messages is generated from imfile. I've been particularly > concentrating on a host cas1, who's ip ends in .165.151, as it > generates quite a bit of application logs that I scrape with imfile. > And the spoof target address ends in .165.95. > > > > > > > > > > > From what I have seen, it may also happen that the debug log info > > > > is > > > > not sufficient. Just treat this as advance warning ;) > > > > > > > > We can probably get around a limitation by creating an additional > > > > log > > > > file, so that I can see all message properties. Please add > > > > > > > > *.* /var/log/rsyslog-props.log;RSYSLOG_DebugFormat > > > > > > > > To your config (the file name is obviously irrelevant, but the > > > > template is important!). > > > > > > > > Make sure that the debug log and the property file are from the > > > > same > > > > run. > > > > > > > > Rainer > > > > > > > > > Can do. I'll go look into this now. > > > > > > > > > > > I see that some of the imfile generated messages are making it > > > > > through > > > > > to the SIEM from time to time, but the vast majority are being > > > > > dropped. > > > > > I have to wonder if I'm overflowing a buffer in the output > > > > > modules. > > > > > > > > > > Thanks for taking a look! > > > > > > > > > > ----- Original Message ----- > > > > > > From: "Rainer Gerhards" <[email protected]> > > > > > > To: "rsyslog-users" <[email protected]> > > > > > > Sent: Thursday, December 13, 2012 10:05:02 AM > > > > > > Subject: Re: [rsyslog] imfile and omudpspoof > > > > > > > > > > > > Can you post a (sufficiently large or complete) debug log > > > > > > that > > > > > > exposes the problem? The list will probably reject it, so it > > > > > > is > > > > > > best > > > > > > to put it on something like pastbin. > > > > > > > > > > > > Rainer > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: [email protected] [mailto:rsyslog- > > > > > > > [email protected]] On Behalf Of Rick Brown > > > > > > > Sent: Thursday, December 13, 2012 4:01 PM > > > > > > > To: rsyslog-users > > > > > > > Subject: Re: [rsyslog] imfile and omudpspoof > > > > > > > > > > > > > > > > On Tue, 11 Dec 2012, Rick Brown wrote: > > > > > > > > > > > > > > > > > > > I use imfile to gather application logs such as > > > > > > > > > > apache, > > > > > > > > > > tomcat, > > > > > > > > > > php, etc. and send those on to my syslog server along > > > > > > > > > > with > > > > > > > > > > the > > > > > > > > > > client machines normal syslog traffic. My syslog > > > > > > > > > > server > > > > > > > > > > then > > > > > > > > > > dutifully writes all the messages locally and > > > > > > > > > > additionally > > > > > > > > > > forwards the messages on to a SIEM product via > > > > > > > > > > omudpspoof. > > > > > > > > > > > > > > > > > > > > Watching packet captures, I can see some messages are > > > > > > > > > > being > > > > > > > > > > spoofed > > > > > > > > > > and sent on to the SIEM, but some are not. At first > > > > > > > > > > glance, > > > > > > > > > > it > > > > > > > > > > appears that all regular syslog messages that are > > > > > > > > > > generated > > > > > > > > > > on > > > > > > > > > > the > > > > > > > > > > client are being spoofed and sent on to the SIEM, but > > > > > > > > > > most, > > > > > > > > > > if > > > > > > > > > > not > > > > > > > > > > all messages generated via imfile on the client are > > > > > > > > > > not > > > > > > > > > > being > > > > > > > > > > spoofed and sent on to the SIEM at all. > > > > > > > > > > > > > > > > > > > > I've tried the standard > > > > > > > > > > *.* :omudspoof: > > > > > > > > > > > > > > > > > > > > as well as > > > > > > > > > > $template spooftemplate,"%fromhost-ip% %rawmsg%" > > > > > > > > > > *.* :omudpspoof:;spooftemplate > > > > > > > > > > > > > > > > > > > > and > > > > > > > > > > $template spooftemplate,"%rawmsg%" > > > > > > > > > > *.* :omudpspoof:;spooftemplate > > > > > > > > > > > > > > > > > > > > All with the same effect. Am I missing something > > > > > > > > > > here? > > > > > > > > > > Is > > > > > > > > > > anyone > > > > > > > > > > else doing similar, or seen similar behavior? > > > > > > > > > > > > > > > > > > > > For the record, I'm running a patched version of > > > > > > > > > > 5.8.11. > > > > > > > > > > The > > > > > > > > > > patch, > > > > > > > > > > now that I'm reading it again, was to protect against > > > > > > > > > > calling > > > > > > > > > > more > > > > > > > > > > than one instance of libnet code in the omudpspoof > > > > > > > > > > module. > > > > > > > > > > > > > > > > > > the udpspoof module grabs the first field from the > > > > > > > > > template > > > > > > > > > and > > > > > > > > > uses > > > > > > > > > that as the > > > > > > > > > IP address to spoof from. What is the fromhost-ip when > > > > > > > > > you > > > > > > > > > get > > > > > > > > > the > > > > > > > > > files from > > > > > > > > > imfile? > > > > > > > > > > > > > > > > > > > > > > > > Ahh, you may be onto something there.. the messages > > > > > > > > coming > > > > > > > > into my > > > > > > > > syslog server have just the hostname in it, not FQDN, and > > > > > > > > not > > > > > > > > IP > > > > > > > > address. The few that I see leaving for the SIEM have > > > > > > > > the > > > > > > > > IP > > > > > > > > address in that field of the message. Now to figure out > > > > > > > > why > > > > > > > > they're different. > > > > > > > > > > > > > > Hrm. Upon inspection, all logs are coming in with just > > > > > > > the > > > > > > > hostname, > > > > > > > so that's a dead end. All normal syslogs (like cron.info, > > > > > > > kernel.notice, daemon.debug, etc.) I can see are being > > > > > > > spoofed > > > > > > > and > > > > > > > sent > > > > > > > on to the SIEM, but the logs generated by imfile (all > > > > > > > configured as > > > > > > > news.info) from client machines get logged on the syslog > > > > > > > server, > > > > > > > but > > > > > > > aren't being spoofed and sent on to the SIEM. > > > > > > > > > > > > -- > > > Rick Brown > > > Office of Information Technology > > > Georgia Institute of Technology > ___________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > > POST if you DON'T LIKE THAT. > > > > -- > Rick Brown > Office of Information Technology > Georgia Institute of Technology > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
