> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of Rainer Gerhards > Sent: Monday, December 17, 2012 6:27 PM > To: rsyslog-users > Subject: Re: [rsyslog] imfile and omudpspoof > > > > Just a quick follow-up: I currently work with Rick on this issue. > It > > looks like there are actually two issues in one: > > > > > > 1) in v5, it can happen that an action in front of an included > get's > > reordered, and this seems to have happened here. This is actually the > > second time in many years that I see that. I won't fix that in v5, as > > it would probably lead to something like the v7 engine ;) Rick has > > moved to v7. > > > > > > 2) omudpspoof discards messages that are larger than the network- > > supported MTU. I am working to fragment them. Note that I know that > > some receivers have problems with fragmented UDP, but hey, what else > > shall we do? Messages over 64K can never be transmitted in a single > UDP > > packet, and as such 7.2.5 will now truncate them to 64K. I will > > probably change v5 so that >1472 byte messages get truncated, but the > > message sent instead of being discarded. > > > > > > Will keep you posted when there is notable news (but maybe you just > > see a new release announcement if everything works out and no new > > problems pop up). > > > > I would suggest making the default spoof template do the truncation, > so > > that > > someone who's system can handle larger messages can set it longer. > Side-note: v7 get's a "mtu" action parameter, so that folks with larger mtu's can avoid fragmentation (right now integrating that one ;)). Just the default is kept at 1500.
Rainer > That's rather complicated, because the template system can limit fields > but not the overall string. > > If the consensus is that this is a bad change, I'll revert the change > and keep the old behavior. I just thought I solve this hard-to-find- > and-debug issue, but I don't want to invest any real time in v5 (I > don't even know if I ever will do a new release...). > > What do you say in the light of this? > > Rainer > > > > David Lang > > > > > Rainer > > > > > >> -----Original Message----- > > >> From: [email protected] [mailto:rsyslog- > > >> [email protected]] On Behalf Of Rick Brown > > >> Sent: Friday, December 14, 2012 3:24 PM > > >> To: rsyslog-users > > >> Subject: Re: [rsyslog] imfile and omudpspoof > > >> > > >> ----- Original Message ----- > > >>> From: "Rainer Gerhards" <[email protected]> > > >>> To: "rsyslog-users" <[email protected]> > > >>> Sent: Friday, December 14, 2012 9:06:10 AM > > >>> Subject: Re: [rsyslog] imfile and omudpspoof > > >>> > > >>>> -----Original Message----- > > >>>> From: [email protected] [mailto:rsyslog- > > >>>> [email protected]] On Behalf Of Rick Brown > > >>>> Sent: Friday, December 14, 2012 3:04 PM > > >>>> To: rsyslog-users > > >>>> Subject: Re: [rsyslog] imfile and omudpspoof > > >>>> > > >>>> ----- Original Message ----- > > >>>>> From: "Rainer Gerhards" <[email protected]> > > >>>>> To: "rsyslog-users" <[email protected]> > > >>>>> Sent: Friday, December 14, 2012 2:40:05 AM > > >>>>> Subject: Re: [rsyslog] imfile and omudpspoof > > >>>>> > > >>>>> > > >>>>> > > >>>>>> -----Original Message----- > > >>>>>> From: Rick Brown [mailto:[email protected]] > > >>>>>> Sent: Thursday, December 13, 2012 7:06 PM > > >>>>>> To: Rainer Gerhards > > >>>>>> Subject: Re: [rsyslog] imfile and omudpspoof > > >>>>>> > > >>>>>> You're right, it's rather large. I gzipp'd the output and > > >>>>>> posted > > >>>>>> it > > >>>>>> at xxxx > > >>>>> > > >>>>> Mhhh... That log does not include anything from imfile. Did you > > >>>>> accidently capture the wrong system? > > >>>>> > > >>>> > > >>>> I captured from the syslog server in the middle. Let me try to > > be > > >>>> clearer in stating the problem. > > >>>> > > >>>> Client A scrapes log files with imfile and sends them to my > syslog > > >>>> server. The syslog server is configured to forward the messages > > to > > >>>> a > > >>>> SIEM using imudpspoof. The syslog server successfully writes > the > > >>>> logs > > >>>> to disk, however not all messages are being spoofed and sent on. > > >>>> The > > >>>> person who runs the SIEM noted the difference and I then > observed > > >>>> tcpdumps on the syslog server. Watching in two terminals: > > >>>> > > >>>> tcpdump host A and host syslog > > >>>> > > >>>> and: > > >>>> tcpdump host syslog and host SIEM > > >>>> > > >>>> I see many more messages in the top terminal than the bottom. > On > > >>>> first > > >>>> glance, it looked like none of the messages generated by imfile > on > > >>>> client A were making it to the SIEM, but now that I've stared at > > it > > >>>> long enough, I do occassionally see a couple go by. If I > > restarted > > >>>> rsyslog while watching the tcpdumps, I see a burst of the imfile > > >>>> messages get spoofed, then they stop for a while, then a few > will > > >>>> trickle by. > > >>> > > >>> OK - I misunderstood the initial description. So let's continue > > with > > >>> that debug log file. Can you tell me which logs lines are > > >>> imfile-originated (and not delivered) and which one are not? > > >>> > > >>> Rainer > > >> > > >> Ahh, good. All of the imfile messages are set to facility news > and > > we > > >> no longer actually run news servers, so it's a safe bet that any > > news > > >> messages is generated from imfile. I've been particularly > > >> concentrating on a host cas1, who's ip ends in .165.151, as it > > >> generates quite a bit of application logs that I scrape with > imfile. > > >> And the spoof target address ends in .165.95. > > >> > > >>>> > > >>>> > > >>>>> From what I have seen, it may also happen that the debug log > info > > >>>>> is > > >>>>> not sufficient. Just treat this as advance warning ;) > > >>>>> > > >>>>> We can probably get around a limitation by creating an > additional > > >>>>> log > > >>>>> file, so that I can see all message properties. Please add > > >>>>> > > >>>>> *.* /var/log/rsyslog-props.log;RSYSLOG_DebugFormat > > >>>>> > > >>>>> To your config (the file name is obviously irrelevant, but the > > >>>>> template is important!). > > >>>>> > > >>>>> Make sure that the debug log and the property file are from the > > >>>>> same > > >>>>> run. > > >>>>> > > >>>>> Rainer > > >>>> > > >>>> > > >>>> Can do. I'll go look into this now. > > >>>> > > >>>> > > >>>>>> I see that some of the imfile generated messages are making it > > >>>>>> through > > >>>>>> to the SIEM from time to time, but the vast majority are being > > >>>>>> dropped. > > >>>>>> I have to wonder if I'm overflowing a buffer in the output > > >>>>>> modules. > > >>>>>> > > >>>>>> Thanks for taking a look! > > >>>>>> > > >>>>>> ----- Original Message ----- > > >>>>>>> From: "Rainer Gerhards" <[email protected]> > > >>>>>>> To: "rsyslog-users" <[email protected]> > > >>>>>>> Sent: Thursday, December 13, 2012 10:05:02 AM > > >>>>>>> Subject: Re: [rsyslog] imfile and omudpspoof > > >>>>>>> > > >>>>>>> Can you post a (sufficiently large or complete) debug log > > >>>>>>> that > > >>>>>>> exposes the problem? The list will probably reject it, so it > > >>>>>>> is > > >>>>>>> best > > >>>>>>> to put it on something like pastbin. > > >>>>>>> > > >>>>>>> Rainer > > >>>>>>> > > >>>>>>>> -----Original Message----- > > >>>>>>>> From: [email protected] [mailto:rsyslog- > > >>>>>>>> [email protected]] On Behalf Of Rick Brown > > >>>>>>>> Sent: Thursday, December 13, 2012 4:01 PM > > >>>>>>>> To: rsyslog-users > > >>>>>>>> Subject: Re: [rsyslog] imfile and omudpspoof > > >>>>>>>> > > >>>>>>>>>> On Tue, 11 Dec 2012, Rick Brown wrote: > > >>>>>>>>>> > > >>>>>>>>>>> I use imfile to gather application logs such as > > >>>>>>>>>>> apache, > > >>>>>>>>>>> tomcat, > > >>>>>>>>>>> php, etc. and send those on to my syslog server along > > >>>>>>>>>>> with > > >>>>>>>>>>> the > > >>>>>>>>>>> client machines normal syslog traffic. My syslog > > >>>>>>>>>>> server > > >>>>>>>>>>> then > > >>>>>>>>>>> dutifully writes all the messages locally and > > >>>>>>>>>>> additionally > > >>>>>>>>>>> forwards the messages on to a SIEM product via > > >>>>>>>>>>> omudpspoof. > > >>>>>>>>>>> > > >>>>>>>>>>> Watching packet captures, I can see some messages are > > >>>>>>>>>>> being > > >>>>>>>>>>> spoofed > > >>>>>>>>>>> and sent on to the SIEM, but some are not. At first > > >>>>>>>>>>> glance, > > >>>>>>>>>>> it > > >>>>>>>>>>> appears that all regular syslog messages that are > > >>>>>>>>>>> generated > > >>>>>>>>>>> on > > >>>>>>>>>>> the > > >>>>>>>>>>> client are being spoofed and sent on to the SIEM, but > > >>>>>>>>>>> most, > > >>>>>>>>>>> if > > >>>>>>>>>>> not > > >>>>>>>>>>> all messages generated via imfile on the client are > > >>>>>>>>>>> not > > >>>>>>>>>>> being > > >>>>>>>>>>> spoofed and sent on to the SIEM at all. > > >>>>>>>>>>> > > >>>>>>>>>>> I've tried the standard > > >>>>>>>>>>> *.* :omudspoof: > > >>>>>>>>>>> > > >>>>>>>>>>> as well as > > >>>>>>>>>>> $template spooftemplate,"%fromhost-ip% %rawmsg%" > > >>>>>>>>>>> *.* :omudpspoof:;spooftemplate > > >>>>>>>>>>> > > >>>>>>>>>>> and > > >>>>>>>>>>> $template spooftemplate,"%rawmsg%" > > >>>>>>>>>>> *.* :omudpspoof:;spooftemplate > > >>>>>>>>>>> > > >>>>>>>>>>> All with the same effect. Am I missing something > > >>>>>>>>>>> here? > > >>>>>>>>>>> Is > > >>>>>>>>>>> anyone > > >>>>>>>>>>> else doing similar, or seen similar behavior? > > >>>>>>>>>>> > > >>>>>>>>>>> For the record, I'm running a patched version of > > >>>>>>>>>>> 5.8.11. > > >>>>>>>>>>> The > > >>>>>>>>>>> patch, > > >>>>>>>>>>> now that I'm reading it again, was to protect against > > >>>>>>>>>>> calling > > >>>>>>>>>>> more > > >>>>>>>>>>> than one instance of libnet code in the omudpspoof > > >>>>>>>>>>> module. > > >>>>>>>>>> > > >>>>>>>>>> the udpspoof module grabs the first field from the > > >>>>>>>>>> template > > >>>>>>>>>> and > > >>>>>>>>>> uses > > >>>>>>>>>> that as the > > >>>>>>>>>> IP address to spoof from. What is the fromhost-ip when > > >>>>>>>>>> you > > >>>>>>>>>> get > > >>>>>>>>>> the > > >>>>>>>>>> files from > > >>>>>>>>>> imfile? > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> Ahh, you may be onto something there.. the messages > > >>>>>>>>> coming > > >>>>>>>>> into my > > >>>>>>>>> syslog server have just the hostname in it, not FQDN, and > > >>>>>>>>> not > > >>>>>>>>> IP > > >>>>>>>>> address. The few that I see leaving for the SIEM have > > >>>>>>>>> the > > >>>>>>>>> IP > > >>>>>>>>> address in that field of the message. Now to figure out > > >>>>>>>>> why > > >>>>>>>>> they're different. > > >>>>>>>> > > >>>>>>>> Hrm. Upon inspection, all logs are coming in with just > > >>>>>>>> the > > >>>>>>>> hostname, > > >>>>>>>> so that's a dead end. All normal syslogs (like cron.info, > > >>>>>>>> kernel.notice, daemon.debug, etc.) I can see are being > > >>>>>>>> spoofed > > >>>>>>>> and > > >>>>>>>> sent > > >>>>>>>> on to the SIEM, but the logs generated by imfile (all > > >>>>>>>> configured as > > >>>>>>>> news.info) from client machines get logged on the syslog > > >>>>>>>> server, > > >>>>>>>> but > > >>>>>>>> aren't being spoofed and sent on to the SIEM. > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Rick Brown > > >>>> Office of Information Technology > > >>>> Georgia Institute of Technology > > >> ___________________ > > >>> rsyslog mailing list > > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >>> http://www.rsyslog.com/professional-services/ > > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > > >>> POST if you DON'T LIKE THAT. > > >>> > > >> > > >> -- > > >> Rick Brown > > >> Office of Information Technology > > >> Georgia Institute of Technology > > >> > > >> _______________________________________________ > > >> rsyslog mailing list > > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > > >> http://www.rsyslog.com/professional-services/ > > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > > POST > > >> if you DON'T LIKE THAT. > > > _______________________________________________ > > > rsyslog mailing list > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > > http://www.rsyslog.com/professional-services/ > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > > if you DON'T LIKE THAT. > > > > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > > if you DON'T LIKE THAT. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
