> -----Original Message----- > From: [email protected] [mailto:rsyslog- > [email protected]] On Behalf Of [email protected] > Sent: Tuesday, July 13, 2010 12:25 AM > To: rsyslog-users > Subject: Re: [rsyslog] Last message repeated n times problem > > On Mon, 12 Jul 2010, Rainer Gerhards wrote: > > > [snip] > > Well, as you can see: > >> 14:43:15.483698 88:43:e1:41:15:3f > 00:0c:29:a2:86:f1, ethertype > IPv4 > >> (0x0800), length 76: (tos 0x0, ttl 63, id 0, offset 0, flags [DF], > >> proto: UDP (17), length: 62) 10.167.3.18.514 > 10.167.2.65.514: [udp > >> sum ok] SYSLOG, length: 34 > >> Facility daemon (3), Severity error (3) > >> Msg: last message repeated 5 times\012 > >> 0x0000: 3c32 373e 6c61 7374 206d 6573 7361 6765 > >> 0x0010: 2072 6570 6561 7465 6420 3520 7469 6d65 > >> 0x0020: 730a > >> 0x0000: 4500 003e 0000 4000 3f11 210f 0aa7 0312 > >> E..>....@.?.!..... > >> 0x0010: 0aa7 0241 0202 0202 002a df44 3c32 373e > >> ...A.....*.D<27> > >> 0x0020: 6c61 7374 206d 6573 7361 6765 2072 6570 > >> last.message.rep > >> 0x0030: 6561 7465 6420 3520 7469 6d65 730a > >> eated.5.times. > >> > > the message is totally malformed. > > > >> In both cases it seems to me that the IP address of the sender has > been > >> included in the packet (0a a7 03 12) which translates as the IP > >> 10.167.3.18 which is the sender. > >> > >> Is this an rsyslog issue? Is it a known problem? > > > > The sender must be fixed to emit correct format. It is a known > problem with > > some senders (sysklogd namely), but we have not yet provided a work- > around > > for it because that causes unnecessary performance loss on the > rsyslog side. > > However, thinking now about it, a special message parser module could > be used > > to solve that situation. That way, only those that actually have this > problem > > would need to bear the cost of the work-around. Let me think a bit > about > > this... > > if I'm understanding this correctly the filter seems like it would be > pretty simple. > > accept the message as being potentially well formed IFF (if and only > if) > it meets the following criteria. > > 1. has a valid <pri> tag > > 2. characters 4, 7, 16 are ' ' > > 3. characters 10,13 are ':' > > this isn't absolute proof that this is a valid timestamp, but it's > pretty > good and very quick to check in C. >
That's definitely a way to go. Unfortunately, pmrfc3164 already does a lot of guesswork for other cases where things are malformed (just as an example, there are devices that put an additional space in front of the date, invalidating all the offsets you give...). The problem is that in a strict sense, RFC3164 does not demand anything. So there are lots of things that are mutually exclusive. In the current parser, I try to weigh importance of occurrence and performance. So the most prominent cases are probably handled, conflicting less common cases not. "last message repeated n times" is pretty useless, so it has low priority. I have also a "performance filter" on my mind, that means I do only those things that seem reasonable. However, the whole idea of introducing message parsers was to solve exactly that problem. So I think it is natural to write a new message parser rather than try (and probably fail) to fiddle even more with the 3164 parser, which already is overwhelmed with guesswork. For "last message...", I would need to do a full string compare, anything else would be incompatible with the already-existing guesswork in pmrfc3164. Rainer > if not, it can't possibly be a valid message, set the timestamp > (current > time) and hostname (based on the source IP in the packet). > > then check the word staring at position 17 to see if it is made up of > characters that mean it could be a hostname (a-zA-Z0-9.-) > > if not, it can't possibly be a valid hostname, set the hostname (based > on > the source IP in the packet) > > > now, if something passes this test but has other garbage in the > hostname > field, that will take system specific fixups. In this case, after the > tests I describe you would still have the syslog tag of 'last' and the > message 'message repeated 5 times' > > that would need to be fixed by a filter that checked for 'last message > repeated' and put it all in the message (filling in a dummy or blank > syslog tag) > > there are unfortunantly a bunch of such filters that are needed. now > that > they are easy to create we can hopefully get a bunch of them done so > that > only the people who need them bother with them. > > one question on stackable filters. > > how much overhead is there in putting the tests in separate filters > as > opposed to in one filter? > > for example, take 3 tests > > 1. 'message forwarded from hostname' -> hostname > > 2. 'last message repeated' -> move to message body > > 3. 'hostname : %PIXN-NNNN' -> syslogtag= %PIXN-NNNN > > what would the impact be of having each of these as separate filters vs > one filter with configurable tests? (i.e. is there a noticable overhead > in > stacking the filters) > > David Lang > > >> Thanks. > >> Jon. > >> > >> This email is private > > > > No, it isn't -- it was sent to a public mailing list and is probably > archived > > on a myriad of sites by now. Please note that as of the ToS of the > mailing > > list, we do not accept any liability. It would be decent to tell your > mail > > folks to turn off this disclaimer for list-directed mail. Thanks! > > > > Rainer > > > >> and may be confidential and is for the intended > >> recipient only. If misdirected, please notify us by telephone and > >> confirm that it has been deleted from your system and any copies > >> destroyed. If you are not the intended recipient you are strictly > >> prohibited from using, printing, copying, distributing or > disseminating > >> this email or any information contained in it. We use reasonable > >> endeavours to virus scan all emails leaving the Company but no > warranty > >> is given that this email and any attachments are virus free. You > should > >> undertake your own virus checking. The right to monitor email > >> communications through our network is reserved by us. > >> > >> Telindus Limited is a company registered in England and Wales under > >> number 02020395. The registered office is Centurion, Riverside Way, > >> Watchmoor Park, Blackwater Valley Road, Camberley, Surrey, GU15 3YA. > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com
