It sounds like what you are saying is that parsers operate in the transport layer. However, according to the following "message parsers do NOT operate at the transport layer, they operate, as their name implies, on messages. So we can not use message parsers to change the underlying framing": http://www.rsyslog.com/doc/master/concepts/messageparser.html
If there are two different types of parsers (byte parsers and message parsers) it's not clear how to configure them. On Thu, Dec 10, 2015 at 9:58 PM, David Lang <[email protected]> wrote: > you are talking about different parsers > > ruleset parsers take the data arriving over the wire, figure out where the > end of the message is, and what part is the timestamp, hostname, etc. This > populates the normal properties in rsyslog. > > you then can use mmnormalize/mmjsonparse to do additional parsing of the > message. > > Until some other parser extracts meaning out of the bytes arriving over > the wire, there is nothing for mmnormalize to work from, you can't even > tell where one message finishes and the next one starts. > > The reason why it's strongly recommended to leave the RFC3164 parser in > the list is that it's designed to extract as much meaning as possible from > complete garbage, there is no message that the rfc3164 parser will say > cannot be handled. So if you don't list the rfc3164 last, the other parsers > will never be applied (the rfc3164 parser will parse the message and that > will be that). All the other parsers are very picky, and will only > successfully parse messages that exactly comply with their specs. > > does this clarify things a bit. > > David Lang > > > On Thu, 10 Dec 2015, Alec Swan wrote: > > That should work. But then I don't understand why it's always recommended >> to add RFC3164 parser at the end ( >> >> http://www.rsyslog.com/doc/master/configuration/ruleset/rsconf1_rulesetparser.html >> ) >> in order to avoid dropping messages? With your approach messages are >> always >> parsed either by mmnormalize or <parse failed stuff>. >> >> On Thu, Dec 10, 2015 at 1:16 PM, David Lang <[email protected]> wrote: >> >> On Thu, 10 Dec 2015, Alec Swan wrote: >>> >>> I've read a several articles about the benefits of parser chains for >>> >>>> rulesets but couldn't find any examples for the new style of >>>> configuration. >>>> >>>> Here is my scenario. I need to parse logs from my-app.log file written >>>> by >>>> an app. Recently, the format of this file changed slightly, e.g. extra >>>> tabs >>>> removed between parts of each log message. I am using mmnormalize to >>>> parse >>>> the old format using old.rb liblognorm rule. I would like to be able to >>>> use >>>> new.rb rule to parse the log file in case old.rb rule fails. >>>> >>>> What do I need to do in the following to accomplish this? >>>> >>>> input(type = "imfile" File = "my-app.logs" Ruleset = "my-app") >>>> >>>> ruleset(name = "my-app") { >>>> >>>> action(type = "mmnormalize" rulebase = "/etc/rsyslog.d/rules/old.rb") >>>> >>>> # here I would like to use new.rb in case the action above failed to >>>> parse log file >>>> >>>> action(type = "omfile" template="es-payload" >>>> file="/tmp/my-app-parsed.log") >>>> } >>>> >>>> >>> what I've resorted to doing is: >>> >>> action(type="mmnormalize" ruleBase="/etc/rsyslog.rb") >>> if $!originalmsg != "" then { >>> <parse failed stuff> >>> } >>> >>> David Lang >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
