Thanks for the info. When dealing with large log processors, additional flexibility could be useful.
Often, logs need to be separated and translated, or handled differently according to the sender or other details. Getting 200+gb of logs per day always involves special headaches. Could a future enhancement provide more flexibility by allowing script/rule to manage it immediately out of the input module? Often, a minor adjustment results in a valid parseable message. The sender ip, message length, tls certificate etc. should be known when it is ingested. In my case, I had a lookup table to determine the log type based on a json list of 500 or so different sending ips. Although parsed by default chain, the work has to be largely redone to get valid properties. I will see if I can coax one of the message modification modules to do the work, but it feels like a longterm solution could benefit everyone. On Tue, Jul 22, 2025, 13:52 Rainer Gerhards <[email protected]> wrote: > My new AI experiment at https://rsyslog.ai had these comments, which I > agree with: > --- > > No, once a log message has been parsed by rsyslog, it cannot be > re-parsed within the same pipeline. The message parsing step is a > one-time operation that occurs immediately after the input stage. Once > a parser (e.g., pmciscoios or the default rfc5424/rfc3164 parser) has > successfully parsed a message, the message is transformed into > rsyslog's internal structured representation. At that point, the raw > text is no longer available for another parser to process. > > From the documentation: > > "After one parser has been found, the message is considered parsed and > no other parsers are called on that message.". > > This means that when you use call ruleset or other processing steps, > rsyslog will not attempt to run another parser on the message since > parsing is only part of the initial input stage. > > Workarounds > > Parser Chains > If you need multiple parsing attempts, you must define a parser chain > for the ruleset bound to the input. A parser chain tries multiple > parsers in order until one succeeds: > > ruleset(name="ciscoBoth" > parser=["custom.ciscoios.withOrigin", "rsyslog.ciscoios"]) { > ... do processing ... > } > > This ensures that the message is parsed by the first compatible parser > without requiring re-parsing. > > Separate Inputs/Ports > If possible, split incoming log streams by source or port and assign > different parsers/rulesets to each input. > > Use mmnormalize for Post-Processing > If your issue is more about structured parsing rather than syslog > format detection, you can use mmnormalize within the pipeline after > the message is already parsed. > > Unfortunately, there is no built-in way to force re-parsing of an > already parsed message within the same processing flow. > > ---- > > Let me add that re-parsing as an idea came up some time ago. Maybe we > could reconsider it. > > HTH > Rainer > > > El mar, 22 jul 2025 a las 19:38, David Lang via rsyslog > (<[email protected]>) escribió: > > > > Michael Richards wrote: > > > > > Can a log message be re-parsed once it is in the pipeline? > > > > you can use mmnormalize to parse a in the ruleset (you can parse any > variable, > > including rawmsg) > > > > David Lang > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
