On Wed, 23 Nov 2016, [email protected] wrote:
there are three stages in decoding a message
1. the framing of a message (i.e. UDP a packet is a message, TCP newline
indicates the end of a message)
2. the parsing of the message. The syslog format is a header followed by
arbitrary text. This step is decoding the header
note: many log installations don't do anything more than this.
3. parsing the free-form text message, i.e. 'normalizing' the logs
I've made the suggestion in the past that we create a pmnormalize that uses
liblognorm rules against the raw message and can populate the standard
properties, but it hasn't been a high priority (the overhead of parsing the
data with an existing parser and then dealing with it with mmnormalize just
isn't very high)
I'm somehow with you on this. I didn't like my messages being "parsed" by
default RFCs (hence, splitting json)
do you have an example of what was sent over the wire and how it was mishandled?
It may be that we need to tweak the default parsers.
I understand rsyslog was born to handle logs (and their lack of standards),
but I wouldn't forget it's also message-shipper, hence it could make sense to
convert pm to mm and to define the pipeline as I exposed before.
no need to convert, just add the ability to use the slower, but more flexible
mmnormalize.
remember that even when the app developers get fixated on the latest and
greatest logging tool that uses a new over-the-wire protocol that works great in
$popular-language, the vast majority of logs are actually produced by mundane
things like routers, switches, firewalls, appliances where you can't change how
they send logs, so having super-fast hard-coded C to parse them as the default
makes a huge amount of sense.
For those other sources, it is usually trivially easy to get them to output a
syslog header in front of their data, and that would make things work very well
as-is.
If you want to deal with a logsource that can't comply with the very simple
syslog protocol, you need a pm module. This is mostly for things like Cisco
adding a field to the messae, AIX adding "message forwarded from", etc (we
actually could use one that deals with syslog-ng inserting the severity in
the header)
again, IMHO, using pm and mm doesn't make much sense. They could be, both,
sequential actions.
pm and mm don't work on the same data.
pm deals with the raw message that arrives on the wire
mm works with all the variables that exist at the time the mm module is called
mm modules can look at the data s it arrived over the wire, and they can also
look at variables that have been created later.
I would like to see a pmmm module, or at least a pmnormalize module created, but
I wouldn't want to try and replace the existing parsers with them.
to do the imhiredis module, the biggest issue is someone with a really good
understanding of how redis works and the over-the-wire protocol
Bob!
I have played a bit with Redis, and doesn't seem very complex (rsyslog
neither...and look!!!)
90% of it is just getting good documenation of the over-the-wire protocol
the rest of it comes into play with things like the current omkafka problem
where the module is telling rsyslog that the message delivery succeeded when it
has really failed. That's where the deeper knowledge of the other system is
needed.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
