Although I had all these clear, reading it helped me to better understand.
Thank you.
A few questions appear:
* You talk about threads...may we considered a distributed application
architecture/other process reading that queue? That could be done
with a redis/kafka reliable delivery, ie: imhiredis/omhiredis +
http://redis.io/commands/rpoplpush
why add the overhead of transporting the data between machines if you don't need
to? Rsyslog is FAST, it routinely does things on one machine that other systems
need many farms of machines to do.
adding more machines and network connectivity will add complexity and delays,
reducing performance and reliability
* Is the main queue reliable/disk assisted? Of course UDP messages can
be lost if the application crash, but shouldn't with TCP
that depends how you configure it. All queues can be configured to operate in
multiple modes
1. "Direct" (i.e. no queue)
2. "FixedArray" (an array of messages in memory)
3. "LinkedList" (a dynanmically sized array in memory, slightly slower than
FixedArray)
4. "Disk" (no memory queue, everything on disk, very slow)
and either #2 or #3 can be extended with a disk assisted spillover (and the
ability to save the queue contents to disk when you exit)
* By default, messages are parsed to match RFCs and then sent to _mm_.
is there any way to disable parsers? IMHO pipelines could be
simplified if parsers/mm/actions...are considered equals:
# conditionals can be used here
pipeline {
# and here
input() //multiple inputs can be used, order doesn't matter
processor(type="...") //multiple processors are handled as
a sequence (a parser, a mm...)
processor(type="...") //this processor is run after the
previous
output() //multiple outputs can be used, order doesn't matter
}
there are three stages in decoding a message
1. the framing of a message (i.e. UDP a packet is a message, TCP newline
indicates the end of a message)
2. the parsing of the message. The syslog format is a header followed by
arbitrary text. This step is decoding the header
note: many log installations don't do anything more than this.
3. parsing the free-form text message, i.e. 'normalizing' the logs
I've made the suggestion in the past that we create a pmnormalize that uses
liblognorm rules against the raw message and can populate the standard
properties, but it hasn't been a high priority (the overhead of parsing the data
with an existing parser and then dealing with it with mmnormalize just isn't
very high)
Your explanation is so simple that I'm wondering why source seems so obscure
to me...
The devil is in the details, and when you are working to make things very fast,
it gets messy.
I deliberatly simplified a bunch of things to give you the architecture.
You need to first think about what is it that you are trying to understand when
looking at the code.
The core is ugly and messy, it does a lot of stuff.
This is where the queue code lives
This is what has to deal with batches
This is where all the variable manipulation happens
This is where all the filter functions, rainerscript language, etc is defined
This is where the template functionality is defined.
But very few people need to make changes here. The vast majority of the changes
that are interesting to people are in the modules
If you want to send data out to something, you need an om module
If you want to get data from something, you need an im module
these satisfy 90% of the changes that people need
If you want to deal with a logsource that can't comply with the very simple
syslog protocol, you need a pm module. This is mostly for things like Cisco
adding a field to the messae, AIX adding "message forwarded from", etc (we
actually could use one that deals with syslog-ng inserting the severity in the
header)
If you need to do complex manipulation of the log, you need a mm module.
liblognorm handles most of this, but occasionally we see real needs for other
things. We also have mmexternal that lets you send json to stdin on an external
program and watches for json to be returned from stdout (this lets you do things
in any language)
to do the imhiredis module, the biggest issue is someone with a really good
understanding of how redis works and the over-the-wire protocol
to create an omriemann module, what's needed is documenation on what the
over-the-wire protocol is.
David Lang
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
