On Wed, 23 Nov 2016, [email protected] wrote:
Could any of you, aware of rsyslog internals, provide a design
diagram/architecture view of how the components are working?
Perhaps you could use something like:
https://docs.google.com/presentation/d/1UJSO0t2J6mrIyBNSkCUvrR5Q4RWbLcOof2WgB2_qA3g/edit#slide=id.g13bdc00843_0_3
I think, apart from documentation, it could help a lot understanding how to
improve rsyslog.
In fact, I'm missing a few complete examples (from file to file using queues,
mmnorm, json...) to understand the workflow. Is there anyone anywhere?
I may try to modify your documentation shortly, but I think it's actually much
simpler than you are thinking or Rainer's documents make it seem
you have N input modules, each operating independently (each is at least one
thread).
When an input module receives a message, it adds it to the main queue [1]
There is one (sometimes more) worker thread that grabs a message from the queue
[2] and then steps through the config file.
As the worker is going through the config file, it can do the following:
set variables
apply filters
invoke message modification module code (mm*) which set/modify variables
format the message per a template and call an action (action() which invoke
om* modules)
deliver to a different queue (which has it's own workers)
abandon processing of this message (stop)
In addition to the threads that process logs, there is an 'admin' thread that
polices everything else, does garbage collection, etc.
Everything else is encapsulated into modules.
There are several module types:
im* input modules which get logs from ??? (including impstats which gathers data
from rsyslog internals)
pm* parser modules which may be used by input modules to understand the log
format [3]
mm* message modification modules, these can modify the message itself or
create/modify variables. Almost all of them just create/modify variables.
sm* string modules which are C implementations of templates (speed optimization)
om* output modules, which take messages (or a batch of messages) and deliver
them to something
Each module provides code to specify what config options it supports, which
action() parameters it supports/requires, and the code to perform work when
called on each message.
so your file->file processing would be imfile reading a file and adding messages
to the main queue, and a worker process that reads the main queue and writes the
messages to a file using omfile
Does this help?
David Lang
[1] slight simplification, it could add it to a different queue if so
configured. It may add multiple messages to the queue at one time for efficency.
[2] again, a slight simplification, it can grab more messages, see batch
procesing. But each message is then processed individually.
[3] parser modules are supposed to look at the message buffer provided to it by
the input module and populate the standard properties. If they don't understand
the format, they are supposed to report failure and not change anything. There
are a couple that 'cheat' and modify the message buffer to fix known malformed
messages and then claim they fail to let the standard parsers then work on the
now well-formed message. the pmaix* and the first pmcisco* modules did this.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
