github wiki seems to be the correct place. could you create a _draft_
project repo for eRk ?
considering rsyslog is a Rocket-fast SYStem for LOG processing
http://www.rsyslog.com
perhaps we could use a development codename until a popular/commercial
name is found
what about: logwalk? (I really don't care, but I feel this ERK idea is
getting traction...)
El 23/11/16 a las 16:20, Rainer Gerhards escribió:
If we enable the GitHub wiki, that would be a better place for such doc
imho. It is always easy to copy something over once it is written... ;-)
Rainer
Sent from phone, thus brief.
Am 23.11.2016 16:03 schrieb "David Lang" <da...@lang.hm>:
On Wed, 23 Nov 2016, mosto...@gmail.com wrote:
Could any of you, aware of rsyslog internals, provide a design
diagram/architecture view of how the components are working?
Perhaps you could use something like: https://docs.google.com/presen
tation/d/1UJSO0t2J6mrIyBNSkCUvrR5Q4RWbLcOof2WgB2_qA3g/edit#
slide=id.g13bdc00843_0_3
I think, apart from documentation, it could help a lot understanding how
to improve rsyslog.
In fact, I'm missing a few complete examples (from file to file using
queues, mmnorm, json...) to understand the workflow. Is there anyone
anywhere?
I may try to modify your documentation shortly, but I think it's actually
much simpler than you are thinking or Rainer's documents make it seem
you have N input modules, each operating independently (each is at least
one thread).
When an input module receives a message, it adds it to the main queue [1]
There is one (sometimes more) worker thread that grabs a message from the
queue [2] and then steps through the config file.
As the worker is going through the config file, it can do the following:
set variables
apply filters
invoke message modification module code (mm*) which set/modify variables
format the message per a template and call an action (action() which
invoke om* modules)
deliver to a different queue (which has it's own workers)
abandon processing of this message (stop)
In addition to the threads that process logs, there is an 'admin' thread
that polices everything else, does garbage collection, etc.
Everything else is encapsulated into modules.
There are several module types:
im* input modules which get logs from ??? (including impstats which
gathers data from rsyslog internals)
pm* parser modules which may be used by input modules to understand the
log format [3]
mm* message modification modules, these can modify the message itself or
create/modify variables. Almost all of them just create/modify variables.
sm* string modules which are C implementations of templates (speed
optimization)
om* output modules, which take messages (or a batch of messages) and
deliver them to something
Each module provides code to specify what config options it supports,
which action() parameters it supports/requires, and the code to perform
work when called on each message.
so your file->file processing would be imfile reading a file and adding
messages to the main queue, and a worker process that reads the main queue
and writes the messages to a file using omfile
Does this help?
David Lang
[1] slight simplification, it could add it to a different queue if so
configured. It may add multiple messages to the queue at one time for
efficency.
[2] again, a slight simplification, it can grab more messages, see batch
procesing. But each message is then processed individually.
[3] parser modules are supposed to look at the message buffer provided to
it by the input module and populate the standard properties. If they don't
understand the format, they are supposed to report failure and not change
anything. There are a couple that 'cheat' and modify the message buffer to
fix known malformed messages and then claim they fail to let the standard
parsers then work on the now well-formed message. the pmaix* and the first
pmcisco* modules did this.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
