If we enable the GitHub wiki, that would be a better place for such doc imho. It is always easy to copy something over once it is written... ;-)
Rainer Sent from phone, thus brief. Am 23.11.2016 16:03 schrieb "David Lang" <[email protected]>: > On Wed, 23 Nov 2016, [email protected] wrote: > > Could any of you, aware of rsyslog internals, provide a design >> diagram/architecture view of how the components are working? >> >> Perhaps you could use something like: https://docs.google.com/presen >> tation/d/1UJSO0t2J6mrIyBNSkCUvrR5Q4RWbLcOof2WgB2_qA3g/edit# >> slide=id.g13bdc00843_0_3 >> >> I think, apart from documentation, it could help a lot understanding how >> to improve rsyslog. >> In fact, I'm missing a few complete examples (from file to file using >> queues, mmnorm, json...) to understand the workflow. Is there anyone >> anywhere? >> > > I may try to modify your documentation shortly, but I think it's actually > much simpler than you are thinking or Rainer's documents make it seem > > you have N input modules, each operating independently (each is at least > one thread). > > When an input module receives a message, it adds it to the main queue [1] > > There is one (sometimes more) worker thread that grabs a message from the > queue [2] and then steps through the config file. > > As the worker is going through the config file, it can do the following: > > set variables > apply filters > invoke message modification module code (mm*) which set/modify variables > format the message per a template and call an action (action() which > invoke om* modules) > deliver to a different queue (which has it's own workers) > abandon processing of this message (stop) > > > In addition to the threads that process logs, there is an 'admin' thread > that polices everything else, does garbage collection, etc. > > > Everything else is encapsulated into modules. > > There are several module types: > > im* input modules which get logs from ??? (including impstats which > gathers data from rsyslog internals) > > pm* parser modules which may be used by input modules to understand the > log format [3] > > mm* message modification modules, these can modify the message itself or > create/modify variables. Almost all of them just create/modify variables. > > sm* string modules which are C implementations of templates (speed > optimization) > > om* output modules, which take messages (or a batch of messages) and > deliver them to something > > Each module provides code to specify what config options it supports, > which action() parameters it supports/requires, and the code to perform > work when called on each message. > > > so your file->file processing would be imfile reading a file and adding > messages to the main queue, and a worker process that reads the main queue > and writes the messages to a file using omfile > > Does this help? > > David Lang > > > [1] slight simplification, it could add it to a different queue if so > configured. It may add multiple messages to the queue at one time for > efficency. > > [2] again, a slight simplification, it can grab more messages, see batch > procesing. But each message is then processed individually. > > [3] parser modules are supposed to look at the message buffer provided to > it by the input module and populate the standard properties. If they don't > understand the format, they are supposed to report failure and not change > anything. There are a couple that 'cheat' and modify the message buffer to > fix known malformed messages and then claim they fail to let the standard > parsers then work on the now well-formed message. the pmaix* and the first > pmcisco* modules did this. > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
