Hello Rainer, First of all, thanks for your answer! I've been trying something simple to understand why setting the parameter queue.workerThreadMinimumMessages leaded me to some logs not being parsed by my script.
I created a flooding script to fill the syslog with lots of different messages and, while if I use the following setup I lose no message but their execution may be delayed due to the exec time of the script, confirmMessages="on" queue.type="LinkedList" if I use the following: confirmMessages="on" queue.type="LinkedList" queue.workerThreads="20" queue.workerThreadsMinimumMessages="50" I lose some of them. Tests of parsed logs: Amount of logs | Setup1 | Setup2 150.000 | 150.000 | 147734 1.000 | 1.000 | 128 The parsing script is just a copy of https://github.com/rsyslog/rsyslog/blob/master/plugins/external/skeletons/python/plugin-with-feedback.py with no process but printing the stdin parsed values. and my flooding script is the following: #! /usr/bin/python3 # script to fill syslog with random email messages from sys import argv import logging import logging.handlers import random,string my_logger = logging.getLogger('MyLogger') my_logger.setLevel(logging.DEBUG) handler = logging.handlers.SysLogHandler(address = '/dev/log') logger_fmt = logging.Formatter(fmt='test/smtp[%(process)d]: %(message)s') handler.setFormatter(logger_fmt) my_logger.addHandler(handler) count = argv[1] def randomString(stringLength=16): """Generate a random string of letters and digits """ letters = string.ascii_lowercase return ''.join(random.choice(letters) for i in range(stringLength)) n = 0 while n != int(count): postfixid = randomString(16) premail = randomString(5) my_logger.debug('%s: to=<%[email protected]>, relay=gmail-smtp-in.l.google.com[74.125.133.26]:25, delay=1.8, delays=0.14/0.04/0.94/0.64, dsn=2.0.0, status=sent (250 2.0.0 OK 1578415524 x7si71899wrq.32 - gsmtp)', postfixid, premail) n += 1 Do you know, or can we find out, why there are some logs being missed by the parser? Thank you, Eudald El jue., 9 ene. 2020 a las 13:17, Rainer Gerhards (<[email protected]>) escribió: > an artificial delay does not play well with the design. It spawns new > workers based on how full the queue is. > > Have a look here and read at least the section on "Worker Thread Pools": > > https://www.rsyslog.com/doc/v8-stable/concepts/queues.html > > This guide unfortunately is not yet updated to current syntax. You may > want to have a look here for current (and legacy) syntax: > > https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html > > The param in question is queue.workerThreadMinimumMessages - if you > set it very low (e.g. 10) many workers will be spawned. Usually that > means you get a lot of overhead, thus rsyslog spawns them only when > needed. > > HTH > Rainer > > El jue., 9 ene. 2020 a las 9:20, Eudald Valcàrcel > (<[email protected]>) escribió: > > > > How can I find out the amount of work needed? > > Maybe I'm approaching this wrong, but adding delays of 10 seconds in the > plugin don't trigger more instances when I generate 50 consecutive random > logs, resulting in an overall delay of 500 seconds for the processing of > the last log. > > Thank you! > > Eudald > > > > On Thu, Jan 9, 2020, 08:20 Rainer Gerhards <[email protected]> > wrote: > >> > >> El jue., 9 ene. 2020 a las 1:32, Eudald Valcàrcel via rsyslog > >> (<[email protected]>) escribió: > >> > > >> > hello John, > >> > My current setup has queue.workerThreads set at 20, but whenever I > check > >> > processes or I add some testing delays it shows only one process being > >> > executed. > >> > >> That indicates there is not sufficient work to start any additional > workers. > >> > >> Rainer > >> > > >> > Thanks, > >> > Eudald > >> > > >> > On Thu, Jan 9, 2020, 01:24 John Chivian via rsyslog < > >> > [email protected]> wrote: > >> > > >> > > From the documentation... > >> > > > >> > > "By default, the omprog action will start an instance (process) of > the > >> > > external program *per worker thread.* > >> > > (the maximum number of worker threads can be specified with the > >> > > queue.workerThreads > >> > > < > https://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html> > >> > > > >> > > parameter)." > >> > > > >> > > Give your pipeline a few more workers and see if that helps. > >> > > > >> > > Regards, > >> > > > >> > > > >> > > On 1/8/20 5:39 PM, Eudald Valcàrcel via rsyslog wrote: > >> > > > Hello! > >> > > > I've a kind of slow python plugin being fed from rsyslog. > >> > > > I'm using omprog, but I'd like to be able to launch another > instance of > >> > > the > >> > > > plugin if the current one is in use. > >> > > > The problem is that by using sys.stdin.readline() and feeding it > through > >> > > > omprog with a template ending in "\n" the plugin never gets an > EOF and > >> > > > waits to process the next batch of data. > >> > > > > >> > > > Is there any way to accomplish that? > >> > > > > >> > > > Thank you! > >> > > > Eudald > >> > > > _______________________________________________ > >> > > > rsyslog mailing list > >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > > http://www.rsyslog.com/professional-services/ > >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> > > DON'T LIKE THAT. > >> > > > >> > > > >> > > _______________________________________________ > >> > > rsyslog mailing list > >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > > http://www.rsyslog.com/professional-services/ > >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >> > > DON'T LIKE THAT. > >> > _______________________________________________ > >> > rsyslog mailing list > >> > http://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
