an artificial delay does not play well with the design. It spawns new workers based on how full the queue is.
Have a look here and read at least the section on "Worker Thread Pools": https://www.rsyslog.com/doc/v8-stable/concepts/queues.html This guide unfortunately is not yet updated to current syntax. You may want to have a look here for current (and legacy) syntax: https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html The param in question is queue.workerThreadMinimumMessages - if you set it very low (e.g. 10) many workers will be spawned. Usually that means you get a lot of overhead, thus rsyslog spawns them only when needed. HTH Rainer El jue., 9 ene. 2020 a las 9:20, Eudald Valcàrcel (<[email protected]>) escribió: > > How can I find out the amount of work needed? > Maybe I'm approaching this wrong, but adding delays of 10 seconds in the > plugin don't trigger more instances when I generate 50 consecutive random > logs, resulting in an overall delay of 500 seconds for the processing of the > last log. > Thank you! > Eudald > > On Thu, Jan 9, 2020, 08:20 Rainer Gerhards <[email protected]> wrote: >> >> El jue., 9 ene. 2020 a las 1:32, Eudald Valcàrcel via rsyslog >> (<[email protected]>) escribió: >> > >> > hello John, >> > My current setup has queue.workerThreads set at 20, but whenever I check >> > processes or I add some testing delays it shows only one process being >> > executed. >> >> That indicates there is not sufficient work to start any additional workers. >> >> Rainer >> > >> > Thanks, >> > Eudald >> > >> > On Thu, Jan 9, 2020, 01:24 John Chivian via rsyslog < >> > [email protected]> wrote: >> > >> > > From the documentation... >> > > >> > > "By default, the omprog action will start an instance (process) of the >> > > external program *per worker thread.* >> > > (the maximum number of worker threads can be specified with the >> > > queue.workerThreads >> > > <https://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html> >> > > >> > > parameter)." >> > > >> > > Give your pipeline a few more workers and see if that helps. >> > > >> > > Regards, >> > > >> > > >> > > On 1/8/20 5:39 PM, Eudald Valcàrcel via rsyslog wrote: >> > > > Hello! >> > > > I've a kind of slow python plugin being fed from rsyslog. >> > > > I'm using omprog, but I'd like to be able to launch another instance of >> > > the >> > > > plugin if the current one is in use. >> > > > The problem is that by using sys.stdin.readline() and feeding it >> > > > through >> > > > omprog with a template ending in "\n" the plugin never gets an EOF and >> > > > waits to process the next batch of data. >> > > > >> > > > Is there any way to accomplish that? >> > > > >> > > > Thank you! >> > > > Eudald >> > > > _______________________________________________ >> > > > rsyslog mailing list >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > > http://www.rsyslog.com/professional-services/ >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >> > > > myriad >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > > DON'T LIKE THAT. >> > > >> > > >> > > _______________________________________________ >> > > rsyslog mailing list >> > > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > > http://www.rsyslog.com/professional-services/ >> > > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > > DON'T LIKE THAT. >> > _______________________________________________ >> > rsyslog mailing list >> > http://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
