The omkafka action worker doesn't perform deliveries. Its responsibility is limited to enqueue and ack-processing. Actual deliveries are done by broker-bound threads and although the design delivers sufficient throughput with small kafka clusters, it chokes on lock contention as broker count goes up.
I created it with this patch (https://github.com/edenhill/librdkafka/pull/679) to fix this problem (IIRC we had a successful 300 broker benchmark aggregating to 6M events per second), this allows controlling #threads for deliveries independent of #brokers reducing severity of contention (have a look at screenshots in the PR). The maintainer of librdkafka had a different design in mind, I haven't followed up on the change after fixing it for myself. On Wed, Jan 3, 2018 at 10:25 PM, Andrew Griffin via rsyslog <[email protected]> wrote: > I use omkafka heavily - the confParam setting is what you’re looking for, > basically all of the standard Kafka producer / consumer related parameters > are available there. (see > https://kafka.apache.org/documentation/#producerconfigs > <https://kafka.apache.org/documentation/#producerconfigs>). What you set and > how you set it is heavily dependent on your use case - number of producers, > number of consumers, number / size of topics, throughput expectations, etc. > But in general for omkafka (producer) you’ll get the most results tuning > buffer.memory and batch.size. Those will help you adjust the size and > frequency of producing messages. > > To add threads out to Kafka? I’m not positive but I’d assume adding multiple > rulesets may accomplish that goal - say, multiple listening ports, each with > it’s own dedicated ruleset. That would add additional rsyslog queues, and > thus multiple output queues. Though in our setup we have 1 instance of > rsyslog running per Kafka broker, and have never run in to a case where > either rsyslog or Kafka gets bottlenecked > > Andrew > >> On Dec 23, 2017, at 8:56 PM, deoren >> <[email protected]> wrote: >> >> >> >> On 12/22/2017 9:52 AM, Luigi Tagliamonte via rsyslog wrote: >>> Hi there! >>> What are the tunable parameters for this module, like: >>> - an option to increase the number of threads for kafka processing >>> - number of messages to process per req. >>> - etc.. >>> Regards >>> L. >> >> >> Module docs: >> >> * http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.html >> * http://www.rsyslog.com/doc/v8-stable/configuration/modules/imkafka.html >> >> Disclaimer: I know little (if anything) about Apache Kafka, but I found >> the following mentioned in the module docs: >> >> ############################################### >> confParam [parameter] >> Type: Array >> >> Default: none >> >> Permits to specify Kafka options. Rather than offering a myriad of >> config settings to match the Kafka parameters, we provide this setting >> here as a vehicle to set any Kafka parameter. This has the big advantage >> that Kafka parameters that come up in new releases can immediately be used. >> >> Note that we use librdkafka for the Kafka connection, so the parameters >> are actually those that librdkafka supports. As of our understanding, >> this is a superset of the native Kafka parameters. >> ############################################### >> >> >> That bit is the same for both modules. The omkafka module has this in >> addition to the above: >> >> >> ############################################### >> topicConfParam [parameter] >> Type: Array >> >> Default: none >> >> In essence the same as confParam, but for the Kafka topic. >> ############################################### >> >> >> I assume that means that you can use the confParam parameter for the >> rsyslog imkafka/omkafka modules to pass through native Kafka >> parameters/settings. >> >> Hope that helps. If not, I hope that someone with more knowledge of this >> module can chime in and assist. >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. -- Regards, Janmejay http://codehunk.wordpress.com _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
