Re: [rsyslog] Imkafka/omkafka tuning

[singh.janmejay via rsyslog]

The omkafka action worker doesn't perform deliveries. Its
responsibility is limited to enqueue and ack-processing. Actual
deliveries are done by broker-bound threads and although the design
delivers sufficient throughput with small kafka clusters, it chokes on
lock contention as broker count goes up.

I created it with this patch
(https://github.com/edenhill/librdkafka/pull/679) to fix this problem
(IIRC we had a successful 300 broker benchmark aggregating to 6M
events per second), this allows controlling #threads for deliveries
independent of #brokers reducing severity of contention (have a look
at screenshots in the PR). The maintainer of librdkafka had a
different design in mind, I haven't followed up on the change after
fixing it for myself.

On Wed, Jan 3, 2018 at 10:25 PM, Andrew Griffin via rsyslog
 wrote:
> I use omkafka heavily - the confParam setting is what you’re looking for, 
> basically all of the standard Kafka producer / consumer related parameters 
> are available there.  (see  
> https://kafka.apache.org/documentation/#producerconfigs 
> ).  What you set and 
> how you set it is heavily dependent on your use case - number of producers, 
> number of consumers, number / size of topics, throughput expectations, etc.  
> But in general for omkafka (producer) you’ll get the most results tuning 
> buffer.memory and batch.size.  Those will help you adjust the size and 
> frequency of producing messages.
>
> To add threads out to Kafka?  I’m not positive but I’d assume adding multiple 
> rulesets may accomplish that goal - say, multiple listening ports, each with 
> it’s own dedicated ruleset.  That would add additional rsyslog queues, and 
> thus multiple output queues. Though in our setup we have 1 instance of 
> rsyslog running per Kafka broker, and have never run in to a case where 
> either rsyslog or Kafka gets bottlenecked
>
> Andrew
>
>> On Dec 23, 2017, at 8:56 PM, deoren 
>>  wrote:
>>
>>
>>
>> On 12/22/2017 9:52 AM, Luigi Tagliamonte via rsyslog wrote:
>>> Hi there!
>>> What are the tunable parameters for this module, like:
>>> - an option to increase the number of threads for kafka processing
>>> - number of messages to process per req.
>>> - etc..
>>> Regards
>>> L.
>>
>>
>> Module docs:
>>
>> * http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.html
>> * http://www.rsyslog.com/doc/v8-stable/configuration/modules/imkafka.html
>>
>> Disclaimer: I know little (if anything) about Apache Kafka, but I found
>> the following mentioned in the module docs:
>>
>> ###
>> confParam [parameter]
>> Type: Array
>>
>> Default: none
>>
>> Permits to specify Kafka options. Rather than offering a myriad of
>> config settings to match the Kafka parameters, we provide this setting
>> here as a vehicle to set any Kafka parameter. This has the big advantage
>> that Kafka parameters that come up in new releases can immediately be used.
>>
>> Note that we use librdkafka for the Kafka connection, so the parameters
>> are actually those that librdkafka supports. As of our understanding,
>> this is a superset of the native Kafka parameters.
>> ###
>>
>>
>> That bit is the same for both modules. The omkafka module has this in
>> addition to the above:
>>
>>
>> ###
>> topicConfParam [parameter]
>> Type: Array
>>
>> Default: none
>>
>> In essence the same as confParam, but for the Kafka topic.
>> ###
>>
>>
>> I assume that means that you can use the confParam parameter for the
>> rsyslog imkafka/omkafka modules to pass through native Kafka
>> parameters/settings.
>>
>> Hope that helps. If not, I hope that someone with more knowledge of this
>> module can chime in and assist.
>> ___
>> rsyslog mailing list
>> http://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>> LIKE THAT.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



-- 
Regards,
Janmejay
http://codehunk.wordpress.com
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

Re: [rsyslog] Imkafka/omkafka tuning

[Andrew Griffin via rsyslog]

I use omkafka heavily - the confParam setting is what you’re looking for, 
basically all of the standard Kafka producer / consumer related parameters are 
available there.  (see  https://kafka.apache.org/documentation/#producerconfigs 
).  What you set and 
how you set it is heavily dependent on your use case - number of producers, 
number of consumers, number / size of topics, throughput expectations, etc.  
But in general for omkafka (producer) you’ll get the most results tuning 
buffer.memory and batch.size.  Those will help you adjust the size and 
frequency of producing messages. 

To add threads out to Kafka?  I’m not positive but I’d assume adding multiple 
rulesets may accomplish that goal - say, multiple listening ports, each with 
it’s own dedicated ruleset.  That would add additional rsyslog queues, and thus 
multiple output queues. Though in our setup we have 1 instance of rsyslog 
running per Kafka broker, and have never run in to a case where either rsyslog 
or Kafka gets bottlenecked

Andrew

> On Dec 23, 2017, at 8:56 PM, deoren 
>  wrote:
> 
> 
> 
> On 12/22/2017 9:52 AM, Luigi Tagliamonte via rsyslog wrote:
>> Hi there!
>> What are the tunable parameters for this module, like:
>> - an option to increase the number of threads for kafka processing
>> - number of messages to process per req.
>> - etc..
>> Regards
>> L.
> 
> 
> Module docs:
> 
> * http://www.rsyslog.com/doc/v8-stable/configuration/modules/omkafka.html
> * http://www.rsyslog.com/doc/v8-stable/configuration/modules/imkafka.html
> 
> Disclaimer: I know little (if anything) about Apache Kafka, but I found
> the following mentioned in the module docs:
> 
> ###
> confParam [parameter]
> Type: Array
> 
> Default: none
> 
> Permits to specify Kafka options. Rather than offering a myriad of
> config settings to match the Kafka parameters, we provide this setting
> here as a vehicle to set any Kafka parameter. This has the big advantage
> that Kafka parameters that come up in new releases can immediately be used.
> 
> Note that we use librdkafka for the Kafka connection, so the parameters
> are actually those that librdkafka supports. As of our understanding,
> this is a superset of the native Kafka parameters.
> ###
> 
> 
> That bit is the same for both modules. The omkafka module has this in
> addition to the above:
> 
> 
> ###
> topicConfParam [parameter]
> Type: Array
> 
> Default: none
> 
> In essence the same as confParam, but for the Kafka topic.
> ###
> 
> 
> I assume that means that you can use the confParam parameter for the
> rsyslog imkafka/omkafka modules to pass through native Kafka
> parameters/settings.
> 
> Hope that helps. If not, I hope that someone with more knowledge of this
> module can chime in and assist.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.



smime.p7s
Description: S/MIME cryptographic signature
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.