[
https://issues.apache.org/jira/browse/KAFKA-7749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724110#comment-16724110
]
Manjeet Duhan commented on KAFKA-7749:
--------------------------------------
I was more concerned about sink connectors at current scenario so did not get
the chance to look into producer configuration. My changes specifically applied
to consumers and more specifically to sink connectors.
If i want to change max.poll.records for my connector ,The only way is to
update consumer.max.poll.records at worker level which is applied to all
connector in the group. This requires worker restart as well.
With my changes , we can pass consumer.max.poll.records along with post
configuration at connector start.
1. This is applied to single connector
2. No Worker restart required
My changes will accept all consumer properties which worker accept starting
with consumer.*
Yes user can pass any value starting with consumer.* which will be passed on to
consumer but kafka consumer will not recognize this or might fail. This flow
will remain same as existing.
I have just 2 lines of code change.
> confluent does not provide option to set consumer properties at connector
> level
> -------------------------------------------------------------------------------
>
> Key: KAFKA-7749
> URL: https://issues.apache.org/jira/browse/KAFKA-7749
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Manjeet Duhan
> Priority: Major
>
> _We want to increase consumer.max.poll.record to increase performance but
> this value can only be set in worker properties which is applicable to all
> connectors given cluster._
> __
> _Operative Situation :- We have one project which is communicating with
> Elasticsearch and we set consumer.max.poll.record=500 after multiple
> performance tests which worked fine for an year._
> _Then one more project onboarded in the same cluster which required
> consumer.max.poll.record=5000 based on their performance tests. This
> configuration is moved to production._
> _Admetric started failing as it was taking more than 5 minutes to process
> 5000 polled records and started throwing commitfailed exception which is
> vicious cycle as it will process same data over and over again._
> __
> _We can control above if start consumer using plain java but this control was
> not available at each consumer level in confluent connector._
> _I have overridden kafka code to accept connector properties which will be
> applied to single connector and others will keep on using default properties
> . These changes are already running in production for more than 5 months._
> _Some of the properties which were useful for us._
> max.poll.records
> max.poll.interval.ms
> request.timeout.ms
> key.deserializer
> value.deserializer
> heartbeat.interval.ms
> session.timeout.ms
> auto.offset.reset
> connections.max.idle.ms
> enable.auto.commit
>
> auto.commit.interval.ms
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
