[
https://issues.apache.org/jira/browse/KAFKA-7749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739888#comment-16739888
]
Paul Davidson commented on KAFKA-7749:
--------------------------------------
Hi [~sliebau]. You can probably ignore the comment about task level settings
now. It was actually motivated by the problem described here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-411%3A+Add+option+to+make+Kafka+Connect+task+client+ID+values+unique
and here https://issues.apache.org/jira/browse/KAFKA-5061 . We seem to be
close to resolving that particular issue by simply including the task id in the
default client ID. I can't think of any other specific cases where task-level
settings would be particularly useful.
> confluent does not provide option to set consumer properties at connector
> level
> -------------------------------------------------------------------------------
>
> Key: KAFKA-7749
> URL: https://issues.apache.org/jira/browse/KAFKA-7749
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Manjeet Duhan
> Priority: Major
>
> _We want to increase consumer.max.poll.record to increase performance but
> this value can only be set in worker properties which is applicable to all
> connectors given cluster._
> __
> _Operative Situation :- We have one project which is communicating with
> Elasticsearch and we set consumer.max.poll.record=500 after multiple
> performance tests which worked fine for an year._
> _Then one more project onboarded in the same cluster which required
> consumer.max.poll.record=5000 based on their performance tests. This
> configuration is moved to production._
> _Admetric started failing as it was taking more than 5 minutes to process
> 5000 polled records and started throwing commitfailed exception which is
> vicious cycle as it will process same data over and over again._
> __
> _We can control above if start consumer using plain java but this control was
> not available at each consumer level in confluent connector._
> _I have overridden kafka code to accept connector properties which will be
> applied to single connector and others will keep on using default properties
> . These changes are already running in production for more than 5 months._
> _Some of the properties which were useful for us._
> max.poll.records
> max.poll.interval.ms
> request.timeout.ms
> key.deserializer
> value.deserializer
> heartbeat.interval.ms
> session.timeout.ms
> auto.offset.reset
> connections.max.idle.ms
> enable.auto.commit
>
> auto.commit.interval.ms
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
