[
https://issues.apache.org/jira/browse/KAFKA-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705974#comment-17705974
]
Chris Egerton edited comment on KAFKA-14746 at 3/28/23 12:42 PM:
-----------------------------------------------------------------
[~mimaison] do you think it's worth considering a KIP to alter the behavior in
this scenario? I know that some connectors may be relying on the infinite-retry
logic for the {{taskConfigs}} method but in my experience a lot of the time
it's unintentional and there's a false expectation that throwing from that
method causes the connector to fail.
I also experimented with standalone mode to see if the same issue was present,
and found that although we don't do retries, that mode has its own issues in
this scenario (filed KAFKA-14858 to track).
Given the discrepancy between the two, I wonder if the intent of the retry
logic in distributed mode was more to handle failures to send task configs to
the leader of the cluster (and possibly write them to the config topic, if the
worker was the leader) than handle failures to generate that set of task
configs to begin with.
was (Author: chrisegerton):
[~mimaison] do you think it's worth considering a KIP to alter the behavior in
this scenario? I know that some connectors may be relying on the infinite-retry
logic for the {{taskConfigs}} method but in my experience a lot of the time
it's unintentional and there's a false expectation that throwing from that
method causes the connector to fail.
I also experimented with standalone mode to see if the same issue was present,
and found that although we don't do retries, that mode has its own issues in
this scenario (filed KAFKA-14858 to track).
> Throwing in Connector.taskConfigs generates a lot of logs
> ---------------------------------------------------------
>
> Key: KAFKA-14746
> URL: https://issues.apache.org/jira/browse/KAFKA-14746
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
> Reporter: Mickael Maison
> Assignee: Spacrocket
> Priority: Major
>
> If a Connector throws in its taskConfigs() method, the runtime ends up
> retrying using DistributedHerder.RECONFIGURE_CONNECTOR_TASKS_BACKOFF_MS which
> is a fixed value (250ms). For each retry, the runtime prints the connector
> configuration and the enriched configuration so this can quickly generate a
> lot of logs.
> There is some value in throwing in taskConfigs() as it allows to fail fast in
> case the connector is given bad credentials. For example this is what some of
> the Debezium connectors do:
> https://github.com/debezium/debezium/blob/main/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerConnector.java#L56-L69
> The way Connectors are expected to work today is to instead always create
> tasks and let each task fail in case the configuration is wrong. We should
> document that and make it clear in the javadoc that throwing in taskConfigs
> is not recommended.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
