yashmayya opened a new pull request, #13530: URL: https://github.com/apache/kafka/pull/13530
From [KAFKA-14858](https://issues.apache.org/jira/browse/KAFKA-14858): > In distributed mode, if a connector throws an exception from its taskConfigs method (invoked by the herder, through the Worker class, [here](https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1960)), we wait for an exponential backoff period (see [KAFKA-14732](https://issues.apache.org/jira/browse/KAFKA-14732)) and then [retry the operation](https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1907-L1911). > > However, in standalone mode, not only do we not retry the operation, we do not even log the exception. In addition, when REST calls are made that require generating new task configs for a connector (which include creating and reconfiguring a connector), if the connector's taskConfigs method throws an exception, those requests will time out since the [callback](https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/standalone/StandaloneHerder.java#L183) we use to respond to those requests never gets invoked. > > At a bare minimum, we should: > > - Log any exceptions thrown from the taskConfigs method at ERROR level > - Invoke any callbacks passed in to the relevant StandaloneHerder methods with any exceptions thrown by the taskConfigs method > > We might also consider introducing the same kind of exponential backoff retry logic used by distributed mode, but this can be addressed separately since it would be a much larger change in behavior and may break existing user's deployments. - This patch handles exceptions thrown from `Connector::taskConfigs` by either logging the exception or completing callbacks exceptionally as appropriate (in the `StandaloneHerder`). - As discussed in [KAFKA-14746](https://issues.apache.org/jira/browse/KAFKA-14746), the infinite retry mechanism in distributed mode likely never intended to cover exceptions thrown from `Connector::taskConfigs` (which is something we might want to remove in the future) and this patch therefore doesn't introduce a similar retry mechanism here. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
