yashmayya opened a new pull request, #13530:
URL: https://github.com/apache/kafka/pull/13530

   From [KAFKA-14858](https://issues.apache.org/jira/browse/KAFKA-14858):
   
   > In distributed mode, if a connector throws an exception from its 
taskConfigs method (invoked by the herder, through the Worker class, 
[here](https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1960)),
 we wait for an exponential backoff period (see 
[KAFKA-14732](https://issues.apache.org/jira/browse/KAFKA-14732)) and then 
[retry the 
operation](https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1907-L1911).
   >
   > However, in standalone mode, not only do we not retry the operation, we do 
not even log the exception. In addition, when REST calls are made that require 
generating new task configs for a connector (which include creating and 
reconfiguring a connector), if the connector's taskConfigs method throws an 
exception, those requests will time out since the 
[callback](https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/standalone/StandaloneHerder.java#L183)
 we use to respond to those requests never gets invoked.
   >
   > At a bare minimum, we should:
   > 
   > - Log any exceptions thrown from the taskConfigs method at ERROR level
   > - Invoke any callbacks passed in to the relevant StandaloneHerder methods 
with any exceptions thrown by the taskConfigs method
   >
   > We might also consider introducing the same kind of exponential backoff 
retry logic used by distributed mode, but this can be addressed separately 
since it would be a much larger change in behavior and may break existing 
user's deployments.
   
   - This patch handles exceptions thrown from `Connector::taskConfigs` by 
either logging the exception or completing callbacks exceptionally as 
appropriate (in the `StandaloneHerder`).
   - As discussed in 
[KAFKA-14746](https://issues.apache.org/jira/browse/KAFKA-14746), the infinite 
retry mechanism in distributed mode likely never intended to cover exceptions 
thrown from `Connector::taskConfigs` (which is something we might want to 
remove in the future) and this patch therefore doesn't introduce a similar 
retry mechanism here.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to