jihoonson commented on a change in pull request #6383: Prevent failed
KafkaConsumer creation from blocking overlord startup
URL: https://github.com/apache/incubator-druid/pull/6383#discussion_r222189463
##########
File path:
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/supervisor/KafkaSupervisor.java
##########
@@ -416,61 +421,87 @@ public void start()
ioConfig.getStartDelay(),
spec.toString()
);
+
+ initializationExec.shutdownNow();
}
catch (Exception e) {
if (consumer != null) {
consumer.close();
}
log.makeAlert(e, "Exception starting KafkaSupervisor[%s]", dataSource)
.emit();
- throw Throwables.propagate(e);
}
}
}
+ @Override
+ public void start()
+ {
+ synchronized (stateChangeLock) {
+ Preconditions.checkState(!lifecycleStarted, "already started");
+ Preconditions.checkState(!exec.isShutdown(), "already stopped");
+
+ // Try normal initialization first, if that fails then schedule periodic
initialization retries
+ tryInit();
+ if (!started) {
+ initializationExec.scheduleAtFixedRate(
Review comment:
Hmm, I think backoff is more appropriate for transient errors. It starts
with retrying in a very short interval at first, but the retry interval grows
as the failure continues. This makes sense because it can recover quickly with
short retry periods if the error is transient. If the error is permanent, the
retry period would grow in the end and it won't use excessive resources.
Regarding too large backoff time, `RetryUtils.retry()` has the
`MAX_SLEEP_MILLIS` which I think it makes sense to make it configurable.
Regarding blocking call, please check my below comment. What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
