AHeise commented on code in PR #155:
URL:
https://github.com/apache/flink-connector-kafka/pull/155#discussion_r1993033623
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/subscriber/KafkaSubscriberUtils.java:
##########
@@ -57,12 +65,42 @@ static Map<String, TopicDescription> getTopicMetadata(
}
static Map<String, TopicDescription> getTopicMetadata(
- AdminClient adminClient, Set<String> topicNames) {
- try {
- return
adminClient.describeTopics(topicNames).allTopicNames().get();
- } catch (Exception e) {
- throw new RuntimeException(
- String.format("Failed to get metadata for topics %s.",
topicNames), e);
+ AdminClient adminClient, Set<String> topicNames, Properties
properties) {
+ int maxRetries =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_MAX_RETRY,
+ Integer::parseInt);
+ long retryDelay =
+ KafkaSourceOptions.getOption(
+ properties,
+
KafkaSourceOptions.TOPIC_METADATA_REQUEST_RETRY_INTERVAL_MS,
+ Long::parseLong);
+ for (int attempt = 0; attempt <= maxRetries; attempt++) {
+ try {
+ return
adminClient.describeTopics(topicNames).allTopicNames().get();
+ } catch (Exception e) {
+ if (attempt == maxRetries) {
+ throw new RuntimeException(
+ String.format("Failed to get metadata for topics
%s.", topicNames), e);
+ } else {
+ LOG.warn(
+ "Attempt {} to get metadata for topics {} failed.
Retrying in {} ms...",
+ attempt,
+ topicNames,
+ retryDelay);
+ try {
+ TimeUnit.MILLISECONDS.sleep(retryDelay);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt(); // Restore
interrupted state
+ LOG.error("Thread was interrupted during metadata
fetch retry delay.", ie);
+ }
+ }
+ }
Review Comment:
Could you please share the failure? If it's something that the AdminClient
doesn't deem to be retriable than it's obvious that we need your solution.
It actually may be worth to draft a test that fails without your fix and
succeeds with it. We do have some tests that remove the kafka broker (search
for `stopBroker`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
