suez1224 commented on code in PR #155:
URL:
https://github.com/apache/flink-connector-kafka/pull/155#discussion_r1986246516
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/subscriber/KafkaSubscriberUtils.java:
##########
@@ -57,12 +65,42 @@ static Map<String, TopicDescription> getTopicMetadata(
}
static Map<String, TopicDescription> getTopicMetadata(
- AdminClient adminClient, Set<String> topicNames) {
- try {
- return
adminClient.describeTopics(topicNames).allTopicNames().get();
- } catch (Exception e) {
- throw new RuntimeException(
- String.format("Failed to get metadata for topics %s.",
topicNames), e);
+ AdminClient adminClient, Set<String> topicNames, Properties
properties) {
+ int maxRetries =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_MAX_RETRY,
+ Integer::parseInt);
+ long retryDelay =
+ KafkaSourceOptions.getOption(
+ properties,
+
KafkaSourceOptions.TOPIC_METADATA_REQUEST_RETRY_INTERVAL_MS,
+ Long::parseLong);
+ for (int attempt = 0; attempt <= maxRetries; attempt++) {
+ try {
+ return
adminClient.describeTopics(topicNames).allTopicNames().get();
+ } catch (Exception e) {
+ if (attempt == maxRetries) {
+ throw new RuntimeException(
+ String.format("Failed to get metadata for topics
%s.", topicNames), e);
+ } else {
+ LOG.warn(
+ "Attempt {} to get metadata for topics {} failed.
Retrying in {} ms...",
+ attempt,
+ topicNames,
+ retryDelay);
+ try {
+ TimeUnit.MILLISECONDS.sleep(retryDelay);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt(); // Restore
interrupted state
+ LOG.error("Thread was interrupted during metadata
fetch retry delay.", ie);
+ }
+ }
+ }
Review Comment:
thanks for the suggestion, @AHeise . However, the default value for the
`retries` config is already set to MAX_INT (see
[code](https://github.com/apache/kafka/blob/3.4/clients/src/main/java/org/apache/kafka/clients/admin/AdminClientConfig.java#L179-L184)
and [confluent doc
here](https://docs.confluent.io/platform/current/installation/configuration/admin-configs.html#retries).
And I believe Flink does not overwrite the config value. w/o my PR, the flink
job will fail as soon as the metadata request fails. So I don't think this
config control the behavior for failed metadata requests from AdminClient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
