AHeise commented on code in PR #155:
URL:
https://github.com/apache/flink-connector-kafka/pull/155#discussion_r1971207127
##########
flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/subscriber/KafkaSubscriberUtils.java:
##########
@@ -57,12 +65,42 @@ static Map<String, TopicDescription> getTopicMetadata(
}
static Map<String, TopicDescription> getTopicMetadata(
- AdminClient adminClient, Set<String> topicNames) {
- try {
- return
adminClient.describeTopics(topicNames).allTopicNames().get();
- } catch (Exception e) {
- throw new RuntimeException(
- String.format("Failed to get metadata for topics %s.",
topicNames), e);
+ AdminClient adminClient, Set<String> topicNames, Properties
properties) {
+ int maxRetries =
+ KafkaSourceOptions.getOption(
+ properties,
+ KafkaSourceOptions.TOPIC_METADATA_REQUEST_MAX_RETRY,
+ Integer::parseInt);
+ long retryDelay =
+ KafkaSourceOptions.getOption(
+ properties,
+
KafkaSourceOptions.TOPIC_METADATA_REQUEST_RETRY_INTERVAL_MS,
+ Long::parseLong);
+ for (int attempt = 0; attempt <= maxRetries; attempt++) {
+ try {
+ return
adminClient.describeTopics(topicNames).allTopicNames().get();
+ } catch (Exception e) {
+ if (attempt == maxRetries) {
+ throw new RuntimeException(
+ String.format("Failed to get metadata for topics
%s.", topicNames), e);
+ } else {
+ LOG.warn(
+ "Attempt {} to get metadata for topics {} failed.
Retrying in {} ms...",
+ attempt,
+ topicNames,
+ retryDelay);
+ try {
+ TimeUnit.MILLISECONDS.sleep(retryDelay);
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt(); // Restore
interrupted state
+ LOG.error("Thread was interrupted during metadata
fetch retry delay.", ie);
+ }
+ }
+ }
Review Comment:
Instead of introducing our own retry logic, can't we reuse what's already
implemented in the KafkaAdmin?
```
public static final String RETRIES_CONFIG = "retries";
public static final String RETRIES_DOC = "Setting a value greater than
zero will cause the client to resend any request that fails with a potentially
transient error." +
" It is recommended to set the value to either zero or `MAX_VALUE`
and use corresponding timeout parameters to control how long a client should
retry a request.";
public static final String RETRY_BACKOFF_MS_CONFIG = "retry.backoff.ms";
public static final String RETRY_BACKOFF_MS_DOC = "The amount of time to
wait before attempting to retry a failed request to a given topic partition.
This avoids repeatedly sending requests in a tight loop under some failure
scenarios.";
```
You should be able to set it with even without your PR like this
```
properties.retries = '10',
properties.retry.backoff.ms = '30000',
```
But that also influences the consumer retry behavior.
We could think about supporting `properties.admin.retry = 10`. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
