clintropolis commented on a change in pull request #6455: Remove
consumer.listTopics() method in case when too many topics in kafka causes the
FullGC in Overlord
URL: https://github.com/apache/incubator-druid/pull/6455#discussion_r224854820
##########
File path:
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/supervisor/KafkaSupervisor.java
##########
@@ -1034,24 +1034,9 @@ protected void tryInit()
private void updatePartitionDataFromKafka()
{
- Map<String, List<PartitionInfo>> topics;
- try {
- synchronized (consumerLock) {
- topics = consumer.listTopics(); // updates the consumer's list of
partitions from the brokers
- }
- }
- catch (Exception e) { // calls to the consumer throw NPEs when the broker
doesn't respond
- log.warn(
- e,
- "Unable to get partition data from Kafka for brokers [%s], are the
brokers up?",
-
ioConfig.getConsumerProperties().get(KafkaSupervisorIOConfig.BOOTSTRAP_SERVERS_KEY)
- );
- return;
- }
-
- List<PartitionInfo> partitions = topics.get(ioConfig.getTopic());
+ List<PartitionInfo> partitions =
consumer.partitionsFor(ioConfig.getTopic());
Review comment:
Why did the lock and exception handling get removed? The lock seems needed,
and checking the docs `partitionsFor` can throw a handful of exceptions too:
```
Throws:
WakeupException - if wakeup() is called before or while this function is
called
InterruptException - if the calling thread is interrupted before or while
this function is called
AuthorizationException - if not authorized to the specified topic
TimeoutException - if the topic metadata could not be fetched before
expiration of the configured request timeout
KafkaException - for any other unrecoverable errors
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
