clintropolis commented on a change in pull request #7428: Add errors and state
to stream supervisor status API endpoint
URL: https://github.com/apache/incubator-druid/pull/7428#discussion_r281368233
##########
File path:
extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/KafkaRecordSupplier.java
##########
@@ -122,37 +126,44 @@ public void seekToLatest(Set<StreamPartition<Integer>>
partitions)
@Override
public Long getLatestSequenceNumber(StreamPartition<Integer> partition)
{
- Long currPos = consumer.position(new TopicPartition(partition.getStream(),
partition.getPartitionId()));
+ Long currPos = getPosition(partition);
seekToLatest(Collections.singleton(partition));
- Long nextPos = consumer.position(new TopicPartition(partition.getStream(),
partition.getPartitionId()));
+ Long nextPos = getPosition(partition);
seek(partition, currPos);
return nextPos;
}
@Override
public Long getEarliestSequenceNumber(StreamPartition<Integer> partition)
{
- Long currPos = consumer.position(new TopicPartition(partition.getStream(),
partition.getPartitionId()));
+ Long currPos = getPosition(partition);
seekToEarliest(Collections.singleton(partition));
- Long nextPos = consumer.position(new TopicPartition(partition.getStream(),
partition.getPartitionId()));
+ Long nextPos = getPosition(partition);
seek(partition, currPos);
return nextPos;
}
@Override
public Long getPosition(StreamPartition<Integer> partition)
{
- return consumer.position(new TopicPartition(partition.getStream(),
partition.getPartitionId()));
+ return wrapExceptions(() -> consumer.position(new TopicPartition(
+ partition.getStream(),
+ partition.getPartitionId()
+ )));
}
@Override
public Set<Integer> getPartitionIds(String stream)
{
- List<PartitionInfo> partitions = consumer.partitionsFor(stream);
- if (partitions == null) {
- throw new ISE("Topic [%s] is not found in KafkaConsumer's list of
topics", stream);
- }
- return
partitions.stream().map(PartitionInfo::partition).collect(Collectors.toSet());
+ return wrapExceptions(() -> {
+ // use consumer.listTopics() instead of partitionsFor() to force a
remote call so we can detect stream issues
+ Map<String, List<PartitionInfo>> topics = consumer.listTopics();
Review comment:
I think this was explicitly changed to `partitionsFor` because of the
overhead of `listTopics` if you have a ton of topics. See #6455
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
