abhishekagarwal87 commented on a change in pull request #12161:
URL: https://github.com/apache/druid/pull/12161#discussion_r788753383
##########
File path:
extensions-core/kinesis-indexing-service/src/main/java/org/apache/druid/indexing/kinesis/KinesisRecordSupplier.java
##########
@@ -663,34 +660,35 @@ public String
getEarliestSequenceNumber(StreamPartition<String> partition)
return getSequenceNumber(partition, ShardIteratorType.TRIM_HORIZON);
}
+ /**
+ * Use the API listShards which is the recommended way instead of
describeStream
+ * listShards can return 1000 shards per call and has a limit of 100TPS
+ * This makes the method resilient to LimitExceeded exceptions (compared to
100 shards, 10 TPS of describeStream)
+ *
+ * @param stream name of stream
+ *
+ * @return Set of Shard ids
+ */
@Override
public Set<String> getPartitionIds(String stream)
{
- return wrapExceptions(
- () -> {
- final Set<String> retVal = new HashSet<>();
- DescribeStreamRequest request = new DescribeStreamRequest();
- request.setStreamName(stream);
-
- while (request != null) {
- final DescribeStreamResult result =
kinesis.describeStream(request);
- final StreamDescription streamDescription =
result.getStreamDescription();
- final List<Shard> shards = streamDescription.getShards();
-
- for (Shard shard : shards) {
- retVal.add(shard.getShardId());
- }
-
- if (streamDescription.isHasMoreShards()) {
-
request.setExclusiveStartShardId(Iterables.getLast(shards).getShardId());
- } else {
- request = null;
- }
- }
-
+ return wrapExceptions(() -> {
+ final Set<String> retVal = new HashSet<>();
Review comment:
this is getting modified later in the code. do you mean returning an
immutable copy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
