kfaraz commented on code in PR #13503:
URL: https://github.com/apache/druid/pull/13503#discussion_r1041715675
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentAllocationQueue.java:
##########
@@ -198,6 +199,24 @@ public Future<SegmentIdWithShardSpec>
add(SegmentAllocateRequest request)
return futureReference.get();
}
+ /**
+ * Returns the key for a batch that is not added to the queue yet and/or has
+ * available space. Throws an exception if the queue is already full and no
+ * batch has available capacity.
+ */
+ private AllocateRequestKey getKeyForAvailableBatch(SegmentAllocateRequest
request)
+ {
+ for (int batchIncrementalId = 0; batchIncrementalId < MAX_QUEUE_SIZE;
++batchIncrementalId) {
+ AllocateRequestKey nextKey = new AllocateRequestKey(request,
maxWaitTimeMillis, batchIncrementalId);
+ AllocateRequestBatch nextBatch = keyToBatch.get(nextKey);
+ if (nextBatch == null || nextBatch.size() < MAX_BATCH_SIZE) {
+ return nextKey;
+ }
+ }
+
+ throw new ISE("Allocation queue is at capacity, all batches are full.");
Review Comment:
`MAX_QUEUE_SIZE` applies to batches. I am using it in the loop above because
that is the maximum possible number of batches, even for the same datasource,
group id, etc
It is extremely unlikely to hit the max queue limit. It can happen if say,
we have 2000 datasources, all doing streaming ingestion concurrently. If the
queue is full, new incoming allocate actions are rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
