shibd commented on code in PR #991:
URL: https://github.com/apache/pulsar-client-go/pull/991#discussion_r1141872335
##########
pulsar/consumer_partition.go:
##########
@@ -1752,6 +1762,37 @@ func (pc *partitionConsumer) markScaleIfNeed() {
}
}
+func (pc *partitionConsumer) reserveMemory(size int64) {
+ pc.client.memLimit.ForceReserveMemory(size)
+ if pc.client.memLimit.CurrentUsagePercent() >=
receiverQueueShrinkMemThreshold {
Review Comment:
> By the way, is it necessary for the producer to trigger the consumer
shrink? If the memory usage exceeds the threshold, the consumer will always
shrink it's receiver queue, right?
Sorry, the previous example is not quite accurate.
I think the key difference is whether one `consumer` or `producer` needs to
trigger all consumers to reduce the queue size.
- The java client implementation:
When a producer or consumer call the `ReserveMemory` option, If the memory
exceeds the threshold, it triggers all consumers to shrink the queue size
- Your implementation:
When a consumer checks that memory exceeds a threshold, it will only try to
reduce its own queue size.
I think there is a memory usage starvation issue here:
- One client instance has `1` producer instance and `3` consumer.
- If the queue of these three consumers is already relatively large, if the
user of `consumer-1` and `consumer 2` is not actively called (there are many
messages piled up in queurCh), then if `consumer-3` checks and reduces its
queue size after receiving a message, although the memory limit is reduced, but
`consumer-1` and `consumer-2` still occupy more memory (a lot of pending
consumption),
- The same way, if the producer wants to send a message, Since it cannot
trigger other consumers to reduce the queue size, it will also send more slowly.
In short, in this case, Memory is wasted on inactive consumers(consumer-1,
consumer-2)
If it is in implementing the Java client, the problem does not exist.
Each time, all the consumers will reduce the queue size, and then let them
re-promote the queue size according to the activity of the consumers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
