poorbarcode commented on code in PR #21488: URL: https://github.com/apache/pulsar/pull/21488#discussion_r1383732900
########## pip/pip-314.md: ########## @@ -0,0 +1,57 @@ +# PIP-314: Add metrics pulsar_subscription_redelivery_messages + +# Background knowledge + +## Delivery of messages in normal + +To simplify the description of the mechanism, let's take the policy [Auto Split Hash Range](https://pulsar.apache.org/docs/3.0.x/concepts-messaging/#auto-split-hash-range) as an example: + +| `0 ~ 16,384` | `16,385 ~ 32,768` | `32,769 ~ 65,536` | +|-------------------|-------------------|--------------------------------| +| ------- C1 ------ | ------- C2 ------ | ------------- C3 ------------- | + +- If the entry key is between `-1(non-include) ~ 16,384(include)`, it is delivered to C1 +- If the entry key is between `16,384(non-include) ~ 32,768(include)`, it is delivered to C2 +- If the entry key is between `32,768(non-include) ~ 65,536(include)`, it is delivered to C3 + +# Motivation + +For the example above, if `C1` is stuck or consumed slowly, the Broker will push the entries that should be delivered to `C1` into a memory collection `redelivery_messages` and read next entries continue, then the collection `redelivery_messages` becomes larger and larger and take up a lot of memory. When sending messages, it will also determine the key of the entries in the collection `redelivery_messages`, affecting performance. Review Comment: > When you write "V1 is stuck" you mean the internal queue of C1 is full? After a client-side consumer's incoming queue is full, the Broker will stop delivering messages to it and just push these messages into the queue `redelivery_messages` > When that happens, messages are continued to be read from the topic, but messages whose keys belongs to c1 will be placed in a buffer called Redelivery Mesages? Yes. > Is this buffer one for all "stuck" consumers or each consumer has it? If the `stuck` consumer is still online, it is for `stuck` consumer. Once it is offline, the messages should be delivered to another consumer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
