rdhabalia opened a new pull request #9789:
URL: https://github.com/apache/pulsar/pull/9789
### Motivation
We have been frequently seeing issue where subscription gets stuck on
different topics and broker is not dispatching messages though consumer has
available-permits and no pending reads (example #9788). It can happen due to
regression bug or unknown issue when expiry runs.. one of the workarounds is
manually unload the topic and reload it which is not feasible if this happens
frequently to many topics. Or broker should have the capability to discover
such stuck subscriptions and unblock them.
Below example shows that:
subscription has available-permit>0, there is no pending reads, cursor's
read-position is not moving forward and that builds the backlog until we unload
the topic. It happens frequently due to unknown reason:
```
STATS-INTERNAL:
"sub1" : {
"markDeletePosition" : "11111111:15520",
"readPosition" : "11111111:15521",
"waitingReadOp" : false,
"pendingReadOps" : 0,
"messagesConsumedCounter" : 115521,
"cursorLedger" : 585099247,
"cursorLedgerLastEntry" : 597,
"individuallyDeletedMessages" : "[]",
"lastLedgerSwitchTimestamp" : "2021-02-25T19:55:50.357Z",
"state" : "Open",
"numberOfEntriesSinceFirstNotAckedMessage" : 1,
"totalNonContiguousDeletedMessagesRange" : 0,
STATS:
"sub1" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"msgBacklog" : 30350,
"blockedSubscriptionOnUnackedMsgs" : false,
"msgDelayed" : 0,
"unackedMessages" : 0,
"type" : "Shared",
"msgRateExpired" : 0.0,
"consumers" : [ {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"msgRateRedeliver" : 0.0,
"consumerName" : "C1",
"availablePermits" : 723,
"unackedMessages" : 0,
"blockedConsumerOnUnackedMsgs" : false,
"metadata" : { },
"connectedSince" : "2021-02-25T19:55:50.358285Z",
```
### Modification
Add capability in broker to periodically check if subscription is stuck and
unblock it if needed. This check is controlled by flag and for initial release
it can be disabled by default (and we can enable by default in later release)
### Result
It helps broker to handle stuck subscription and logs the message for later
debugging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
