GitHub user mrmomr40 closed a discussion: Topic consuming blocked on one failed
message in Apache Pulsar
I have a problem with `Key_Shared` subscription.
If a consumer fails to process one message, processing of all other messages
with other keys is also stalled. Including the other consumers of the
subscription (there are 10-15 of them). On a processing fail, I don’t send a
nack in order not to break the message order. The consumer just has a minute to
try to process other keys, and then restarted.
I was able to reproduce similar behaviour in test. I produce 1000 messages with
4 different keys. Only one of the 1000 messages fails to be processed. An
individual ack is performed successfully from the consumer point of view, even
after the corrupted message fails. But, apparently, the acks are not “commited”
and discarded by the broker. Interesting thing is that if I produce messages
with some time gap (say, 1 second), the corrupted message doesn’t block the
further messages. So, I suspect some batching be the cause. However, as I said,
I acknowledge individually.
What could cause the blocking of the all the keys?
I have Pulsar 2.8.1, but the same behaviour is also observed on 3.0.0. You can
find the fragment of topic-stats below.
```
"subscriptions" : {
"subscriptionName" : {
"msgRateOut" : 0.0,
"msgThroughputOut" : 0.0,
"bytesOutCounter" : 1099336,
"msgOutCounter" : 11480,
"msgRateRedeliver" : 0.0,
"messageAckRate" : 0.0,
"chunkedMessageRate" : 0,
"msgBacklog" : 4,
"backlogSize" : 119640,
"earliestMsgPublishTimeInBacklog" : 0,
"msgBacklogNoDelayed" : 4,
"blockedSubscriptionOnUnackedMsgs" : false,
"unackedMessages" : 820,
"type" : "Key_Shared",
"msgRateExpired" : 0.0,
"totalMsgExpired" : 0,
"lastExpireTimestamp" : 0,
"lastConsumedFlowTimestamp" : 1691146017587,
"lastConsumedTimestamp" : 1691146017565,
"lastAckedTimestamp" : 0,
"lastMarkDeleteAdvancedTimestamp" : 0,
"consumers" : [ {
...
"messageAckRate" : 0.0,
...
"availablePermits" : 680,
"unackedMessages" : 820,
...
"blockedConsumerOnUnackedMsgs" : false,
"readPositionWhenJoining" : "16:3",
"lastAckedTimestamp" : 0,
"lastConsumedTimestamp" : 1691146017565,
"lastConsumedFlowTimestamp" : 1691146017582,
...
"lastAckedTime" : "1970-01-01T00:00:00Z",
"lastConsumedTime" : "2023-08-04T10:46:57.565Z"
},
```
GitHub link: https://github.com/apache/pulsar/discussions/20946
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
