devinbost edited a comment on issue #6054:
URL: https://github.com/apache/pulsar/issues/6054#issuecomment-818463224
The permits seem to jump to negative numbers in the broker debug logs with
very little explanation. (The logs below merge together the logs from all three
brokers in this test cluster.)
If I filter out ZK logs, after this permit-related line:
> Apr 12 18:55:52 fab08 8f9131691776: 2021-04-13T00:55:52,442
[pulsar-io-25-64] DEBUG org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant/myNamespace/topic-1-PersistentSubscription{topic=persistent://myTenant/myNamespace/topic-1,
name=myTenant/myNamespace/function-2}] Added more flow control message permits
500 (old was: 419), blocked = false
I see some ManagedLedgerImpl logs, including this one:
> Apr 12 18:55:52 fab08 8f9131691776: 2021-04-13T00:55:52,442
[bookkeeper-ml-workers-OrderedExecutor-7-0] DEBUG
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl -
[myTenant/myNamespace/persistent/topic-1] Resending 100 pending messages
Then, there are more ManagedLedgerImpl and some OpReadEntry logs.
Then, there are EntryCacheImpl and ManagedLedgerImpl logs.
Then, I get this:
> Apr 12 18:56:22 fab08 8f9131691776: 2021-04-13T00:56:22,301
[pulsar-io-25-64] DEBUG org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant/myNamespace/topic-1-PersistentSubscription{topic=persistent://myTenant/myNamespace/topic-1,
name=myTenant/myNamespace/function-2}] Added more flow control message permits
500 (old was: -14), blocked = false
The next time I get a `Distributing x messages to n consumers` involving
that topic, it says:
`Distributing 49 messages to 1 consumers`, which is immediately followed by:
`No consumers found with available permits, storing 29 positions for later
replay`
So, it's like the permits are decrementing in some way that's not reflected
in the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
