devinbost edited a comment on issue #6054:
URL: https://github.com/apache/pulsar/issues/6054#issuecomment-823815763
I was able to demonstrate the issue with the new debug statements currently
in a build I created from master (2.8.0-SNAPSHOT). If you study the logs below,
you will see that more permits were dispatched than available, which blocks
dispatching of new messages and results in repeated entries of this message:
`Trigger new read after receiving flow control message with permits -11
after adding 0 permits`
```
2021-04-21T05:40:47,933 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 500 message permits in
broker.service.Consumer before updating dispatcher for consumer
2021-04-21T05:40:47,933 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits 596 after adding 500 permits
2021-04-21T05:40:47,933 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added more flow control message permits
500 (old was: 96), blocked = false
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(248 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(249 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(14 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(10 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(14 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(12 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(13 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(1 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(15 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(15 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:47,943 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(16 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
