devinbost commented on issue #6054:
URL: https://github.com/apache/pulsar/issues/6054#issuecomment-823815763
I was able to demonstrate the issue with the new debug statements currently
in a build I created from master (2.8.0-SNAPSHOT). If you study the logs below,
you will see that more permits were dispatched than available, which blocks
dispatching of new messages and results in repeated entries of this message:
`Trigger new read after receiving flow control message with permits -11
after adding 0 permits`
> 2021-04-21T05:40:47,933 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 500 message permits in
broker.service.Consumer before updating dispatcher for consumer
> 2021-04-21T05:40:47,933 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits 596 after adding 500 permits
> 2021-04-21T05:40:47,933 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added more flow control message permits
500 (old was: 96), blocked = false
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(248 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(249 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(14 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(10 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(14 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(12 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(13 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(1 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(15 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,942 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(15 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:47,943 [bookkeeper-ml-workers-OrderedExecutor-0-0] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1] Added -(16 minus 0) permits to
TOTAL_AVAILABLE_PERMITS_UPDATER in PersistentDispatcherMultipleConsumers
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.Consumer -
[persistent://myTenant2/myNamespace/topic-1-partition-0-PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}] Added 0 blocked permits to
broker.service.Consumer for consumer
> 2021-04-21T05:40:48,028 [pulsar-io-28-47] DEBUG
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- [persistent://myTenant2/myNamespace/topic-1-partition-0 /
myTenant/myNamespace/function1-Consumer{subscription=PersistentSubscription{topic=persistent://myTenant2/myNamespace/topic-1-partition-0,
name=myTenant/myNamespace/function1}, consumerId=0, consumerName=fa0ee,
address=/10.20.69.28:55028}] Trigger new read after receiving flow control
message with permits -11 after adding 0 permits
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
