devinbost commented on issue #10813:
URL: https://github.com/apache/pulsar/issues/10813#issuecomment-854130449
@eolivelli @lhotari
Negative permits just means the dispatcher is waiting for permits from the
consumer, which implies it's waiting for the consumer to complete processing of
the messages. I'd like to know if the consumer is waiting for ack's from the
broker.
If you get a thread dump of the consumer when it's frozen, is it waiting for
the producer semaphor? Look for something like this in the consumer thread dump:
```
"myTenant/myNamespace/function-filter-0" prio=5 tid=32 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
local variable:
java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#40
local variable:
java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#41
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
local variable: java.util.concurrent.Semaphore$FairSync#1
at java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
at
org.apache.pulsar.client.impl.ProducerImpl.canEnqueueRequest(ProducerImpl.java:748)
at
org.apache.pulsar.client.impl.ProducerImpl.sendAsync(ProducerImpl.java:391)
local variable: org.apache.pulsar.client.impl.ProducerImpl#1
local variable: org.apache.pulsar.client.impl.ProducerImpl$1#1
local variable: io.netty.buffer.UnpooledHeapByteBuf#17
local variable: org.apache.pulsar.common.api.proto.MessageMetadata#83
local variable: org.apache.pulsar.client.impl.MessageImpl#82
at
org.apache.pulsar.client.impl.ProducerImpl.internalSendAsync(ProducerImpl.java:290)
local variable: java.util.concurrent.CompletableFuture#33
at
org.apache.pulsar.client.impl.TypedMessageBuilderImpl.sendAsync(TypedMessageBuilderImpl.java:103)
```
Another thing to look for is in a consumer heap dump. I'd like to know if
`ProducerImpl.pendingMessages` has messages in it (and how many.) If it's
empty, I'd like to know if `ConsumerImpl.incomingMessages` has messages in it
or not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
