HI Robbie.
Looks like we found root cause.
This is network problem.
Our java app run in Azure Cloud in VM. The was network issues - socket
Connections were exhausted.
So, we have next situation:
1) SB queue is empty
2) Java app constantly says that it ready to accept 10 messages (we have 10
The second connection open is specific to my IDE environment.
I run my java Spring Boot App with IDEA.
IDEA has integration with Spring Boot and perform call to /health HTTP
endpoint which is checks that connection is created.
Stack trace for 'normal' start:
>The only odd thing was that there was no message transfer to
>the first consumer, which your subsequent re-test being 'successful'
>suggests is because the test enviornment wasnt clean to start with and
>perhaps you had other consumers running.
Yes, it is strange. I run in clean environment and
I notice that containerId with value 'xxx' is hardcoded in our app. May this
is issue?
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
-
To unsubscribe, e-mail:
> Do you have logs from that run?
Yes I have logs from run on clean freshly created queue in Azure SB:
output: https://gist.github.com/qwazer/88380c98325f24ffcf55549300f4b409
jms_trace: https://gist.github.com/qwazer/3a4524b52dcdb39040d2b122f8820b99
--
Sent from:
Seems experiment was not clean.
If I use freshly created queue in Azure SB I see that only 1 message in SB
has increased delivery count (although it was not consumed by app, but this
is expected).
Looking for leaking AMQP links, etc
--
Sent from:
jms.connection.parameters still the same
amqp.idleTimeout=12
jms.prefetchPolicy.all=0
amqp.drainTimeout=0
amqp.traceFrames=true
jms.connectTimeout=12
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
>Once the message
>arrives, the credit has been used and as only 1 was outstanding, any
>drain attempt that occurred can be considered complete. The behaviour
>should be visible in your logs.
Seems it is not true. See results of my experiment:
Experiment with Azure SB and AMQP flow control
>Are these logs from such an experiment, as they seem valid from a
>protocol perspective (delivery with id 0 arrives, then is accepted)?
Yes these logs are from the experiment.
1) Start debug session of my java program. I send `Flow` frame with 1 credit
limit, then receive 'Transfer' then
Hi Robbie
Indeed we have some different cases on PROD.
1) Some messages processed by our app several times. Frequent use case.
2) Sometimes we got message with JMXDeliveryCount=2 with no log entries with
JMXDeliveryCount=1 on the same message before or after.
3) We processed message with
Thanks for your comments.
I'll try to collect more trace logs and will return with my findings.
2 problems here:
1) I cannot reliably reproduce this issue locally
2) Log rotation on PROD is too fast, so It's hard to get logs in moment of
issue is reproducing.
--
Sent from:
I have 3 observation:
1.) Azure Service Bus place lock on message when receive Flow frame with
non-zero link credit
The clock for the expiration of the lock on the message inside the
`queue/topic
I'll try to take heapdump when the issue will be reproduced on PROD
--
Sent from: http://qpid.2158936.n2.nabble.com/Apache-Qpid-users-f2158936.html
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional
Robbie, thanks a lot for answer.
I'm trying to investigate issue more deeply and reproduce it locally (this
is issue from PROD).
Yes, receiveLocalOnly=true is redudant in our config since prefetch is
disabled, I read the source code and documentation.
Also we increase number of CPU cores on
We use qpid-jms-client version 0.27.0 to work with Azure SB.
What the cause of processing messages out of order under load?
Currently we have next situation:
1) read message from SB by driver with JMSXDeliveryCount=1
2) SB place lock and increase deliveryCounter
3) after 5 minutes lock
15 matches
Mail list logo
