On 08/21/2013 03:16 PM, Jimmy Jones wrote:
I've got an simple processing system using the 0.22 C++ broker, all
on one box, where an external system posts messages to the default
headers exchange, and an ingest process receives them using a ring
queue, transforms them and outputs to a different headers exchange.
Various other processes pick messages of interest off that exchange
using ring queues. Recently however the system has been stalling -
I'm still receiving lots of data from the other system, but the
ingest process suddenly goes to <5% CPU usage and its queue fills up
and messages start getting discarded from the ring, the follow on
processes go to practically 0% CPU and qpidd hovers around 95-120%
CPU (normally its ~75%) and the rest of the system pretty much goes
idle (no swapping, there is free memory)
I attached to the ingest process with gdb and it was stuck in send
(waitForCapacity/waitForCompletionImpl) - I notice this can block.
Is there any queue bound to the second headers exchange, i.e. to the one
this ingest process is sending to, that is not a ring queue? (If you run
qpid-config queue -r, you get a quick listing of the queues and their
bindings).
I've run qpid-config queue, and all my queues have --limit-policy=ring, apart
from a UUID one which I presume is qpid-config itself. Are there any other
useful
debugging things I can do?
What does qpid-stat -q show? Is it possible to test whether the broker
is still responsive e,g, by sending and receiving messages through a
test queue/exchange? Are there any errors in the logs? Are any of the
queues durable (and messages persistent)?
Another thing might be a ptrace of the broker process. Maybe two or
three with a short delay between them.
For some reason it seems like the broker is not sending back
confirmation to the sender in the ingest process, causing that to block.
Ring queues shouldn't be subject to producer flow control so we need to
figure out what other reason there could be for that.
If there was a queue to which messages were enqueued that started to
apply rpoducer flow control, then that would block your ingest process
(and since the messages are still coming in, the broker would spend all
its time just removing old ones to make space).
I'd expect the broker to use less CPU when discarding messages rather
than shipping them to consumers? But I'm saying that without much knowledge
of the code!
I would expect that too. However the discarding actually does a lot of
the same things. It also does them on the producers thread which may
then cause other inefficiencies.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
