I'm having loads of problems with RabbitMQ 2.8.x's dead letter handling and I'd
appreciate some advice. It seems, and this is just from observation, that
under load dead letter messages can cause the original queue to be completely
removed; I'm assuming that removal is done because the queue is considered
invalidly configured, which I can't see being my case.
The setup is simple:
producers ---> topic exchange (T1) ---> queue (Q1) ---rejection---> topic
exchange (T2) ---> queue (Q2)
Q1 is setup to dead letter to exchange T2 with a routing key specified, which
causes the messages to be delivered to Q2. I have tried setups where T1 and T2
are the same exchange, or different; messages have been persistent &
non-persistent; all with the same result.
Under normal load our system generates no more than 1000 messages per minute,
so it's very low volume, and each message ranges from a few 100 bytes to a few
kb. These messages appear in bursts: the application is an (internal) web
application and the messages are caused by users interacting with it. The
ratio of messages that will be dead lettered is about 1 in 50; I know this
because I haven't written the code to handle them and, hence, the consumers
reject them. At the moment the system is running with only 8 producers and 1-2
consumers, all written in Ruby.
All-in-all an extremely low volume system.
However, I have been running a very high load (for this system) run: 4
producers are generating about 500 messages per second to T1 and will be
running for a few hours. To cope with this I've started 11 consumers which are
consuming from Q1.
"Randomly" (which is always a good way of describing these things!) the broker
will decide to delete Q1. From the rabbit-sasl.log I see the supervisor report
at the bottom of this email. I'm very new to Erlang but this seems to suggest
that there is a problem in the replace_headers call, but I have no clue how to
go about identifying why.
As I can find no-one else reporting similar issues I have to assume it's
something about the setup I have:
$ erl --version
Erlang R15B (erts-5.9) [source] [64-bit] [smp:16:16] [async-threads:0] [hipe]
[kernel-poll:false]
$ hostname -a
Linux server-name 2.6.32-31-server #61-Ubuntu SMP Fri Apr 8 19:44:42 UTC 2011
x86_64 GNU/Linux
rabbitmq-server-generic-unix-2.8.1 installed not as a separate user as I do not
have admin rights on this machine, which is a 16 processors & 16GB RAM.
Any pointers, help or advice gratefully received.
Thanks,
Matthew
=SUPERVISOR REPORT==== 31-Mar-2012::15:35:16 ===
Supervisor: {local,rabbit_amqqueue_sup}
Context: child_terminated
Reason: {{badrecord,'P_basic'},
[{rabbit_basic,replace_headers,2,[]},
{rabbit_amqqueue_process,make_dead_letter_msg,4,[]},
{rabbit_amqqueue_process,dead_letter_msg_existing_dlx,4,[]},
{gen_server2,handle_msg,2,[]},
{proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,237}]}]}
Offender: [{pid,<0.14125.390>},
{name,rabbit_amqqueue},
{mfa,
{rabbit_amqqueue_process,start_link,
[{amqqueue,
{resource,<<"production">>,queue,
<<"psd.warehouse_two">>},
true,false,none,
[{<<"x-dead-letter-exchange">>,longstr,
<<"psd.deadletters">>},
{<<"x-dead-letter-routing-key">>,longstr,
<<"production.deadletter">>}],
none,[],undefined}]}},
{restart_type,temporary},
{shutdown,4294967295},
{child_type,worker}]
--
You received this message because you are subscribed to the Google Groups
"rabbitmq-discuss" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/rabbitmq-discuss?hl=en.
