'Zombie' messages created in KahaDB after failover, with warning 'Duplicate
message add attempt rejected.'
----------------------------------------------------------------------------------------------------------
Key: AMQ-2803
URL: https://issues.apache.org/activemq/browse/AMQ-2803
Project: ActiveMQ
Issue Type: Bug
Components: Message Store
Affects Versions: 5.3.2
Reporter: Adrian Trenaman
Am doing some failover testing of ActiveMQ, sending 10,000 messages to the
broker with a live consumer; the producer and consumer are transactional, and
the broker is configured to use KahaDB. On failover of the broker, I see log
warnings like this:
{code}
WARN | Duplicate message add attempt rejected. Message id:
ID:Ade's-54024-1277715585703-0:29:1:1:240
WARN | Duplicate message add attempt rejected. Message id:
ID:Ade's-54024-1277715585703-0:24:1:1:241
WARN | Duplicate message add attempt rejected. Message id:
ID:Ade's-54024-1277715585703-0:23:1:1:242
WARN | Duplicate message add attempt rejected. Message id:
ID:Ade's-54024-1277715585703-0:0:1:1:237
{code}
When the test has run its course, I discover that there are no messages lost -
all my messages have gone through successfully. However, I notice that there
are in fact a number of 'zombie' messages on the queue: by 'zombie' I mean that
the messages are in the store, and the JMX queueSize says they're there.
However, if you browse the queue you don't see them, and, if you have a
consumer on the queue, you won't receive them either.
Looking into the code, in {{org.apache.activemq.store.kahadb.MessageDatabase}}
you can see the following very interesting comments concerning the log warning:
{code}
// If the message ID as indexed, then the broker asked us to
store a DUP
// message. Bad BOY! Don't do it, and log a warning.
LOG.warn("Duplicate message add attempt rejected. Message id:
"+command.getMessageId());
// TODO: consider just rolling back the tx.
sd.messageIdIndex.put(tx, command.getMessageId(), previous);
{code}
So. It seems to me that while we're detecting the condition of the broker
trying to reinsert a message that's already in the KahaDB correctly, we're not
handling this case very well. The outcome is that while we do not get any
message loss, we do get worrying warning messages *and* we get erroneous
reporting of the queueSize in JMX which will mislead and create the perception
of undelivered messages.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
