[
https://issues.apache.org/jira/browse/AMQ-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13868631#comment-13868631
]
Arthur Naseef commented on AMQ-4556:
------------------------------------
The test code attached is very helpful, thank you.
In the code, sessions, consumers, and producers are being created in a message
loop, so that every message gets a new session. This is very bad, although it
shouldn't lead to missing messages as described.
Running this program on my system, the client and broker both ramped up to
about 30-40% cpu usage. And message flow was terribly slow.
Another concern - there is no comparison of messages sent and received. To
detect missing messages, are you only looking at counts?
> Store based cursor "forgets" messages
> -------------------------------------
>
> Key: AMQ-4556
> URL: https://issues.apache.org/jira/browse/AMQ-4556
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
> Affects Versions: 5.7.0, 5.8.0
> Environment: Durable queues with JDBC persistent store in a
> master/slave configuration with NIO transport
> Reporter: SunGard Global Services Germany
> Priority: Critical
> Attachments: ActiveMQClient.java, activemq.xml
>
>
> This issue seems to relate to AMQ-2009 and the referenced articles there.
> h3. The issue:
> After a give period of time, the broker seems to "forget" about received
> messages on a queue.
> Receiving of new messages and delivering them is unaffected. Just some are
> lost in the store.
> Checking the DB tables, the missing messages do exists there.
> ActiveMQ statistics counters are also aware of them and count those messages
> as "pending" in the ActiveMQ console (first colummn). But open the details of
> a queue in ActiveMQ console do not show them at all.
> h3. Analysis so far
> We tried several settings for the queues which had no impact on the issue:
> * Activating/Deactivating useCache
> * Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax
> the issue a little bit)
> * KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected
> Also some characteristics about affected/not affected queues might help to
> analyse this further:
> * We have one queue with just one producer and many consumers whereas the
> message is quite small (just headers): No problem on that queue even after
> thousend of messages
> * Queues with multiple producers and consumers and payloaded text-messages
> seem to be unaffected - Maybe the JDBC persistent store trottels the
> processing enough to "solve" the issue
> * Queues with multiple producers and consumers and small messages (just
> headers) seem to "enable" the issue. Even with few messages the issue appears
> h3. "Recovering" lost messages
> Shutdown the (master) broker and restart it. With failover transport this
> happens transparent for the clients.
> On master startup, all messages from the store (DB) are scanned and
> "rediscovered"
> h3. Workaround
> Use another cursor - VM seems to be fine. Additional settings might be
> required to handle shotcommings of the VM cursor (max memory, queue memory
> etc.)
> {code:xml}
> <policyEntry queue=">" ...>
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> {code}
> h3. Test code
> I was not able to create a self-contained unit test.
> But the attached code can be used to reproduce the issue quite reliable.
> It sends 3000 messages. WIth the following settings it will fail/will work
> correctly:
> || Message Size || ActiveMQ cursor || Result ||
> | 1 MB (leave line 20 as it is) | vmQueueCursor | (/) - All messages
> delivered|
> | 1 MB (leave line 20 as it is) | store based Cursor| (/) - All messages
> delivered|
> | remove comment from line 20 | vmQueueCursor| (/) - All messages delivered|
> | remove comment from line 20 | store based Cursor| (x) - Some (~0,5%
> messages lost)|
> For completness also the used activemq-broker.xml was attached - This is the
> default one with the JDBC persistent store added.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
