[
https://issues.apache.org/jira/browse/AMQ-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SunGard Global Services Germany updated AMQ-4556:
-------------------------------------------------
Description:
This issue seems to relate to AMQ-2009 and the referenced articles there.
h3. The issue:
After a give period of time, the broker seems to "forget" about received
messages on a queue.
Receiving of new messages and delivering them is unaffected. Just some are lost
in the store.
Checking the DB tables, the missing messages do exists there.
ActiveMQ statistics counters are also aware of them and count those messages as
"pending" in the ActiveMQ console (first colummn). But open the details of a
queue in ActiveMQ console do not show them at all.
h3. Analysis so far
We tried several settings for the queues which had no impact on the issue:
* Activating/Deactivating useCache
* Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax
the issue a little bit)
* KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected
Also some characteristics about affected/not affected queues might help to
analyse this further:
* We have one queue with just one producer and many consumers whereas the
message is quite small (just headers): No problem on that queue even after
thousend of messages
* Queues with multiple producers and consumers and payloaded text-messages seem
to be unaffected - Maybe the JDBC persistent store trottels the processing
enough to "solve" the issue
* Queues with multiple producers and consumers and small messages (just
headers) seem to "enable" the issue. Even with few messages the issue appears
h3. "Recovering" lost messages
Shutdown the (master) broker and restart it. With failover transport this
happens transparent for the clients.
On master startup, all messages from the store (DB) are scanned and
"rediscovered"
h3. Workaround
Use another cursor - VM seems to be fine. Additional settings might be required
to handle shotcommings of the VM cursor (max memory, queue memory etc.)
{code:xml}
<policyEntry queue=">" ...>
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
{code}
h3. Test code
I was not able to create a self-contained unit test.
But the attached code can be used to reproduce the issue quite reliable.
It sends 3000 messages. WIth the following settings it will fail/will work
correctly:
|| Message Size || ActiveMQ cursor || Result ||
| 1 MB (leave line 20 as it is) | vmQueueCursor | (/) - All messages delivered|
| 1 MB (leave line 20 as it is) | store based Cursor| (/) - All messages
delivered|
| remove comment from line 20 | vmQueueCursor| (/) - All messages delivered|
| remove comment from line 20 | store based Cursor| (x) - Some (~0,5% messages
lost)|
For completness also the used activemq-broker.xml was attached - This is the
default one with the JDBC persistent store added.
was:
This issue seems to relate to AMQ-2009 and the referenced articles there.
h3. The issue:
After a give period of time, the broker seems to "forget" about received
messages on a queue.
Receiving of new messages and delivering them is unaffected. Just some are lost
in the store.
Checking the DB tables, the missing messages do exists there.
ActiveMQ statistics counters are also aware of them and count those messages as
"pending" in the ActiveMQ console (first colummn). But open the details of a
queue in ActiveMQ console do not show them at all.
h3. Analysis so far
We tried several settings for the queues which had no impact on the issue:
* Activating/Deactivating useCache
* Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax
the issue a little bit)
* KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected
Also some characteristics about affected/not affected queues might help to
analyse this further:
* We have one queue with just one producer and many consumers whereas the
message is quite small (just headers): No problem on that queue even after
thousend of messages
* Queues with multiple producers and consumers and payloaded text-messages seem
to be unaffected - Maybe the JDBC persistent store trottels the processing
enough to "solve" the issue
* Queues with multiple producers and consumers and small messages (just
headers) seem to "enable" the issue. Even with few messages the issue appears
h3. "Recovering" lost messages
Shutdown the (master) broker and restart it. With failover transport this
happens transparent for the clients.
On master startup, all messages from the store (DB) are scanned and
"rediscovered"
h3. Workaround
Use another cursor - VM seems to be fine. Additional settings might be required
to handle shotcommings of the VM cursor (max memory, queue memory etc.)
{code:xml}
<policyEntry queue=">" ...>
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>
{code}
h3. Test code
I was not able to create a self-contained unit test.
But the attached code can be used to reproduce the issue quite reliable.
It sends 3000 messages. WIth the following settings it will fail/will work
correctly:
|| Message Size || ActiveMQ cursor || Result ||
| 1 MB (leave line 20 as it is) | vmQueueCursor | (/) |
| 1 MB (leave line 20 as it is) | store based Cursor| (/) |
| remove comment from line 20 | vmQueueCursor| (/) |
| remove comment from line 20 | store based Cursor| (x) |
For completness also the used activemq-broker.xml was attached - This is the
default one with the JDBC persistent store added.
> Store based cursor "forgets" messages
> -------------------------------------
>
> Key: AMQ-4556
> URL: https://issues.apache.org/jira/browse/AMQ-4556
> Project: ActiveMQ
> Issue Type: Bug
> Components: Message Store
> Affects Versions: 5.7.0, 5.8.0
> Environment: Durable queues with JDBC persistent store in a
> master/slave configuration with NIO transport
> Reporter: SunGard Global Services Germany
> Priority: Critical
> Attachments: ActiveMQClient.java, activemq.xml
>
>
> This issue seems to relate to AMQ-2009 and the referenced articles there.
> h3. The issue:
> After a give period of time, the broker seems to "forget" about received
> messages on a queue.
> Receiving of new messages and delivering them is unaffected. Just some are
> lost in the store.
> Checking the DB tables, the missing messages do exists there.
> ActiveMQ statistics counters are also aware of them and count those messages
> as "pending" in the ActiveMQ console (first colummn). But open the details of
> a queue in ActiveMQ console do not show them at all.
> h3. Analysis so far
> We tried several settings for the queues which had no impact on the issue:
> * Activating/Deactivating useCache
> * Setting different prefetch values from 0, 1, 100 (at least 1 seems to relax
> the issue a little bit)
> * KahaDB and JDBC persistent (Oracle and MSSQL(jTDS)) are affected
> Also some characteristics about affected/not affected queues might help to
> analyse this further:
> * We have one queue with just one producer and many consumers whereas the
> message is quite small (just headers): No problem on that queue even after
> thousend of messages
> * Queues with multiple producers and consumers and payloaded text-messages
> seem to be unaffected - Maybe the JDBC persistent store trottels the
> processing enough to "solve" the issue
> * Queues with multiple producers and consumers and small messages (just
> headers) seem to "enable" the issue. Even with few messages the issue appears
> h3. "Recovering" lost messages
> Shutdown the (master) broker and restart it. With failover transport this
> happens transparent for the clients.
> On master startup, all messages from the store (DB) are scanned and
> "rediscovered"
> h3. Workaround
> Use another cursor - VM seems to be fine. Additional settings might be
> required to handle shotcommings of the VM cursor (max memory, queue memory
> etc.)
> {code:xml}
> <policyEntry queue=">" ...>
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
> {code}
> h3. Test code
> I was not able to create a self-contained unit test.
> But the attached code can be used to reproduce the issue quite reliable.
> It sends 3000 messages. WIth the following settings it will fail/will work
> correctly:
> || Message Size || ActiveMQ cursor || Result ||
> | 1 MB (leave line 20 as it is) | vmQueueCursor | (/) - All messages
> delivered|
> | 1 MB (leave line 20 as it is) | store based Cursor| (/) - All messages
> delivered|
> | remove comment from line 20 | vmQueueCursor| (/) - All messages delivered|
> | remove comment from line 20 | store based Cursor| (x) - Some (~0,5%
> messages lost)|
> For completness also the used activemq-broker.xml was attached - This is the
> default one with the JDBC persistent store added.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
