[jira] [Updated] (AMQ-7118) KahaDB store limit can be exceeded with durable subscribers.
[ https://issues.apache.org/jira/browse/AMQ-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jamie goodyear updated AMQ-7118: Description: KahaDB store limit can be exceeded with durable subscribers. AMQ with store limit set, we can observe that the usage continues to increase AFTER PFC is engaged. See below output from KahaDB dump in attachments: This appears to be caused by checkpointAckMessageFileMap. The log files are not GC'd, and the KAHA_ACK_MESSAGE is replicated and the DB log files continue to expand - this can become exponential. Side effect of also not checking storage size in checkpoint update can cause the DB log files to exceed any set limits. The real critical part is the duplicated and leaking Kaha messages which appears to happen with durable subscribers. was: KahaDB leak causing store limit to be exceeded with durable subscribers. AMQ with store limit set, we can observe that the usage continues to increase AFTER PFC is engaged. See below output from KahaDB dump in attachments: This appears to be caused by checkpointAckMessageFileMap. The log files are not GC'd, and the KAHA_ACK_MESSAGE is replicated and the DB log files continue to expand - this can become exponential. Side effect of also not checking storage size in checkpoint update can cause the DB log files to exceed any set limits. The real critical part is the duplicated and leaking Kaha messages which appears to happen with durable subscribers. > KahaDB store limit can be exceeded with durable subscribers. > > > Key: AMQ-7118 > URL: https://issues.apache.org/jira/browse/AMQ-7118 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.16.0, 5.15.8 > Environment: JDK 8 >Reporter: Jamie goodyear >Priority: Critical > Fix For: 5.15.8 > > Attachments: kahaCommands.jpg > > > KahaDB store limit can be exceeded with durable subscribers. > AMQ with store limit set, we can observe that the usage continues to increase > AFTER PFC is engaged. > See below output from KahaDB dump in attachments: > This appears to be caused by checkpointAckMessageFileMap. The log files are > not GC'd, and the KAHA_ACK_MESSAGE is replicated and the DB log files > continue to expand - this can become exponential. Side effect of also not > checking storage size in checkpoint update can cause the DB log files to > exceed any set limits. The real critical part is the duplicated and leaking > Kaha messages which appears to happen with durable subscribers. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7118) KahaDB store limit can be exceeded with durable subscribers.
[ https://issues.apache.org/jira/browse/AMQ-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jamie goodyear updated AMQ-7118: Description: KahaDB store limit can be exceeded with durable subscribers. AMQ with store limit set, we can observe that the usage continues to increase AFTER PFC is engaged. Given time, this growth stabilizes. The issue of having exceeded the store limit remains. See below output from KahaDB dump in attachments: This appears to be caused by checkpointAckMessageFileMap. The log files are not GC'd, and the KAHA_ACK_MESSAGE is replicated and the DB log files continue to expand - this can become exponential. Side effect of also not checking storage size in checkpoint update can cause the DB log files to exceed any set limits. The real critical part is the duplicated and leaking Kaha messages which appears to happen with durable subscribers. was: KahaDB store limit can be exceeded with durable subscribers. AMQ with store limit set, we can observe that the usage continues to increase AFTER PFC is engaged. See below output from KahaDB dump in attachments: This appears to be caused by checkpointAckMessageFileMap. The log files are not GC'd, and the KAHA_ACK_MESSAGE is replicated and the DB log files continue to expand - this can become exponential. Side effect of also not checking storage size in checkpoint update can cause the DB log files to exceed any set limits. The real critical part is the duplicated and leaking Kaha messages which appears to happen with durable subscribers. > KahaDB store limit can be exceeded with durable subscribers. > > > Key: AMQ-7118 > URL: https://issues.apache.org/jira/browse/AMQ-7118 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.16.0, 5.15.8 > Environment: JDK 8 >Reporter: Jamie goodyear >Priority: Critical > Fix For: 5.15.8 > > Attachments: kahaCommands.jpg > > > KahaDB store limit can be exceeded with durable subscribers. > AMQ with store limit set, we can observe that the usage continues to increase > AFTER PFC is engaged. Given time, this growth stabilizes. The issue of having > exceeded the store limit remains. > See below output from KahaDB dump in attachments: > This appears to be caused by checkpointAckMessageFileMap. The log files are > not GC'd, and the KAHA_ACK_MESSAGE is replicated and the DB log files > continue to expand - this can become exponential. Side effect of also not > checking storage size in checkpoint update can cause the DB log files to > exceed any set limits. The real critical part is the duplicated and leaking > Kaha messages which appears to happen with durable subscribers. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (AMQ-7118) KahaDB store limit can be exceeded with durable subscribers.
[ https://issues.apache.org/jira/browse/AMQ-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jamie goodyear updated AMQ-7118: Summary: KahaDB store limit can be exceeded with durable subscribers. (was: KahaDB leak causing store limit to be exceeded with durable subscribers.) > KahaDB store limit can be exceeded with durable subscribers. > > > Key: AMQ-7118 > URL: https://issues.apache.org/jira/browse/AMQ-7118 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.16.0, 5.15.8 > Environment: JDK 8 >Reporter: Jamie goodyear >Priority: Critical > Fix For: 5.15.8 > > Attachments: kahaCommands.jpg > > > KahaDB leak causing store limit to be exceeded with durable subscribers. > AMQ with store limit set, we can observe that the usage continues to increase > AFTER PFC is engaged. > See below output from KahaDB dump in attachments: > This appears to be caused by checkpointAckMessageFileMap. The log files are > not GC'd, and the KAHA_ACK_MESSAGE is replicated and the DB log files > continue to expand - this can become exponential. Side effect of also not > checking storage size in checkpoint update can cause the DB log files to > exceed any set limits. The real critical part is the duplicated and leaking > Kaha messages which appears to happen with durable subscribers. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1864) On-Demand Message Redistribution Can Spontaneously Start Failing in Single Direction
[
https://issues.apache.org/jira/browse/ARTEMIS-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705785#comment-16705785
]
Alexander Kosarev commented on ARTEMIS-1864:
I created an issue in Spring's Jira [https://jira.spring.io/browse/SPR-17554.]
Example project and configuration can be found there.
> On-Demand Message Redistribution Can Spontaneously Start Failing in Single
> Direction
>
>
> Key: ARTEMIS-1864
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1864
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
>Affects Versions: 2.5.0
> Environment: RHEL 6.2
>Reporter: Ilkka Virolainen
>Priority: Major
>
> It's possible that the message redistribution of an Artemis cluster can
> spontaneously fail after running a while. I've witnessed this several times
> using a two node colocated replicating cluster with a basic configuration:
> {code:java}
>
>
> netty-connector
> 500
> 5
> true
> ON_DEMAND
> 1
>
>
> {code}
> After running a while (approx. two weeks) one of the nodes (node a) will stop
> consuming messages from the other node's (node b) internal store-and-forward
> queue. This will result in message redistribution not working from node b ->
> node a but will work from node a -> node b. The cause for this is unknown:
> nothing of note is logged for either broker and JMX shows that the cluster
> topology and the broker cluster bridge connection are intact. This will cause
> significant problems, mainly:
> 1. Client communication will only work as expected if the clients happen to
> connect to the right brokers
> 2. Unconsumed messages will end up piling in the internal store-and-forward
> queue and consume unnecessary resources. It's also possible (but not
> verified) that when messages in the internal queue expire, they leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
