[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684398#comment-16684398 ] Jamie goodyear commented on AMQ-7091: - Thank you Alan Protasio for the PR :) > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684381#comment-16684381 ] ASF subversion and git services commented on AMQ-7091: -- Commit 71ce081157a0c4bd007cdf58d68e2e0784f87956 in activemq's branch refs/heads/activemq-5.15.x from [~alanprot] [ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=71ce081 ] AMQ-7091 - O(n) Memory consumption when broker has inactive durable subscribes causing OOM (cherry picked from commit 9012a7871b77da6ecdc403f6b44ef0221345bfb7) > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684377#comment-16684377 ] ASF GitHub Bot commented on AMQ-7091: - Github user asfgit closed the pull request at: https://github.com/apache/activemq/pull/315 > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684375#comment-16684375 ] ASF subversion and git services commented on AMQ-7091: -- Commit 9012a7871b77da6ecdc403f6b44ef0221345bfb7 in activemq's branch refs/heads/master from [~alanprot] [ https://git-wip-us.apache.org/repos/asf?p=activemq.git;h=9012a78 ] AMQ-7091 - O(n) Memory consumption when broker has inactive durable subscribes causing OOM > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684081#comment-16684081 ] Gary Tully commented on AMQ-7091: - [~tabish121] thanks. [~jgenender] Probably no need for any config option then. let it run as is. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16684019#comment-16684019 ] Timothy Bish commented on AMQ-7091: --- [~gtully] I'm afraid I don't recall any longer what the context was for that work. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683962#comment-16683962 ] Jeff Genender commented on AMQ-7091: [~gtully], what if the change was a configuration property? i.e. you can have the cache or you can set the property to do what is being proposed here? So for those strapped for memory, they can just disable the cache? > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683923#comment-16683923 ] Gary Tully commented on AMQ-7091: - [~tabish121] Hi tim, you added the messageReferences cache way back in https://github.com/apache/activemq/commit/943db3c3cb12b4c4504b4966135cf9a0cc69f0ba - this PR proposed to remove that cache b/c it can grow unlimited with the number of pending messages, even it is a just two Longs per message it can be significant. Can you recall the use case that caused that cache to be introduced? I ask b/c it is under threat with this change. It may be that the need for the cache is offset by the changes to what is in the index , or it may be that with the change to what is stored, the cache is more important. My guess is that if it is important you may recall, hense the redirect. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680307#comment-16680307 ] Alan Protasio commented on AMQ-7091: [~gtully] [~jgenender] I did some performance tests before and after the change: The test scenarios were: 1, 10, 100 Consumers Producing and consuming messages in parallel Producing all message before starting the consumers The results are here: [https://github.com/alanprot/activemq/blob/master/AMQ-7091.md] I used KahaDBDurableTopicTest as base for this test. Seems that this change does not have impact performance wise. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679987#comment-16679987 ] Jeff Genender commented on AMQ-7091: This looks good to me...the additional writes are gone. [~gtully], since you are the resident expert in this area, please let us know what you think? > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[
https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678860#comment-16678860
]
Alan Protasio commented on AMQ-7091:
Hi [~gtully]
Thanks again! haha :D
{quote}the solution is to have the data in the pageFile and pageCache such that
it can get flushed from memory at the cost of accessing from pages in normal
operation.
{quote}
Did you see my last proposed change?
I dont know why the bot didn't add the comment (updated the PR) here but i'm
not flushing the data to the pagefile anymore... I'm reusing the data that is
already there:
[https://github.com/apache/activemq/pull/315/commits/98d1be6acaee79f9acd72d2c7c2bdb2358638e18]
We already have this information in other format in the ListIndex ackPositions; So i'm using this.
My comment before was nowadays to discover if one message is referenced by
one subscribe we only do a sd.messageReferences.get(sequenceId) and check if
the reference count is 0.
With my change, we will have to do the same as we already do when we are
sending one message:
See:
[https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2924]
and
[https://github.com/apache/activemq/pull/315/commits/98d1be6acaee79f9acd72d2c7c2bdb2358638e18#diff-e3b8fff8c2133dfd70999705bbb558b3R2935]
As we already have this code when sending a message, i dont know if this is a
problem.
> O(n) Memory consumption when broker has inactive durable subscribes causing
> OOM
> ---
>
> Key: AMQ-7091
> URL: https://issues.apache.org/jira/browse/AMQ-7091
> Project: ActiveMQ
> Issue Type: Bug
> Components: KahaDB
>Affects Versions: 5.15.7
>Reporter: Alan Protasio
>Priority: Major
> Attachments: After.png, Before.png,
> InactiveDurableSubscriberTest.java, memoryAllocation.jpg
>
>
> Hi :D
> One of our brokers was bouncing indefinitely due OOM even though the load
> (TPS) was pretty low.
> Getting the memory dump I could see that almost 90% of the memory was being
> used by
> [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368]
> TreeMap to keep track of what messages were already acked by all Subscribes
> in order to delete them.
> This seems to be a problem as if the broker has an inactive durable
> subscribe, the memory footprint increase proportionally (O) with the number
> of messages sent to the topic in question, causing the broker to die due OOM
> sooner or later (the high memory footprint continue even after a restart).
> You can find attached (memoryAllocation.jpg) a screen shot showing my broker
> using 90% of the memory to keep track of those messages, making it barely
> usable.
> Looking at the code, I could do a change to change the messageReferences to
> use a BTreeIndex:
> final TreeMap messageReferences = new TreeMap<>();
> + BTreeIndex messageReferences;
> Making this change, the memory allocation of the broker stabilized and the
> broker didn't run OOM anymore.
> Attached you can see the code that I used to reproduce this scenario, also
> the memory utilization (HEAP and GC graphs) before and after this change.
> Before the change the broker died in 5 minutes and I could send 48. After
> then change the broker was still pretty healthy after 5 minutes and i could
> send 2265000 to the topic (almost 5x more due high GC pauses).
>
> All test are passing: mvn clean install -Dactivemq.tests=all
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678790#comment-16678790 ] Gary Tully commented on AMQ-7091: - [~alanprot] you have to ask the computer to see if there is any negative performance impact. This reference count in memory is an optimisation introduced for a performance issue, probably a typical fanout case with lots of subs. See the linked issue. This issue is about that cache being unbounded in terms of memory usage, the solution is to have the data in the pageFile and pageCache such that it can get flushed from memory at the cost of accessing from pages in normal operation. I would like to see the actual trade off in publish/ack latency quantified. It may well be negligible; but we need to respect the original use case to verify. In addition, if this warrants a version update, the auto migration path needs to be validated. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16678599#comment-16678599 ] Alan Protasio commented on AMQ-7091: [~gtully] [~jgoodyear] Seems that all tests succeed! > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677471#comment-16677471 ] Alan Protasio commented on AMQ-7091: Hi Guys I changed my PR to use the SequenceSets that already exists in the index... [https://github.com/apache/activemq/pull/315] I'm running all the tests now... Thanks > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677145#comment-16677145 ] Alan Protasio commented on AMQ-7091: Cool! Thanks guys to looking into that... I will change my PR! :D Thanks again!!! :D > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16677021#comment-16677021 ] Jamie goodyear commented on AMQ-7091: - Hi [~alanprot], After some conversation with the community, the SeuqnceSet in AMQ is good to use for this scenario, because then we can control the sizes and toss them when memory is short: https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/disk/util/SequenceSet.java Please have a look at the above, and lets update your PR. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676861#comment-16676861 ] Jamie goodyear commented on AMQ-7091: - Thanks Gary. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676857#comment-16676857 ] Jamie goodyear commented on AMQ-7091: - [~cshannon], [~gtully] I've reviewed Alan's patch and it looks good. The bump in KahaDB version to 7 on Master would open the door to us doing some more changes to KahaDB/MessageDatabase (perhaps some gentle refactoring). I'd like to know what you think. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676855#comment-16676855 ] Gary Tully commented on AMQ-7091: - that info does not need to be persisted, it is duplicate info. I would like to understand the tradeoff - that in mem reference map was introduced as on optimisation via: https://issues.apache.org/jira/browse/AMQ-3467 in 2011 but is in memory only to be fast. It may be possible to squash that map to a sequenceSet of offsets that share a reference count which would address the mem usage without the need to persist to disk. The implications of additional page writes on every add/ack may be significant for the simple fanout (when users keep up) case. Typically the broker has plenty of memory for this sort of thing. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16676851#comment-16676851 ] Jamie goodyear commented on AMQ-7091: - The patch looks good to me - thank you Alan. > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). > > All test are passing: mvn clean install -Dactivemq.tests=all -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AMQ-7091) O(n) Memory consumption when broker has inactive durable subscribes causing OOM
[ https://issues.apache.org/jira/browse/AMQ-7091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674544#comment-16674544 ] ASF GitHub Bot commented on AMQ-7091: - GitHub user alanprot opened a pull request: https://github.com/apache/activemq/pull/315 AMQ-7091 - O(n) Memory consumption when broker has inactive durable s… …ubscribes causing OOM You can merge this pull request into a Git repository by running: $ git pull https://github.com/alanprot/activemq AMQ-7091 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/activemq/pull/315.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #315 commit 29dde9fa81be3bd20fd840daa178259e74cc032e Author: Alan Protasio Date: 2018-11-04T06:22:32Z AMQ-7091 - O(n) Memory consumption when broker has inactive durable subscribes causing OOM > O(n) Memory consumption when broker has inactive durable subscribes causing > OOM > --- > > Key: AMQ-7091 > URL: https://issues.apache.org/jira/browse/AMQ-7091 > Project: ActiveMQ > Issue Type: Bug > Components: KahaDB >Affects Versions: 5.15.7 >Reporter: Alan Protasio >Priority: Major > Attachments: After.png, Before.png, > InactiveDurableSubscriberTest.java, memoryAllocation.jpg > > > Hi :D > One of our brokers was bouncing indefinitely due OOM even though the load > (TPS) was pretty low. > Getting the memory dump I could see that almost 90% of the memory was being > used by > [messageReferences|https://github.com/apache/activemq/blob/master/activemq-kahadb-store/src/main/java/org/apache/activemq/store/kahadb/MessageDatabase.java#L2368] > TreeMap to keep track of what messages were already acked by all Subscribes > in order to delete them. > This seems to be a problem as if the broker has an inactive durable > subscribe, the memory footprint increase proportionally (O) with the number > of messages sent to the topic in question, causing the broker to die due OOM > sooner or later (the high memory footprint continue even after a restart). > You can find attached (memoryAllocation.jpg) a screen shot showing my broker > using 90% of the memory to keep track of those messages, making it barely > usable. > Looking at the code, I could do a change to change the messageReferences to > use a BTreeIndex: > final TreeMap messageReferences = new TreeMap<>(); > + BTreeIndex messageReferences; > Making this change, the memory allocation of the broker stabilized and the > broker didn't run OOM anymore. > Attached you can see the code that I used to reproduce this scenario, also > the memory utilization (HEAP and GC graphs) before and after this change. > Before the change the broker died in 5 minutes and I could send 48. After > then change the broker was still pretty healthy after 5 minutes and i could > send 2265000 to the topic (almost 5x more due high GC pauses). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
