[jira] [Commented] (ARTEMIS-1559) Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM exception
[
https://issues.apache.org/jira/browse/ARTEMIS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458726#comment-16458726
]
Tommy Lillehagen commented on ARTEMIS-1559:
---
Yes, that's in line with our observations. And as you say, the memory is indeed
claimable by the GC, so it's not a leak per se. What we're concerned about is
the lack of protection on the broker side against "malicious" clients as
demonstrated in the repro-branch. I.e., are there any options that can be used
on the broker to protect against this kind of behaviour? Or would we simply
have to deal with that by implementing our own barrier, e.g., adding an
exponential backoff or similar in front of the broker?
> Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM
> exception
> --
>
> Key: ARTEMIS-1559
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1559
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Tommy Lillehagen
>Priority: Major
> Attachments: mem-leak-source.png, mem-leak.png
>
>
> Repro steps:
> * We have a broker (A) set up with SSL enabled (using Artemis v2.4 / v2.3)
> * A client (B) set up to use plain (non-TLS) communication (same version of
> Artemis)
> * Trying to establish a connection between (B) and (A) triggers multiple
> retries
> * Each message gets, from what I can tell, rejected quickly by (A), but each
> iteration leaks heap memory
> * Due to the number of retries in a short amount a time (~1000s from what I
> can tell) causes the above to grow the heap by 470M or so within less than 10
> seconds (the set timeout)
> * This consequently results in an out of memory exception
> The above behaviour is observed in both version 2.3 and 2.4.
> We've tested older versions (2.1 and 2.2), and neither of those manifest the
> same problem.
> I've run some profiling on (A), and {{NettyAcceptor.initChannel}}
> ({{getSslHandler()}}) seems to be a critical point (will include screenshots).
> That being said, most of the accumulated heap memory seems to be claimable
> and is mostly collected during the next GC cycle in the tests that I've run.
> Source: https://github.com/corda/corda/pull/2252
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1559) Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM exception
[
https://issues.apache.org/jira/browse/ARTEMIS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458719#comment-16458719
]
Justin Bertram commented on ARTEMIS-1559:
-
[~tommy.lillehagen], I was able to reproduce this in part. All I did was change
the {{connectionFactory.ConnectionFactory}} in the {{jndi.properties}} of the
{{ssl-enabled}} example to this:
{noformat}
connectionFactory.ConnectionFactory=tcp://localhost:5500?sslEnabled=false=activemq/server0/activemq.example.truststore=activemqexample=-1
{noformat}
The log was filled with a tremendous amount of these:
{noformat}
ERROR [org.apache.activemq.artemis.core.server] AMQ224088: Timeout (10 seconds)
while handshaking has occurred.
{noformat}
I attached JConsole to the broker process and look at the memory usage and
while it did spike while the client was attempting to connect after the client
failed and a GC was performed the heap usage went back to the same level as
before the client connection. In other words, I don't see any kind of leak.
Also, I would expect memory to spike when the broker receives lots of what it
believes are SSL connections due to the overhead that SSL imposes. Therefore, I
don't see any problems on the broker-side of things. That said, the client
shouldn't be initiating so many connections, and that will take some more
investigation to diagnose and fix.
> Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM
> exception
> --
>
> Key: ARTEMIS-1559
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1559
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Tommy Lillehagen
>Priority: Major
> Labels: memory-leak
> Attachments: mem-leak-source.png, mem-leak.png
>
>
> Repro steps:
> * We have a broker (A) set up with SSL enabled (using Artemis v2.4 / v2.3)
> * A client (B) set up to use plain (non-TLS) communication (same version of
> Artemis)
> * Trying to establish a connection between (B) and (A) triggers multiple
> retries
> * Each message gets, from what I can tell, rejected quickly by (A), but each
> iteration leaks heap memory
> * Due to the number of retries in a short amount a time (~1000s from what I
> can tell) causes the above to grow the heap by 470M or so within less than 10
> seconds (the set timeout)
> * This consequently results in an out of memory exception
> The above behaviour is observed in both version 2.3 and 2.4.
> We've tested older versions (2.1 and 2.2), and neither of those manifest the
> same problem.
> I've run some profiling on (A), and {{NettyAcceptor.initChannel}}
> ({{getSslHandler()}}) seems to be a critical point (will include screenshots).
> That being said, most of the accumulated heap memory seems to be claimable
> and is mostly collected during the next GC cycle in the tests that I've run.
> Source: https://github.com/corda/corda/pull/2252
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1559) Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM exception
[
https://issues.apache.org/jira/browse/ARTEMIS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449748#comment-16449748
]
Tommy Lillehagen commented on ARTEMIS-1559:
---
[~jbertram] [~gaohoward] - has any of you had a chance to look at this yet?
> Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM
> exception
> --
>
> Key: ARTEMIS-1559
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1559
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Tommy Lillehagen
>Priority: Major
> Labels: memory-leak
> Attachments: mem-leak-source.png, mem-leak.png
>
>
> Repro steps:
> * We have a broker (A) set up with SSL enabled (using Artemis v2.4 / v2.3)
> * A client (B) set up to use plain (non-TLS) communication (same version of
> Artemis)
> * Trying to establish a connection between (B) and (A) triggers multiple
> retries
> * Each message gets, from what I can tell, rejected quickly by (A), but each
> iteration leaks heap memory
> * Due to the number of retries in a short amount a time (~1000s from what I
> can tell) causes the above to grow the heap by 470M or so within less than 10
> seconds (the set timeout)
> * This consequently results in an out of memory exception
> The above behaviour is observed in both version 2.3 and 2.4.
> We've tested older versions (2.1 and 2.2), and neither of those manifest the
> same problem.
> I've run some profiling on (A), and {{NettyAcceptor.initChannel}}
> ({{getSslHandler()}}) seems to be a critical point (will include screenshots).
> That being said, most of the accumulated heap memory seems to be claimable
> and is mostly collected during the next GC cycle in the tests that I've run.
> Source: https://github.com/corda/corda/pull/2252
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1559) Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM exception
[
https://issues.apache.org/jira/browse/ARTEMIS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387504#comment-16387504
]
Tommy Lillehagen commented on ARTEMIS-1559:
---
Just to clarify, as pointed out previously, the exhaustive memory usage occurs
on the server-side.
> Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM
> exception
> --
>
> Key: ARTEMIS-1559
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1559
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Tommy Lillehagen
>Priority: Major
> Labels: memory-leak
> Attachments: mem-leak-source.png, mem-leak.png
>
>
> Repro steps:
> * We have a broker (A) set up with SSL enabled (using Artemis v2.4 / v2.3)
> * A client (B) set up to use plain (non-TLS) communication (same version of
> Artemis)
> * Trying to establish a connection between (B) and (A) triggers multiple
> retries
> * Each message gets, from what I can tell, rejected quickly by (A), but each
> iteration leaks heap memory
> * Due to the number of retries in a short amount a time (~1000s from what I
> can tell) causes the above to grow the heap by 470M or so within less than 10
> seconds (the set timeout)
> * This consequently results in an out of memory exception
> The above behaviour is observed in both version 2.3 and 2.4.
> We've tested older versions (2.1 and 2.2), and neither of those manifest the
> same problem.
> I've run some profiling on (A), and {{NettyAcceptor.initChannel}}
> ({{getSslHandler()}}) seems to be a critical point (will include screenshots).
> That being said, most of the accumulated heap memory seems to be claimable
> and is mostly collected during the next GC cycle in the tests that I've run.
> Source: https://github.com/corda/corda/pull/2252
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1559) Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM exception
[
https://issues.apache.org/jira/browse/ARTEMIS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386488#comment-16386488
]
Tommy Lillehagen commented on ARTEMIS-1559:
---
Hi again [~jbertram],
I've pushed a separate, smaller example reproducing the issue described above.
The culprit seems to be how we set {{reconnectAttempts}} to {{-1}} (unlimited)
on the {{ServerLocator}}.
You can find the project and the basic steps to reproduce here:
[https://github.com/tlil/scratchpad/tree/tlil/ARTEMIS-1559/artemis-oom-repro]
Let me know if anything is unclear.
Cheers,
Tommy
> Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM
> exception
> --
>
> Key: ARTEMIS-1559
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1559
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Tommy Lillehagen
>Priority: Major
> Labels: memory-leak
> Attachments: mem-leak-source.png, mem-leak.png
>
>
> Repro steps:
> * We have a broker (A) set up with SSL enabled (using Artemis v2.4 / v2.3)
> * A client (B) set up to use plain (non-TLS) communication (same version of
> Artemis)
> * Trying to establish a connection between (B) and (A) triggers multiple
> retries
> * Each message gets, from what I can tell, rejected quickly by (A), but each
> iteration leaks heap memory
> * Due to the number of retries in a short amount a time (~1000s from what I
> can tell) causes the above to grow the heap by 470M or so within less than 10
> seconds (the set timeout)
> * This consequently results in an out of memory exception
> The above behaviour is observed in both version 2.3 and 2.4.
> We've tested older versions (2.1 and 2.2), and neither of those manifest the
> same problem.
> I've run some profiling on (A), and {{NettyAcceptor.initChannel}}
> ({{getSslHandler()}}) seems to be a critical point (will include screenshots).
> That being said, most of the accumulated heap memory seems to be claimable
> and is mostly collected during the next GC cycle in the tests that I've run.
> Source: https://github.com/corda/corda/pull/2252
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1559) Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM exception
[
https://issues.apache.org/jira/browse/ARTEMIS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383721#comment-16383721
]
Tommy Lillehagen commented on ARTEMIS-1559:
---
Hi [~jbertram],
Sure thing – won't have time this week, but will try to send something your way
next week.
Cheers,
Tommy
> Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM
> exception
> --
>
> Key: ARTEMIS-1559
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1559
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Tommy Lillehagen
>Priority: Major
> Labels: memory-leak
> Attachments: mem-leak-source.png, mem-leak.png
>
>
> Repro steps:
> * We have a broker (A) set up with SSL enabled (using Artemis v2.4 / v2.3)
> * A client (B) set up to use plain (non-TLS) communication (same version of
> Artemis)
> * Trying to establish a connection between (B) and (A) triggers multiple
> retries
> * Each message gets, from what I can tell, rejected quickly by (A), but each
> iteration leaks heap memory
> * Due to the number of retries in a short amount a time (~1000s from what I
> can tell) causes the above to grow the heap by 470M or so within less than 10
> seconds (the set timeout)
> * This consequently results in an out of memory exception
> The above behaviour is observed in both version 2.3 and 2.4.
> We've tested older versions (2.1 and 2.2), and neither of those manifest the
> same problem.
> I've run some profiling on (A), and {{NettyAcceptor.initChannel}}
> ({{getSslHandler()}}) seems to be a critical point (will include screenshots).
> That being said, most of the accumulated heap memory seems to be claimable
> and is mostly collected during the next GC cycle in the tests that I've run.
> Source: https://github.com/corda/corda/pull/2252
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1559) Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM exception
[
https://issues.apache.org/jira/browse/ARTEMIS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383670#comment-16383670
]
Justin Bertram commented on ARTEMIS-1559:
-
Instructions should probably include steps about:
* what GitHub repo to clone
* how to build Corda (which is required to run the test)
* what process to attach the profiler to (I see 3 related to Kotlin, 3 related
to Gradle, and 1 related to Corda)
* expected console output from the test (I see a
{{ActiveMQConnectionTimedOutException}}; is that expected?)
At the end of the day the Gradle/Kotlin/Corda layers makes diagnosing the
underlying issue fairly difficult. Ideally I'd like to have a simple Java
reproducer running against a bare Artemis broker and nothing more. I just
don't have a lot of time to spend digging into all the surrounding layers given
competing priorities. You guys obviously understand how Artemis works. Could
you just rip out the essential core client logic that leads to this condition
and slap it into a modified version of the "ssl-enabled" example?
> Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM
> exception
> --
>
> Key: ARTEMIS-1559
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1559
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Tommy Lillehagen
>Priority: Major
> Labels: memory-leak
> Attachments: mem-leak-source.png, mem-leak.png
>
>
> Repro steps:
> * We have a broker (A) set up with SSL enabled (using Artemis v2.4 / v2.3)
> * A client (B) set up to use plain (non-TLS) communication (same version of
> Artemis)
> * Trying to establish a connection between (B) and (A) triggers multiple
> retries
> * Each message gets, from what I can tell, rejected quickly by (A), but each
> iteration leaks heap memory
> * Due to the number of retries in a short amount a time (~1000s from what I
> can tell) causes the above to grow the heap by 470M or so within less than 10
> seconds (the set timeout)
> * This consequently results in an out of memory exception
> The above behaviour is observed in both version 2.3 and 2.4.
> We've tested older versions (2.1 and 2.2), and neither of those manifest the
> same problem.
> I've run some profiling on (A), and {{NettyAcceptor.initChannel}}
> ({{getSslHandler()}}) seems to be a critical point (will include screenshots).
> That being said, most of the accumulated heap memory seems to be claimable
> and is mostly collected during the next GC cycle in the tests that I've run.
> Source: https://github.com/corda/corda/pull/2252
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ARTEMIS-1559) Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM exception
[
https://issues.apache.org/jira/browse/ARTEMIS-1559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357374#comment-16357374
]
Tommy Lillehagen commented on ARTEMIS-1559:
---
Hi [~gaohoward],
I've created a simple test branch on our open source repo which hopefully will
make it easier to reproduce:
* https://github.com/corda/corda/compare/tlil/ARTEMIS-1559/artemis-oom-repro
Instructions on how to reproduce it can be found in the README file, here:
*
https://github.com/corda/corda/blob/9ed6f7652fe7b571034dd66763d1e955bf2d1d83/experimental/bugs/artemis-oom-repro-test/README.md
Thanks,
Tommy
> Repeated retries of unencrypted traffic to SSL-enabled broker causes OOM
> exception
> --
>
> Key: ARTEMIS-1559
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1559
> Project: ActiveMQ Artemis
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Tommy Lillehagen
>Priority: Major
> Labels: memory-leak
> Attachments: mem-leak-source.png, mem-leak.png
>
>
> Repro steps:
> * We have a broker (A) set up with SSL enabled (using Artemis v2.4 / v2.3)
> * A client (B) set up to use plain (non-TLS) communication (same version of
> Artemis)
> * Trying to establish a connection between (B) and (A) triggers multiple
> retries
> * Each message gets, from what I can tell, rejected quickly by (A), but each
> iteration leaks heap memory
> * Due to the number of retries in a short amount a time (~1000s from what I
> can tell) causes the above to grow the heap by 470M or so within less than 10
> seconds (the set timeout)
> * This consequently results in an out of memory exception
> The above behaviour is observed in both version 2.3 and 2.4.
> We've tested older versions (2.1 and 2.2), and neither of those manifest the
> same problem.
> I've run some profiling on (A), and {{NettyAcceptor.initChannel}}
> ({{getSslHandler()}}) seems to be a critical point (will include screenshots).
> That being said, most of the accumulated heap memory seems to be claimable
> and is mostly collected during the next GC cycle in the tests that I've run.
> Source: https://github.com/corda/corda/pull/2252
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
