[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2021-03-21 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305886#comment-17305886
 ] 

ASF subversion and git services commented on AMQ-6894:
--

Commit 9c64c63c7b2609d732a63ad6c3413c2712525e02 in activemq's branch 
refs/heads/activemq-5.16.x from Andrei Shakirin
[ https://gitbox.apache.org/repos/asf?p=activemq.git;h=9c64c63 ]

AMQ-6894: limit poison exception message to 1024

(cherry picked from commit d0dab2e88b44702d26dd0883c9940b97ee03fdeb)


> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2021-03-21 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305887#comment-17305887
 ] 

ASF subversion and git services commented on AMQ-6894:
--

Commit 1bf603d90f993868414ac6313c730123c0aca9c3 in activemq's branch 
refs/heads/activemq-5.15.x from Andrei Shakirin
[ https://gitbox.apache.org/repos/asf?p=activemq.git;h=1bf603d ]

AMQ-6894: limit poison exception message to 1024

(cherry picked from commit d0dab2e88b44702d26dd0883c9940b97ee03fdeb)


> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2021-03-21 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305884#comment-17305884
 ] 

ASF subversion and git services commented on AMQ-6894:
--

Commit d0dab2e88b44702d26dd0883c9940b97ee03fdeb in activemq's branch 
refs/heads/master from Andrei Shakirin
[ https://gitbox.apache.org/repos/asf?p=activemq.git;h=d0dab2e ]

AMQ-6894: limit poison exception message to 1024


> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2021-03-21 Thread ASF subversion and git services (Jira) 


[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305885#comment-17305885
 ] 

ASF subversion and git services commented on AMQ-6894:
--

Commit ea5b2d88130d96d260ca28570ed77d48cd433624 in activemq's branch 
refs/heads/master from Jean-Baptiste Onofré
[ https://gitbox.apache.org/repos/asf?p=activemq.git;h=ea5b2d8 ]

Merge pull request #274 from ashakirin/master

AMQ-6894: limit poison exception message to 1024

> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2018-02-07 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16355856#comment-16355856
 ] 

ASF GitHub Bot commented on AMQ-6894:
-

GitHub user ashakirin opened a pull request:

https://github.com/apache/activemq/pull/274

AMQ-6894: limit poison exception message to 1024

AMQ-6894: Limit poison exception message to 1024 to avoid IOException be 
encoding

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ashakirin/activemq master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq/pull/274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #274


commit d0dab2e88b44702d26dd0883c9940b97ee03fdeb
Author: Andrei Shakirin 
Date:   2018-02-07T18:28:55Z

AMQ-6894: limit poison exception message to 1024




> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2018-02-04 Thread Andrei Shakirin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351765#comment-16351765
 ] 

Andrei Shakirin commented on AMQ-6894:
--

The following client code configured with failover transport will kill the 
server after some time:

{code}
public void onMessage(Message message) {
...
 StringBuffer bigBuffer = new StringBuffer(Short.MAX_VALUE);
 for(int i = 1; i < Short.MAX_VALUE; i++) { 
   bigBuffer.append("1"); 
 }
 throw new RuntimeException(bigBuffer.toString());
}
{code}

I see two AMQ problems here:
1) Exception message have to be controlled and limited before set in 
dlqDeliveryFailureCause: some exceptions coming from thirdparty and not under 
client handler control. Following code in ActiveMQMessageConsumer.rollback() 
have to be fixed:
{code}
  ack.setPoisonCause(new Throwable("Exceeded redelivery policy limit:" + 
redeliveryPolicy
  + ", cause:" + lastMd.getRollbackCause(), lastMd.getRollbackCause()));
{code}
It is necessary to check length of lastMd.getRollbackCause().toString() and cut 
to reasonable length


2) Failover reconnection by EVERY IOException is IMO very dangerous

> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2018-02-04 Thread Andrei Shakirin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351763#comment-16351763
 ] 

Andrei Shakirin commented on AMQ-6894:
--

The following client code configured with failover transport will kill the 
server after some time:
{code}
public void onMessage(Message message) {
...
StringBuffer bigBuffer = new StringBuffer(Short.MAX_VALUE);
for(int i = 1; i < Short.MAX_VALUE; i++)

{ bigBuffer.append("1"); }
throw new RuntimeException(bigBuffer.toString());
}
{code}

I see two AMQ problems here:
1) Exception message have to be controlled and limited before set in 
dlqDeliveryFailureCause: some exceptions coming from thirdparty and not under 
client handler control. Following code in ActiveMQMessageConsumer.rollback() 
have to be fixed:
{code}
ack.setPoisonCause(new Throwable("Exceeded redelivery policy limit:" + 
redeliveryPolicy
  + ", cause:" + lastMd.getRollbackCause(), lastMd.getRollbackCause()));
{code}
It is necessary to check length of lastMd.getRollbackCause().toString() and cut 
to reasonable length

2) Failover reconnection by EVERY IOException is IMO very dangerous

> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2018-02-04 Thread Andrei Shakirin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351761#comment-16351761
 ] 

Andrei Shakirin commented on AMQ-6894:
--

The following client code configured with failover transport will kill the 
server after some time:

public void onMessage(Message message) {
...
  StringBuffer bigBuffer = new StringBuffer(Short.MAX_VALUE);
  for(int i = 1; i < Short.MAX_VALUE; i++) {
 bigBuffer.append("1");
  }
  throw new RuntimeException(bigBuffer.toString());
}
I see two AMQ problems here:
1) Exception message have to be controlled and limited before set in 
dlqDeliveryFailureCause: some exceptions coming from thirdparty and not under 
client handler control. Following code in ActiveMQMessageConsumer.rollback() 
have to be fixed:
{code}
ack.setPoisonCause(new Throwable("Exceeded redelivery 
policy limit:" + redeliveryPolicy
+ ", cause:" + lastMd.getRollbackCause(), 
lastMd.getRollbackCause()));
{code}
It is necessary to check length of lastMd.getRollbackCause().toString() and cut 
to reasonable length
2) Failover reconnection by EVERY IOException is IMO very dangerous

> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2018-02-01 Thread Andrei Shakirin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349902#comment-16349902
 ] 

Andrei Shakirin commented on AMQ-6894:
--

The following client code configured with failover transport will kill the 
server after some time:
{code}
public void onMessage(Message message) {
...
StringBuffer bigBuffer = new StringBuffer(Short.MAX_VALUE);
...
throw new RuntimeException(bigBuffer.toString());
}
{code}

I see two AMQ problems here:
1) Exception message have to be controlled and limited before set in 
dlqDeliveryFailureCause: some exceptions coming from thirdparty and not under 
client handler control
2) Failover reconnection by EVERY IOException is IMO very dangerous

> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2018-01-31 Thread Andrei Shakirin (JIRA) 

[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347076#comment-16347076
 ] 

Andrei Shakirin commented on AMQ-6894:
--

Thanks JB,

I have one more founding on client side.
There is an exception:
{code}
org.apache.activemq.transport.failover.FailoverTransportTransport 
(tcp://activemq-vm-primary:61616) failed , attempting to automatically 
reconnect: java.io.IOException: Encountered a String value that is too long to 
encode.
{code}

It could be the case that client tries to inject long binary property into JMS 
message, receives IOException and trying to reconnect. After some time, 
unlimited re-connections kills the server.

I am not 100% sure that is the cause, but perhaps it helps to your analyse

> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Attachments: activemq-part.zip
>
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AMQ-6894) Excessive number of connections by failover transport with priorityBackup

2018-01-31 Thread JIRA 

[ 
https://issues.apache.org/jira/browse/AMQ-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347015#comment-16347015
 ] 

Jean-Baptiste Onofré commented on AMQ-6894:
---

Thanks, gonna take a look.

> Excessive number of connections by failover transport with priorityBackup
> -
>
> Key: AMQ-6894
> URL: https://issues.apache.org/jira/browse/AMQ-6894
> Project: ActiveMQ
>  Issue Type: Bug
>  Components: Broker
>Affects Versions: 5.14.5
>Reporter: Andrei Shakirin
>Priority: Major
> Attachments: activemq-part.zip
>
>
> My clients connect to AMQ with this connection string:
> (tcp://amq1:61616,tcp://amq2:61616)?randomize=false=true
>  It works - for some time. But sooner or later my AMQ server becomes 
> unresponsive because the host it runs on runs out of resources (threads).
> Suddenly AMQ Server log explodes with the messages like:
> {code}
> 2018-01-26 09:26:16,909 | WARN  | Failed to register MBean 
> org.apache.activemq 
> :type=Broker,brokerName=activemq-vm-primary,connector=clientConnectors,connect
> orName=default,connectionViewType=clientId,connectionName=ID_ca8f70e115d0-3708
> 7-1516883370639-0_22 | org.apache.activemq.broker.jmx.ManagedTransportConnecti
> on | ActiveMQ Transport: tcp:///172.10.7.56:55548@61616
> 2018-01-26 09:26:21,375 | WARN  | Ignoring ack received before dispatch; 
> result of failover with an outstanding ack. Acked messages will be replayed 
> if present on this broker. Ignored ack: MessageAck \{commandId = 157, 
> responseRequired = false, ackType = 2, consumerId = 
> ID:ca8f70e115d0-37087-1516883370639-1:22:10:1, firstMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, lastMessageId = 
> ID:a95345a9c0df-33771-1516883685728-1:17:5:1:23, destination = 
> queue://MY_QUEUE_OUT, transactionId = null, messageCount = 1, poisonCause = 
> null} | org.apache.activemq.broker.region.PrefetchSubscription | ActiveMQ 
> Transport: tcp:///172.16.6.56:55464@61616
> 2018-01-26 09:26:39,211 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:55860 failed: java.net.SocketException: Connection reset | 
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> 2018-01-26 09:26:47,175 | WARN  | Transport Connection to: 
> tcp://172.10.6.56:57012 failed: java.net.SocketException: Broken pipe (Write 
> failed) | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ 
> InactivityMonitor Worker
> {code}
> After short period of time AMQ server comes out of resources with 
> "java.lang.OutOfMemoryError: unable to create new native thread" error. The 
> AMQ service process in this case has a huge number of threads (some thousands)
>  
> The client side log contains a lot of reconnection attempts messages like:
> {code}
> 2018-01-26 00:10:31,387 WARN    
> [\{{bundle.name,org.apache.activemq.activemq-osgi}{bundle.version,5.14.1}\{bundle.id,181}}]
>  [null]  org.apache.activemq.transport.failover.FailoverTransport  
> Failed to connect to [tcp://activemq-vm-primary:61616, 
> tcp://activemq-vm-secondary:61616] after: 810 attempt(s) continuing to retry.
> {code}
> It seems that client creates a huge number of connections by failover retry 
> and after some time kills the server.
> Issue looks very similar to described in 
> https://issues.apache.org/jira/browse/AMQ-6603, however server isn't 
> configured with access control settings.
> I found the description of similar problem into 
> [http://activemq.2283324.n4.nabble.com/ActiveMQ-5-2-OutOfMemoryError-unable-to-create-new-native-thread-td2366585.html],
>   but without concrete suggestion.
>  
> Part of server log is attached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)