Help: activeMQ stuck after "Could not accept connection error"

[gramonta]

Issue: under unstable network conditions and/or network outages, ActiveMQ
connection is not re-established following the error:
Could not accept connection from tcp://hostname.domain.com:portNumber

similar to
http://activemq.2283324.n4.nabble.com/ActiveMQ-Stops-Accepting-Connections-td4663395.html
but did not get any wire format exception...

Any help with recovering from this error would be much appreciated.
Not sure if it is a bug or if there are settings that can help this from
happening.

ActiveMQ versions 5.13.0 (broker) 5.13.1 (spoke)

Setup: we have servers worldwide and network connectivity is not optimal
everyday everywhere.
Network outage are not unusual, sometimes for hours, and this error requires
a restart of the broker as restarting the spoke does not help.
ActiveMQ traffic is very irregular with long periods of inactivity and
periods of high activity.

Configuration
Spoke: acts as a listener, but is the originator of the messages (no network
connector definition in its activemq.xml)
Broker: establishes connection to spokes, uses static bridge, duplex
connection.

Sample network connector:


  
  
  
  
  
  



Test steps to re-create the error seen on production servers.
In this test, there are no messages sent.
This is just one scenario, there maybe others leading to the same situation,
but following the steps below ensure the issue can be reproduced.
For instance, the same result can be achieved without step 5, but the wait
may be a lot longer as it is all a timing issue.

1. Start with an established healthy connection
2. On the spoke, using the linux network emulator, simulate an 80% packet
loss (in the real world it would require be much less, but in a test
environment with AMQ running in VMs in the same VM cluster and not to wait
forever, this does the trick).
   The command used is tc qdisc [add/replace] dev apdaterName root netem
loss 80%
3. Eventually this will cause AMQ to drop the consumer (i.e. when monitoring
the status via the AMQ console, it shows 0 consumer)
4. While there is 0 consumer, ensure that both server and client still show
a TCP established connection (netstat -nat | grep portNumber shows
ESTABLISHED)
5. Severe the network connection completely by simulating 100% packet loss
(tc qdisc replace dev apdaterName root netem loss 100%)
6. Wait -it takes anything from 2-10 minutes- until the error Could not
accept connection occurs on the spoke. At that point the TCP connection has
been dropped on the spoke.
7. Re-enable the network (tc qdisc replace dev apdaterName root netem loss
0%), but the connection is never re-established

Result:
The exception associated with "Could not accept connection" will vary per
test and can be anything from ssl handshake error, or no route to host, or
EOFException...
On the broker: there is no further attempt to reconnect and the TCP
connection remains established. ActiveMQ must be restarted to re-establish a
valid connection.

Note: if the broker happens to drop the TCP connection BEFORE the spoke,
then the connection is re-established successfully.

Using the soTimeout/soWriteTimeout keepAlive transport options on the broker
does not seem to affect the behavior
Interestingly, the debug logging socket close occurred when shutting down
AMQ in this scenario.

Log snippet:

activemq.log.66:2016-05-19 17:56:07,998 | DEBUG | No connection attempt made
in time for ssl:///172.27.11.189:5902! Throwing InactivityIOException. |
org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
InactivityMonitor ReadCheckTimer
activemq.log.68:2016-05-19 17:55:37,964 | DEBUG | No message received since
last read check for ssl:///172.27.11.189:5902. Throwing
InactivityIOException. |
org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
InactivityMonitor ReadCheckTimer
activemq.log.68:2016-05-19 17:55:37,965 | DEBUG | Running
WriteCheck[tcp://172.27.11.189:5902] |
org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
InactivityMonitor Worker
activemq.log.68:2016-05-19 17:55:37,966 | DEBUG | Running
ReadCheck[tcp://172.27.11.189:5902] |
org.apache.activemq.transport.AbstractInactivityMonitor | ActiveMQ
InactivityMonitor Worker
activemq.log.68:2016-05-19 17:55:37,968 | WARN  | Network connection between
vm://localhost#112 and ssl:///172.27.11.189:5902 shutdown due to a remote
error: org.apache.activemq.transport.InactivityIOException: Channel was
inactive for too (>3) long: tcp://172.27.11.189:5902 |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
InactivityMonitor Worker
activemq.log.68:2016-05-19 17:55:37,968 | DEBUG | The remote Exception was:
org.apache.activemq.transport.InactivityIOException: Channel was inactive
for too (>3) long: tcp://172.27.11.189:5902 |
org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
InactivityMonitor Worker
activemq.log.68:org.apache.activemq.transport.InactivityIOException: Channel
was 

[GitHub] activemq-artemis pull request: ARTEMIS-533, ARTEMIS-531, ARTEMIS-5...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/532


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] activemq-artemis pull request: Rework journal reclaimer

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/531


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Artemis Replication before acknowledgement

[Justin Bertram]

Can you supply your client test code?  Better yet, could you modify the 
replicated-failback-static example to reproduce your use-case (minus the 
simulated network delay, of course) and then push it to your GitHub repo?


Justin

- Original Message -
From: "adclayton" 
To: dev@activemq.apache.org
Sent: Monday, May 23, 2016 2:27:55 PM
Subject: Re: Artemis Replication before acknowledgement

My cluster configuration is a simple replicated-failback-static configuration
that can be found in the example from the Artemis project given below.
https://github.com/apache/activemq-artemis/tree/master/examples/features/ha/replicated-failback-static

My use case is that my cluster must have zero message loss when failing
over.  The only way to achieve this is with synchronous replication.  The
live server must not acknowledge a message as received and persisted until
it has confirmed the message has been replicated to the backup.

I setup two artemis instances.  The master on my local, and the backup in a
linux VM.  I then used linux netem to simulate a latency of several seconds
between the two machines.  I then ran a test where I pushed many hundreds of
messages onto a durable queue in a transacted session.  My test confirmed
that I was able to push the messages in a few hundred milliseconds, thus
proving that the messages were not replicated to the backup server before
the live server confirmed that they were successfully placed on the queue.  

Is this the expected behavior for this configuration?  Is there another
configuration which does the synchronous replication?  If so can you point
me to an example of this configuration?



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/Artemis-Replication-before-acknowledgement-tp4712215p4712279.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

How to inject properties in to login.config for LDAPLoginModule

[ctalluri]

I want to have the values in login.config file something like this 
LdapConfiguration { 
org.apache.activemq.jaas.LDAPLoginModule required 
debug=true 
   
initialContextFactory=com.sun.jndi.ldap.LdapCtxFactory 
connectionURL="${connectionURL}" 
connectionProtocol="" 
connectionUsername="${connectionUser}" 
connectionPassword="${connectionPwd}" 
authentication="${auth} 
}; 
(i.e) should be able to pass the values as properties like "${key-value}"
We can pass using -d option in Java. but we wanted this to be in a
propertyfile so that we can have encryption for Password 

-Thanks



--
View this message in context: 
http://activemq.2283324.n4.nabble.com/How-to-inject-properties-in-to-login-config-for-LDAPLoginModule-tp4712277.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.

[GitHub] activemq-artemis pull request: ARTEMIS-533 & ARTEMIS-531

[jbertram]

GitHub user jbertram opened a pull request:

https://github.com/apache/activemq-artemis/pull/532

ARTEMIS-533 & ARTEMIS-531



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jbertram/activemq-artemis master_work

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/532.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #532


commit 290c9c674fe3105154de9140f4649457b03c5358
Author: jbertram 
Date:   2016-05-23T15:03:14Z

ARTEMIS-533 change HTTP port from CLI

commit 87a19fa8456ca05029be1cf5ec21e8c89170fbae
Author: jbertram 
Date:   2016-05-23T15:43:46Z

ARTEMIS-531 upgrade Jolokia




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] activemq-artemis pull request: Rework journal reclaimer

[jbertram]

Github user jbertram commented on the pull request:

https://github.com/apache/activemq-artemis/pull/531#issuecomment-221057286
  
Can you provide some context and rationale for this change?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] activemq-artemis pull request: a few tiny doc typos

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/activemq-artemis/pull/530


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] activemq-artemis pull request: Rework journal reclaimer

[barreiro]

GitHub user barreiro opened a pull request:

https://github.com/apache/activemq-artemis/pull/531

Rework journal reclaimer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/barreiro/activemq-artemis reclaim

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/531.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #531


commit 4a28cc755da22fd54a65720767aeac00e61b8df3
Author: barreiro 
Date:   2016-05-19T00:51:19Z

Rework journal reclaimer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] activemq-artemis pull request: a few tiny doc typos

[pgfox]

GitHub user pgfox opened a pull request:

https://github.com/apache/activemq-artemis/pull/530

a few tiny doc typos

some minor doc typo fixes

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pgfox/activemq-artemis minor_doc_typos

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/activemq-artemis/pull/530.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #530


commit 8d482bbc0dc60d59dd911908d29f4ee838d57d83
Author: Pat Fox 
Date:   2016-05-23T15:40:42Z

a few tiny doc typos




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] activemq pull request: Fix error message typo.

[sprgn]

Github user sprgn closed the pull request at:

https://github.com/apache/activemq/pull/179


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---