[jira] [Commented] (AMQ-7147) Stale DefaultIOExceptionHandler Thread stuck in while loop

2019-02-06 Thread sunil (JIRA) 


[ 
https://issues.apache.org/jira/browse/AMQ-7147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762294#comment-16762294
 ] 

sunil commented on AMQ-7147:


[~gtully] Thanks Gary for pointing to it:

This is thing we see in logs:

 
{code:java}
2019-02-04 20:20:23,859 | INFO | waiting for broker persistence adapter 
checkpoint to succeed before restarting transports | 
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler: 
restart transports
2019-02-04 20:20:25,934 | ERROR | 
vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com, no longer able to keep the 
exclusive lock so giving up being a master | 
org.apache.activemq.broker.LockableServiceSupport | ActiveMQ JDBC PA Scheduled 
Task
2019-02-04 20:20:25,939 | INFO | Apache ActiveMQ 5.14.3 
(vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com, 
ID:vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com-45631-1549337123293-0:1) 
is shutting down | org.apache.activemq.broker.BrokerService | ActiveMQ JDBC PA 
Scheduled Task
2019-02-04 20:20:25,940 | INFO | Connector openwire stopped | 
org.apache.activemq.broker.TransportConnector | ActiveMQ JDBC PA Scheduled Task
2019-02-04 20:20:25,942 | INFO | Connector amqp stopped | 
org.apache.activemq.broker.TransportConnector | ActiveMQ JDBC PA Scheduled Task
2019-02-04 20:20:25,943 | INFO | Connector stomp stopped | 
org.apache.activemq.broker.TransportConnector | ActiveMQ JDBC PA Scheduled Task
2019-02-04 20:20:25,944 | INFO | Connector mqtt stopped | 
org.apache.activemq.broker.TransportConnector | ActiveMQ JDBC PA Scheduled Task
2019-02-04 20:20:25,945 | INFO | Connector ws stopped | 
org.apache.activemq.broker.TransportConnector | ActiveMQ JDBC PA Scheduled Task
2019-02-04 20:20:25,946 | INFO | Connector ssl stopped | 
org.apache.activemq.broker.TransportConnector | ActiveMQ JDBC PA Scheduled Task
2019-02-04 20:20:25,951 | INFO | 
PListStore:[/opt/vsd/amq/data/vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com/tmp_storage]
 stopped | org.apache.activemq.store.kahadb.plist.PListStoreImpl | ActiveMQ 
JDBC PA Scheduled Task
2019-02-04 20:20:25,956 | INFO | Apache ActiveMQ 5.14.3 
(vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com, 
ID:vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com-45631-1549337123293-0:1) 
uptime 1 hour 28 minutes | org.apache.activemq.broker.BrokerService | ActiveMQ 
JDBC PA Scheduled Task
2019-02-04 20:20:25,957 | INFO | Apache ActiveMQ 5.14.3 
(vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com, 
ID:vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com-45631-1549337123293-0:1) 
is shutdown | org.apache.activemq.broker.BrokerService | ActiveMQ JDBC PA 
Scheduled Task
2019-02-04 20:20:25,958 | INFO | Closing 
org.apache.activemq.xbean.XBeanBrokerFactory$1@6cf0e0ba: startup date [Mon Feb 
04 18:52:06 PST 2019]; root of context hierarchy | 
org.apache.activemq.xbean.XBeanBrokerFactory$1 | ActiveMQ JDBC PA Scheduled Task
2019-02-04 20:20:25,972 | INFO | Destroying Spring FrameworkServlet 
'dispatcher' | /admin | ActiveMQ JDBC PA Scheduled Task
2019-02-04 20:20:25,992 | INFO | Refreshing 
org.apache.activemq.xbean.XBeanBrokerFactory$1@69afa141: startup date [Mon Feb 
04 20:20:25 PST 2019]; root of context hierarchy | 
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2019-02-04 20:20:26,218 | INFO | Using Persistence Adapter: 
JDBCPersistenceAdapter(org.apache.commons.dbcp2.BasicDataSource@443faa85) | 
org.apache.activemq.broker.BrokerService | main
2019-02-04 20:20:26,218 | INFO | JMX consoles can connect to 
service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi | 
org.apache.activemq.broker.jmx.ManagementContext | JMX connector
2019-02-04 20:20:26,234 | INFO | Database adapter driver override recognized 
for : [mysql-ab_jdbc_driver] - adapter: class 
org.apache.activemq.store.jdbc.adapter.MySqlJDBCAdapter | 
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2019-02-04 20:20:26,235 | INFO | 
vsd-kanika-vm-mvdclnx38-210.cnaqa.eng.timetra.com attempting to acquire 
exclusive lease to become the master | 
org.apache.activemq.store.jdbc.LeaseDatabaseLocker | main
2019-02-04 20:20:28,860 | INFO | waiting for broker persistence adapter 
checkpoint to succeed before restarting transports | 
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler: 
restart transports
2019-02-04 20:20:33,868 | INFO | waiting for broker persistence adapter 
checkpoint to succeed before restarting transports | 
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler: 
restart transports
2019-02-04 20:20:38,870 | INFO | waiting for broker persistence adapter 
checkpoint to succeed before restarting transports | 
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler: 
restart transports
2019-02-04 20:20:43,872 | INFO | waiting for broker persistence adapter 
checkpoint to succeed before restarting transports | 
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler: 
restart transports

[jira] [Created] (AMQ-7147) Stale DefaultIOExceptionHandler Thread stuck in while loop

2019-02-05 Thread sunil (JIRA) 
sunil created AMQ-7147:
--

 Summary: Stale DefaultIOExceptionHandler Thread stuck in while loop
 Key: AMQ-7147
 URL: https://issues.apache.org/jira/browse/AMQ-7147
 Project: ActiveMQ
  Issue Type: Bug
Affects Versions: 5.14.3
Reporter: sunil
 Attachments: activemq.log

Hi,

We are currently using activemq version 5.14.3 with Master/Slave/Slave 
architecture with JDBC Persistence adapter , LeaseDatabaseLocker and 
IoException handler.
Below are the configuration:

Persistence Adapter:
{code:java}

 
 
 
 
 
 {code}
IoException Handler:
{code:java}

 
 
 
{code}

We had a network patition event on the master node and after it was removed 
from the partition eventough it stopped the transports and stopped the broker , 
it failed to exit the JVM .
But the new broker instance came up and it entered in to slave mode which is 
good. But when we looked at the logs we saw that every 5 secs:
2019-02-05 16:29:11,098 | INFO | waiting for broker persistence adapter 
checkpoint to succeed before restarting transports | 
org.apache.activemq.util.DefaultIOExceptionHandler | IOExceptionHandler: 
restart transports


When we took the thread dump to see what the ioExceptionHandler thread is hung 
at :
{code:java}
"IOExceptionHandler: restart transports@6055" daemon prio=5 tid=0x1d9 nid=NA 
sleeping
 java.lang.Thread.State: TIMED_WAITING
 at java.lang.Thread.sleep(Unknown Source:-1)
 at java.lang.Thread.sleep(Unknown Source:-1)
 at java.util.concurrent.TimeUnit.sleep(Unknown Source:-1)
 at 
org.apache.activemq.util.DefaultIOExceptionHandler$1$1.run(DefaultIOExceptionHandler.java:111){code}
It is stuck in the while loop:
{code:java}
while (hasLockOwnership() && isPersistenceAdapterDown()) {
 LOG.info("waiting for broker persistence adapter checkpoint to succeed before 
restarting transports");
 TimeUnit.MILLISECONDS.sleep(resumeCheckSleepPeriod);
 }{code}
The broker persistenceAdapter value is null and it fails to exit the while loop 
and the thread is hung there. 
What is the scenario when the persistence Adapter is set to null? and is it bug 
in a code since there is a stale thread running and also reference to the old 
brokerInstance which has the persistence Adapter set to null.

Also can anyone clarify why the systemExitOnShutdown didn't kill the JVM when 
the stop on broker was initiated due to ExceptionHandler code.

Any Help will be appreciated.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)