[
https://issues.apache.org/jira/browse/AMQ-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
DannyChan updated AMQ-9013:
---------------------------
Description:
In our production environment, ActiveMQ often restarts randomly. Refer to
official documents for configuration JDBC Master Slave. Twice failover in half
an hour. check vm and oracle healthy in during failover，please analyze the
problem, thanks
xxxx1 activemq log:
first failover
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | xxxx1 Lease held by xxxx2
till Sun Jul 31 16:25:03 CST 2022
INFO | jvm 1 | 2022/07/31 16:25:01 | ERROR | xxxx1, no longer able to keep
the exclusive lock so giving up being a master
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx1-50824-1658747295825-0:1) is shutting down
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Connector openwire stopped
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO |
PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx1/tmp_storage]
stopped
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx1-50824-1658747295825-0:1) uptime 11 days 13 hours
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx1-50824-1658747295825-0:1) is shutdown
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Closing
org.apache.activemq.xbean.XBeanBrokerFactory$1@6415d653: startup date [Wed Jul
20 03:15:12 CST 2022]; root of context hierarchy
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Destroying Spring
FrameworkServlet 'dispatcher'
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Destroying hawtio
authentication filter
INFO | jvm 1 | 2022/07/31 16:25:03 | Restarting broker
INFO | jvm 1 | 2022/07/31 16:25:03 | Loading message broker from:
xbean:activemq.xml
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@294574e5: startup date [Sun Jul
31 16:25:03 CST 2022]; root of context hierarchy
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO |
PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx1/tmp_storage]
started
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Using Persistence Adapter:
JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@9158362)
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://x.x.x.x:10616/jmxrmi
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Database adapter driver
override recognized for : [oracle_jdbc_driver] - adapter: class
org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter
INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 attempting to acquire
exclusive lease to become the master
INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 Lease held by xxxx2
till Sun Jul 31 16:25:17 CST 2022
INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 failed to acquire
lease. Sleeping for 4000 milli(s) before trying again...
second failover
INFO | jvm 1 | 2022/07/31 16:37:16 | INFO | xxxx1 Lease held by xxxx2
till Sun Jul 31 16:37:20 CST 2022
INFO | jvm 1 | 2022/07/31 16:37:16 | INFO | xxxx1 failed to acquire
lease. Sleeping for 4000 milli(s) before trying again...
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | xxxx1, becoming master with
lease expiry Sun Jul 31 16:37:20 CST 2022 on dataSource:
org.apache.commons.dbcp.BasicDataSource@9158362
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx01-50824-1658747295825-0:2) is starting
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Listening for connections at:
tcp://xxxx01:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Connector openwire started
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx01-50824-1658747295825-0:2) started
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | For help or more information
please see: [http://activemq.apache.org|http://activemq.apache.org/]
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Welcome to hawtio 1.2-M23 :
[http://hawt.io/] : Don't cha wish your console was hawt like me? ;)
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Starting hawtio
authentication filter, JAAS realm: "activemq" authorized role: "admins" role
principal classes: "org.apache.activemq.jaas.GroupPrincipal"
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Using file upload directory:
/tmp/uploads
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | jolokia-agent: Using access
restrictor classpath:/jolokia-access.xml
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | ActiveMQ WebConsole available
at [http://localhost:8151/]
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Initializing Spring
FrameworkServlet 'dispatcher'
xxxx2 activemq log:
first failover
INFO | jvm 1 | 2022/07/31 16:24:56 | INFO | xxxx2 failed to acquire
lease. Sleeping for 4000 milli(s) before trying again...
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | xxxx2, becoming master with
lease expiry Sun Jul 31 16:24:59 CST 2022 on dataSource:
org.apache.commons.dbcp.BasicDataSource@60b25aef
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx202-42432-1658258111781-0:2) is starting
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Listening for connections at:
tcp://xxxx202:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Connector openwire started
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx202-42432-1658258111781-0:2) started
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | For help or more information
please see: [http://activemq.apache.org|http://activemq.apache.org/]
INFO | jvm 1 | 2022/07/31 16:25:01 | WARN | Store limit is 102400 mb,
whilst the data directory:
/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data only has 102006 mb of
usable space - resetting to maximum available disk space: 102006 mb
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Welcome to hawtio 1.2-M23 :
[http://hawt.io/] : Don't cha wish your console was hawt like me? ;)
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Starting hawtio
authentication filter, JAAS realm: "activemq" authorized role: "admins" role
principal classes: "org.apache.activemq.jaas.GroupPrincipal"
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Using file upload directory:
/tmp/uploads
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | jolokia-agent: Using access
restrictor classpath:/jolokia-access.xml
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | ActiveMQ WebConsole available
at [http://localhost:8151/]
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Initializing Spring
FrameworkServlet 'dispatcher'
second failover
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | xxxx2 Lease held by xxxx1
till Sun Jul 31 16:37:24 CST 2022
INFO | jvm 1 | 2022/07/31 16:37:20 | ERROR | xxxx2, no longer able to keep
the exclusive lock so giving up being a master
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx02-42432-1658258111781-0:2) is shutting down
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Connector openwire stopped
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO |
PListStore:[/home/xxxx/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx2/tmp_storage]
stopped
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx02-42432-1658258111781-0:2) uptime 5 days 21 hours
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx02-42432-1658258111781-0:2) is shutdown
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Closing
org.apache.activemq.xbean.XBeanBrokerFactory$1@6735523a: startup date [Mon Jul
25 19:08:17 CST 2022]; root of context hierarchy
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Destroying Spring
FrameworkServlet 'dispatcher'
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Destroying hawtio
authentication filter
INFO | jvm 1 | 2022/07/31 16:37:21 | Restarting broker
INFO | jvm 1 | 2022/07/31 16:37:21 | Loading message broker from:
xbean:activemq.xml
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@4adb323: startup date [Sun Jul
31 16:37:21 CST 2022]; root of context hierarchy
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO |
PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx2/tmp_storage]
started
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Using Persistence Adapter:
JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@1da39517)
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://x.x.x.x:10616/jmxrmi
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Database adapter driver
override recognized for : [oracle_jdbc_driver] - adapter: class
org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter
INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 attempting to acquire
exclusive lease to become the master
INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 Lease held by xxxx1
till Sun Jul 31 16:37:27 CST 2022
INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 failed to acquire
lease. Sleeping for 4000 milli(s) before trying again...
cpu & memory usage:
!image-2022-08-01-09-04-31-857.png!
network:
no dropped no errors
cat activemq.xml
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans
[http://www.springframework.org/schema/beans/spring-beans.xsd]
[http://activemq.apache.org/schema/core]
[http://activemq.apache.org/schema/core/activemq-core.xsd]";>
<!-- Allows us to use system properties as variables in this configuration
file -->
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>[file:$|file://%24/]{activemq.conf}/credentials.properties</value>
</property>
</bean>
<!-- Allows accessing the server log -->
<bean id="logQuery" class="org.fusesource.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="xxxx01"
dataDirectory="${activemq.data}">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to
prevent
slow topic consumers to block producers and affect
other consumers
by limiting the number of messages that are retained
For more information, see:
[http://activemq.apache.org/slow-consumer-handling.html]
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="true" connectorPort="10616"
connectorHost="10.x.x.x"/>
</managementContext>
<persistenceAdapter>
<jdbcPersistenceAdapter dataDirectory="${activemq.data}"
dataSource="#oracle-ds" lockKeepAlivePeriod="3000">
<locker>
<lease-database-locker lockAcquireSleepInterval="4000"/>
</locker>
</jdbcPersistenceAdapter>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans";
class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<bean id="oracle-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName"
value="oracle.jdbc.driver.OracleDriver"/>
<property name="url" value="jdbc:oracle:thin:@//x.x.x.x:1521/amqdb"/>
<property name="username" value="xxxxx"/>
<property name="password" value="xxxxxx"/>
<property name="poolPreparedStatements" value="true"/>
</bean>
<import resource="jetty.xml"/>
</beans>
<!-- END SNIPPET: example -->
was:
In our production environment, ActiveMQ often restarts randomly. Refer to
official documents for configuration JDBC Master Slave. Twice failover in half
an hour. check vm and oracle healthy in during failover，please analyze the
problem, thanks
xxxx1 activemq log:
first failover
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | xxxx1 Lease held by xxxx2
till Sun Jul 31 16:25:03 CST 2022
INFO | jvm 1 | 2022/07/31 16:25:01 | ERROR | xxxx1, no longer able to keep
the exclusive lock so giving up being a master
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx1-50824-1658747295825-0:1) is shutting down
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Connector openwire stopped
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO |
PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx1/tmp_storage]
stopped
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx1-50824-1658747295825-0:1) uptime 11 days 13 hours
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx1-50824-1658747295825-0:1) is shutdown
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Closing
org.apache.activemq.xbean.XBeanBrokerFactory$1@6415d653: startup date [Wed Jul
20 03:15:12 CST 2022]; root of context hierarchy
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Destroying Spring
FrameworkServlet 'dispatcher'
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Destroying hawtio
authentication filter
INFO | jvm 1 | 2022/07/31 16:25:03 | Restarting broker
INFO | jvm 1 | 2022/07/31 16:25:03 | Loading message broker from:
xbean:activemq.xml
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@294574e5: startup date [Sun Jul
31 16:25:03 CST 2022]; root of context hierarchy
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO |
PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx1/tmp_storage]
started
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Using Persistence Adapter:
JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@9158362)
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://x.x.x.x:10616/jmxrmi
INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Database adapter driver
override recognized for : [oracle_jdbc_driver] - adapter: class
org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter
INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 attempting to acquire
exclusive lease to become the master
INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 Lease held by xxxx2
till Sun Jul 31 16:25:17 CST 2022
INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 failed to acquire
lease. Sleeping for 4000 milli(s) before trying again...
second failover
INFO | jvm 1 | 2022/07/31 16:37:16 | INFO | xxxx1 Lease held by xxxx2
till Sun Jul 31 16:37:20 CST 2022
INFO | jvm 1 | 2022/07/31 16:37:16 | INFO | xxxx1 failed to acquire
lease. Sleeping for 4000 milli(s) before trying again...
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | xxxx1, becoming master with
lease expiry Sun Jul 31 16:37:20 CST 2022 on dataSource:
org.apache.commons.dbcp.BasicDataSource@9158362
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx01-50824-1658747295825-0:2) is starting
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Listening for connections at:
tcp://xxxx01:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Connector openwire started
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx1,
ID:xxxx01-50824-1658747295825-0:2) started
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | For help or more information
please see: http://activemq.apache.org
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Welcome to hawtio 1.2-M23 :
http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Starting hawtio
authentication filter, JAAS realm: "activemq" authorized role: "admins" role
principal classes: "org.apache.activemq.jaas.GroupPrincipal"
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Using file upload directory:
/tmp/uploads
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | jolokia-agent: Using access
restrictor classpath:/jolokia-access.xml
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | ActiveMQ WebConsole available
at http://localhost:8151/
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Initializing Spring
FrameworkServlet 'dispatcher'
xxxx2 activemq log:
first failover
INFO | jvm 1 | 2022/07/31 16:24:56 | INFO | xxxx2 failed to acquire
lease. Sleeping for 4000 milli(s) before trying again...
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | xxxx2, becoming master with
lease expiry Sun Jul 31 16:24:59 CST 2022 on dataSource:
org.apache.commons.dbcp.BasicDataSource@60b25aef
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx202-42432-1658258111781-0:2) is starting
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Listening for connections at:
tcp://xxxx202:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Connector openwire started
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx202-42432-1658258111781-0:2) started
INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | For help or more information
please see: http://activemq.apache.org
INFO | jvm 1 | 2022/07/31 16:25:01 | WARN | Store limit is 102400 mb,
whilst the data directory:
/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data only has 102006 mb of
usable space - resetting to maximum available disk space: 102006 mb
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Welcome to hawtio 1.2-M23 :
http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Starting hawtio
authentication filter, JAAS realm: "activemq" authorized role: "admins" role
principal classes: "org.apache.activemq.jaas.GroupPrincipal"
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Using file upload directory:
/tmp/uploads
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | jolokia-agent: Using access
restrictor classpath:/jolokia-access.xml
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | ActiveMQ WebConsole available
at http://localhost:8151/
INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Initializing Spring
FrameworkServlet 'dispatcher'
second failover
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | xxxx2 Lease held by xxxx1
till Sun Jul 31 16:37:24 CST 2022
INFO | jvm 1 | 2022/07/31 16:37:20 | ERROR | xxxx2, no longer able to keep
the exclusive lock so giving up being a master
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx02-42432-1658258111781-0:2) is shutting down
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Connector openwire stopped
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO |
PListStore:[/home/xxxx/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx2/tmp_storage]
stopped
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx02-42432-1658258111781-0:2) uptime 5 days 21 hours
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0 (xxxx2,
ID:xxxx02-42432-1658258111781-0:2) is shutdown
INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Closing
org.apache.activemq.xbean.XBeanBrokerFactory$1@6735523a: startup date [Mon Jul
25 19:08:17 CST 2022]; root of context hierarchy
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Destroying Spring
FrameworkServlet 'dispatcher'
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Destroying hawtio
authentication filter
INFO | jvm 1 | 2022/07/31 16:37:21 | Restarting broker
INFO | jvm 1 | 2022/07/31 16:37:21 | Loading message broker from:
xbean:activemq.xml
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@4adb323: startup date [Sun Jul
31 16:37:21 CST 2022]; root of context hierarchy
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO |
PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx2/tmp_storage]
started
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Using Persistence Adapter:
JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@1da39517)
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | JMX consoles can connect to
service:jmx:rmi:///jndi/rmi://x.x.x.x:10616/jmxrmi
INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Database adapter driver
override recognized for : [oracle_jdbc_driver] - adapter: class
org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter
INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 attempting to acquire
exclusive lease to become the master
INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 Lease held by xxxx1
till Sun Jul 31 16:37:27 CST 2022
INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 failed to acquire
lease. Sleeping for 4000 milli(s) before trying again...
cpu & memory usage:
!image-2022-08-01-09-04-31-857.png!
network:
no dropped no errors
Environment: (was: cat activemq.xml
<!-- START SNIPPET: example -->
<beans
xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd";>
<!-- Allows us to use system properties as variables in this configuration
file -->
<bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
<property name="locations">
<value>file:${activemq.conf}/credentials.properties</value>
</property>
</bean>
<!-- Allows accessing the server log -->
<bean id="logQuery" class="org.fusesource.insight.log.log4j.Log4jLogQuery"
lazy-init="false" scope="singleton"
init-method="start" destroy-method="stop">
</bean>
<!--
The <broker> element is used to configure the ActiveMQ broker.
-->
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="xxxx01"
dataDirectory="${activemq.data}">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<!-- The constantPendingMessageLimitStrategy is used to
prevent
slow topic consumers to block producers and affect
other consumers
by limiting the number of messages that are retained
For more information, see:
http://activemq.apache.org/slow-consumer-handling.html
-->
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<managementContext>
<managementContext createConnector="true" connectorPort="10616"
connectorHost="10.x.x.x"/>
</managementContext>
<persistenceAdapter>
<jdbcPersistenceAdapter dataDirectory="${activemq.data}"
dataSource="#oracle-ds" lockKeepAlivePeriod="3000">
<locker>
<lease-database-locker lockAcquireSleepInterval="4000"/>
</locker>
</jdbcPersistenceAdapter>
</persistenceAdapter>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="70" />
</memoryUsage>
<storeUsage>
<storeUsage limit="100 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="50 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>
<transportConnectors>
<transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
</transportConnectors>
<!-- destroy the spring context on shutdown to stop jetty -->
<shutdownHooks>
<bean xmlns="http://www.springframework.org/schema/beans";
class="org.apache.activemq.hooks.SpringContextHook" />
</shutdownHooks>
</broker>
<bean id="oracle-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">
<property name="driverClassName"
value="oracle.jdbc.driver.OracleDriver"/>
<property name="url" value="jdbc:oracle:thin:@//x.x.x.x:1521/amqdb"/>
<property name="username" value="xxxxx"/>
<property name="password" value="xxxxxx"/>
<property name="poolPreparedStatements" value="true"/>
</bean>
<import resource="jetty.xml"/>
</beans>
<!-- END SNIPPET: example -->
)
> ERROR | XXXX, no longer able to keep the exclusive lock so giving up being a
> master
> ------------------------------------------------------------------------------------
>
> Key: AMQ-9013
> URL: https://issues.apache.org/jira/browse/AMQ-9013
> Project: ActiveMQ
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 5.9.0
> Reporter: DannyChan
> Priority: Major
> Attachments: image-2022-08-01-09-04-31-857.png
>
>
> In our production environment, ActiveMQ often restarts randomly. Refer to
> official documents for configuration JDBC Master Slave. Twice failover in
> half an hour. check vm and oracle healthy in during failover，please analyze
> the problem, thanks
>
> xxxx1 activemq log:
> first failover
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | xxxx1 Lease held by xxxx2
> till Sun Jul 31 16:25:03 CST 2022
> INFO | jvm 1 | 2022/07/31 16:25:01 | ERROR | xxxx1, no longer able to
> keep the exclusive lock so giving up being a master
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx1-50824-1658747295825-0:1) is shutting down
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Connector openwire stopped
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO |
> PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx1/tmp_storage]
> stopped
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx1-50824-1658747295825-0:1) uptime 11 days 13 hours
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx1-50824-1658747295825-0:1) is shutdown
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Closing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@6415d653: startup date [Wed
> Jul 20 03:15:12 CST 2022]; root of context hierarchy
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Destroying Spring
> FrameworkServlet 'dispatcher'
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Destroying hawtio
> authentication filter
> INFO | jvm 1 | 2022/07/31 16:25:03 | Restarting broker
> INFO | jvm 1 | 2022/07/31 16:25:03 | Loading message broker from:
> xbean:activemq.xml
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Refreshing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@294574e5: startup date [Sun
> Jul 31 16:25:03 CST 2022]; root of context hierarchy
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO |
> PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx1/tmp_storage]
> started
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Using Persistence Adapter:
> JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@9158362)
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | JMX consoles can connect to
> service:jmx:rmi:///jndi/rmi://x.x.x.x:10616/jmxrmi
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Database adapter driver
> override recognized for : [oracle_jdbc_driver] - adapter: class
> org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter
> INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 attempting to acquire
> exclusive lease to become the master
> INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 Lease held by xxxx2
> till Sun Jul 31 16:25:17 CST 2022
> INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 failed to acquire
> lease. Sleeping for 4000 milli(s) before trying again...
> second failover
> INFO | jvm 1 | 2022/07/31 16:37:16 | INFO | xxxx1 Lease held by xxxx2
> till Sun Jul 31 16:37:20 CST 2022
> INFO | jvm 1 | 2022/07/31 16:37:16 | INFO | xxxx1 failed to acquire
> lease. Sleeping for 4000 milli(s) before trying again...
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | xxxx1, becoming master with
> lease expiry Sun Jul 31 16:37:20 CST 2022 on dataSource:
> org.apache.commons.dbcp.BasicDataSource@9158362
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx01-50824-1658747295825-0:2) is starting
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Listening for connections
> at:
> tcp://xxxx01:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Connector openwire started
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx01-50824-1658747295825-0:2) started
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | For help or more
> information please see:
> [http://activemq.apache.org|http://activemq.apache.org/]
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Welcome to hawtio 1.2-M23 :
> [http://hawt.io/] : Don't cha wish your console was hawt like me? ;)
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role: "admins" role
> principal classes: "org.apache.activemq.jaas.GroupPrincipal"
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Using file upload
> directory: /tmp/uploads
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | jolokia-agent: Using access
> restrictor classpath:/jolokia-access.xml
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | ActiveMQ WebConsole
> available at [http://localhost:8151/]
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Initializing Spring
> FrameworkServlet 'dispatcher'
> xxxx2 activemq log:
> first failover
> INFO | jvm 1 | 2022/07/31 16:24:56 | INFO | xxxx2 failed to acquire
> lease. Sleeping for 4000 milli(s) before trying again...
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | xxxx2, becoming master with
> lease expiry Sun Jul 31 16:24:59 CST 2022 on dataSource:
> org.apache.commons.dbcp.BasicDataSource@60b25aef
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx202-42432-1658258111781-0:2) is starting
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Listening for connections
> at:
> tcp://xxxx202:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Connector openwire started
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx202-42432-1658258111781-0:2) started
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | For help or more
> information please see:
> [http://activemq.apache.org|http://activemq.apache.org/]
> INFO | jvm 1 | 2022/07/31 16:25:01 | WARN | Store limit is 102400 mb,
> whilst the data directory:
> /home/apache-activemq-5.9.0/bin/linux-x86-64/../../data only has 102006 mb of
> usable space - resetting to maximum available disk space: 102006 mb
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Welcome to hawtio 1.2-M23 :
> [http://hawt.io/] : Don't cha wish your console was hawt like me? ;)
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role: "admins" role
> principal classes: "org.apache.activemq.jaas.GroupPrincipal"
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Using file upload
> directory: /tmp/uploads
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | jolokia-agent: Using access
> restrictor classpath:/jolokia-access.xml
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | ActiveMQ WebConsole
> available at [http://localhost:8151/]
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Initializing Spring
> FrameworkServlet 'dispatcher'
> second failover
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | xxxx2 Lease held by xxxx1
> till Sun Jul 31 16:37:24 CST 2022
> INFO | jvm 1 | 2022/07/31 16:37:20 | ERROR | xxxx2, no longer able to
> keep the exclusive lock so giving up being a master
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx02-42432-1658258111781-0:2) is shutting down
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Connector openwire stopped
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO |
> PListStore:[/home/xxxx/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx2/tmp_storage]
> stopped
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx02-42432-1658258111781-0:2) uptime 5 days 21 hours
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx02-42432-1658258111781-0:2) is shutdown
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Closing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@6735523a: startup date [Mon
> Jul 25 19:08:17 CST 2022]; root of context hierarchy
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Destroying Spring
> FrameworkServlet 'dispatcher'
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Destroying hawtio
> authentication filter
> INFO | jvm 1 | 2022/07/31 16:37:21 | Restarting broker
> INFO | jvm 1 | 2022/07/31 16:37:21 | Loading message broker from:
> xbean:activemq.xml
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Refreshing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@4adb323: startup date [Sun Jul
> 31 16:37:21 CST 2022]; root of context hierarchy
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO |
> PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx2/tmp_storage]
> started
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Using Persistence Adapter:
> JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@1da39517)
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | JMX consoles can connect to
> service:jmx:rmi:///jndi/rmi://x.x.x.x:10616/jmxrmi
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Database adapter driver
> override recognized for : [oracle_jdbc_driver] - adapter: class
> org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter
> INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 attempting to acquire
> exclusive lease to become the master
> INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 Lease held by xxxx1
> till Sun Jul 31 16:37:27 CST 2022
> INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 failed to acquire
> lease. Sleeping for 4000 milli(s) before trying again...
> cpu & memory usage:
> !image-2022-08-01-09-04-31-857.png!
> network:
> no dropped no errors
>
> cat activemq.xml
> <!-- START SNIPPET: example -->
> <beans
> xmlns="http://www.springframework.org/schema/beans";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> [http://www.springframework.org/schema/beans/spring-beans.xsd]
> [http://activemq.apache.org/schema/core]
> [http://activemq.apache.org/schema/core/activemq-core.xsd]";>
> <!-- Allows us to use system properties as variables in this
> configuration file -->
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
>
> <value>[file:$|file://%24/]{activemq.conf}/credentials.properties</value>
> </property>
> </bean>
> <!-- Allows accessing the server log -->
> <bean id="logQuery" class="org.fusesource.insight.log.log4j.Log4jLogQuery"
> lazy-init="false" scope="singleton"
> init-method="start" destroy-method="stop">
> </bean>
> <!--
> The <broker> element is used to configure the ActiveMQ broker.
> -->
> <broker xmlns="http://activemq.apache.org/schema/core";
> brokerName="xxxx01" dataDirectory="${activemq.data}">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" >
> <!-- The constantPendingMessageLimitStrategy is used to
> prevent
> slow topic consumers to block producers and affect
> other consumers
> by limiting the number of messages that are retained
> For more information, see:
>
> [http://activemq.apache.org/slow-consumer-handling.html]
> -->
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="1000"/>
> </pendingMessageLimitStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <managementContext>
> <managementContext createConnector="true" connectorPort="10616"
> connectorHost="10.x.x.x"/>
> </managementContext>
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="${activemq.data}"
> dataSource="#oracle-ds" lockKeepAlivePeriod="3000">
> <locker>
> <lease-database-locker lockAcquireSleepInterval="4000"/>
> </locker>
> </jdbcPersistenceAdapter>
> </persistenceAdapter>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="70" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
> </transportConnectors>
> <!-- destroy the spring context on shutdown to stop jetty -->
> <shutdownHooks>
> <bean xmlns="http://www.springframework.org/schema/beans";
> class="org.apache.activemq.hooks.SpringContextHook" />
> </shutdownHooks>
> </broker>
> <bean id="oracle-ds" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName"
> value="oracle.jdbc.driver.OracleDriver"/>
> <property name="url" value="jdbc:oracle:thin:@//x.x.x.x:1521/amqdb"/>
> <property name="username" value="xxxxx"/>
> <property name="password" value="xxxxxx"/>
> <property name="poolPreparedStatements" value="true"/>
> </bean>
> <import resource="jetty.xml"/>
> </beans>
> <!-- END SNIPPET: example -->
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
