[
https://issues.apache.org/jira/browse/AMQ-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573540#comment-17573540
]
DannyChan commented on AMQ-9013:
--------------------------------
[~jbonofre]
I updated the configuration to the description， thanks:D
<persistenceAdapter>
<jdbcPersistenceAdapter dataDirectory="${activemq.data}"
dataSource="#oracle-ds" lockKeepAlivePeriod="3000">
<locker>
<lease-database-locker lockAcquireSleepInterval="4000"/>
</locker>
</jdbcPersistenceAdapter>
</persistenceAdapter>
> ERROR | XXXX, no longer able to keep the exclusive lock so giving up being a
> master
> ------------------------------------------------------------------------------------
>
> Key: AMQ-9013
> URL: https://issues.apache.org/jira/browse/AMQ-9013
> Project: ActiveMQ
> Issue Type: Bug
> Components: JDBC
> Affects Versions: 5.9.0
> Reporter: DannyChan
> Priority: Major
> Attachments: image-2022-08-01-09-04-31-857.png
>
>
> In our production environment, ActiveMQ often restarts randomly. Refer to
> official documents for configuration JDBC Master Slave. Twice failover in
> half an hour. check vm and oracle healthy in during failover，please analyze
> the problem, thanks
>
> xxxx1 activemq log:
> first failover
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | xxxx1 Lease held by xxxx2
> till Sun Jul 31 16:25:03 CST 2022
> INFO | jvm 1 | 2022/07/31 16:25:01 | ERROR | xxxx1, no longer able to
> keep the exclusive lock so giving up being a master
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx1-50824-1658747295825-0:1) is shutting down
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Connector openwire stopped
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO |
> PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx1/tmp_storage]
> stopped
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx1-50824-1658747295825-0:1) uptime 11 days 13 hours
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx1-50824-1658747295825-0:1) is shutdown
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Closing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@6415d653: startup date [Wed
> Jul 20 03:15:12 CST 2022]; root of context hierarchy
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Destroying Spring
> FrameworkServlet 'dispatcher'
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Destroying hawtio
> authentication filter
> INFO | jvm 1 | 2022/07/31 16:25:03 | Restarting broker
> INFO | jvm 1 | 2022/07/31 16:25:03 | Loading message broker from:
> xbean:activemq.xml
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Refreshing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@294574e5: startup date [Sun
> Jul 31 16:25:03 CST 2022]; root of context hierarchy
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO |
> PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx1/tmp_storage]
> started
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Using Persistence Adapter:
> JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@9158362)
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | JMX consoles can connect to
> service:jmx:rmi:///jndi/rmi://x.x.x.x:10616/jmxrmi
> INFO | jvm 1 | 2022/07/31 16:25:03 | INFO | Database adapter driver
> override recognized for : [oracle_jdbc_driver] - adapter: class
> org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter
> INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 attempting to acquire
> exclusive lease to become the master
> INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 Lease held by xxxx2
> till Sun Jul 31 16:25:17 CST 2022
> INFO | jvm 1 | 2022/07/31 16:25:15 | INFO | xxxx1 failed to acquire
> lease. Sleeping for 4000 milli(s) before trying again...
> second failover
> INFO | jvm 1 | 2022/07/31 16:37:16 | INFO | xxxx1 Lease held by xxxx2
> till Sun Jul 31 16:37:20 CST 2022
> INFO | jvm 1 | 2022/07/31 16:37:16 | INFO | xxxx1 failed to acquire
> lease. Sleeping for 4000 milli(s) before trying again...
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | xxxx1, becoming master with
> lease expiry Sun Jul 31 16:37:20 CST 2022 on dataSource:
> org.apache.commons.dbcp.BasicDataSource@9158362
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx01-50824-1658747295825-0:2) is starting
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Listening for connections
> at:
> tcp://xxxx01:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Connector openwire started
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx1, ID:xxxx01-50824-1658747295825-0:2) started
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | For help or more
> information please see:
> [http://activemq.apache.org|http://activemq.apache.org/]
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Welcome to hawtio 1.2-M23 :
> [http://hawt.io/] : Don't cha wish your console was hawt like me? ;)
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role: "admins" role
> principal classes: "org.apache.activemq.jaas.GroupPrincipal"
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Using file upload
> directory: /tmp/uploads
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | jolokia-agent: Using access
> restrictor classpath:/jolokia-access.xml
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | ActiveMQ WebConsole
> available at [http://localhost:8151/]
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Initializing Spring
> FrameworkServlet 'dispatcher'
> xxxx2 activemq log:
> first failover
> INFO | jvm 1 | 2022/07/31 16:24:56 | INFO | xxxx2 failed to acquire
> lease. Sleeping for 4000 milli(s) before trying again...
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | xxxx2, becoming master with
> lease expiry Sun Jul 31 16:24:59 CST 2022 on dataSource:
> org.apache.commons.dbcp.BasicDataSource@60b25aef
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx202-42432-1658258111781-0:2) is starting
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Listening for connections
> at:
> tcp://xxxx202:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Connector openwire started
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx202-42432-1658258111781-0:2) started
> INFO | jvm 1 | 2022/07/31 16:25:01 | INFO | For help or more
> information please see:
> [http://activemq.apache.org|http://activemq.apache.org/]
> INFO | jvm 1 | 2022/07/31 16:25:01 | WARN | Store limit is 102400 mb,
> whilst the data directory:
> /home/apache-activemq-5.9.0/bin/linux-x86-64/../../data only has 102006 mb of
> usable space - resetting to maximum available disk space: 102006 mb
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Welcome to hawtio 1.2-M23 :
> [http://hawt.io/] : Don't cha wish your console was hawt like me? ;)
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Starting hawtio
> authentication filter, JAAS realm: "activemq" authorized role: "admins" role
> principal classes: "org.apache.activemq.jaas.GroupPrincipal"
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Using file upload
> directory: /tmp/uploads
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | jolokia-agent: Using access
> restrictor classpath:/jolokia-access.xml
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | ActiveMQ WebConsole
> available at [http://localhost:8151/]
> INFO | jvm 1 | 2022/07/31 16:25:02 | INFO | Initializing Spring
> FrameworkServlet 'dispatcher'
> second failover
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | xxxx2 Lease held by xxxx1
> till Sun Jul 31 16:37:24 CST 2022
> INFO | jvm 1 | 2022/07/31 16:37:20 | ERROR | xxxx2, no longer able to
> keep the exclusive lock so giving up being a master
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx02-42432-1658258111781-0:2) is shutting down
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Connector openwire stopped
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO |
> PListStore:[/home/xxxx/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx2/tmp_storage]
> stopped
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx02-42432-1658258111781-0:2) uptime 5 days 21 hours
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Apache ActiveMQ 5.9.0
> (xxxx2, ID:xxxx02-42432-1658258111781-0:2) is shutdown
> INFO | jvm 1 | 2022/07/31 16:37:20 | INFO | Closing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@6735523a: startup date [Mon
> Jul 25 19:08:17 CST 2022]; root of context hierarchy
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Destroying Spring
> FrameworkServlet 'dispatcher'
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Destroying hawtio
> authentication filter
> INFO | jvm 1 | 2022/07/31 16:37:21 | Restarting broker
> INFO | jvm 1 | 2022/07/31 16:37:21 | Loading message broker from:
> xbean:activemq.xml
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Refreshing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@4adb323: startup date [Sun Jul
> 31 16:37:21 CST 2022]; root of context hierarchy
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO |
> PListStore:[/home/apache-activemq-5.9.0/bin/linux-x86-64/../../data/xxxx2/tmp_storage]
> started
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Using Persistence Adapter:
> JDBCPersistenceAdapter(org.apache.commons.dbcp.BasicDataSource@1da39517)
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | JMX consoles can connect to
> service:jmx:rmi:///jndi/rmi://x.x.x.x:10616/jmxrmi
> INFO | jvm 1 | 2022/07/31 16:37:21 | INFO | Database adapter driver
> override recognized for : [oracle_jdbc_driver] - adapter: class
> org.apache.activemq.store.jdbc.adapter.OracleJDBCAdapter
> INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 attempting to acquire
> exclusive lease to become the master
> INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 Lease held by xxxx1
> till Sun Jul 31 16:37:27 CST 2022
> INFO | jvm 1 | 2022/07/31 16:37:26 | INFO | xxxx2 failed to acquire
> lease. Sleeping for 4000 milli(s) before trying again...
> cpu & memory usage:
> !image-2022-08-01-09-04-31-857.png!
> network:
> no dropped no errors
>
> cat activemq.xml
> <!-- START SNIPPET: example -->
> <beans
> xmlns="http://www.springframework.org/schema/beans";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> [http://www.springframework.org/schema/beans/spring-beans.xsd]
> [http://activemq.apache.org/schema/core]
> [http://activemq.apache.org/schema/core/activemq-core.xsd]";>
> <!-- Allows us to use system properties as variables in this
> configuration file -->
> <bean
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> <property name="locations">
>
> <value>[file:$|file://%24/]{activemq.conf}/credentials.properties</value>
> </property>
> </bean>
> <!-- Allows accessing the server log -->
> <bean id="logQuery" class="org.fusesource.insight.log.log4j.Log4jLogQuery"
> lazy-init="false" scope="singleton"
> init-method="start" destroy-method="stop">
> </bean>
> <!--
> The <broker> element is used to configure the ActiveMQ broker.
> -->
> <broker xmlns="http://activemq.apache.org/schema/core";
> brokerName="xxxx01" dataDirectory="${activemq.data}">
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" >
> <!-- The constantPendingMessageLimitStrategy is used to
> prevent
> slow topic consumers to block producers and affect
> other consumers
> by limiting the number of messages that are retained
> For more information, see:
>
> [http://activemq.apache.org/slow-consumer-handling.html]
> -->
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="1000"/>
> </pendingMessageLimitStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
> <managementContext>
> <managementContext createConnector="true" connectorPort="10616"
> connectorHost="10.x.x.x"/>
> </managementContext>
> <persistenceAdapter>
> <jdbcPersistenceAdapter dataDirectory="${activemq.data}"
> dataSource="#oracle-ds" lockKeepAlivePeriod="3000">
> <locker>
> <lease-database-locker lockAcquireSleepInterval="4000"/>
> </locker>
> </jdbcPersistenceAdapter>
> </persistenceAdapter>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="70" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
> <transportConnectors>
> <transportConnector name="openwire"
> uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600"/>
> </transportConnectors>
> <!-- destroy the spring context on shutdown to stop jetty -->
> <shutdownHooks>
> <bean xmlns="http://www.springframework.org/schema/beans";
> class="org.apache.activemq.hooks.SpringContextHook" />
> </shutdownHooks>
> </broker>
> <bean id="oracle-ds" class="org.apache.commons.dbcp.BasicDataSource"
> destroy-method="close">
> <property name="driverClassName"
> value="oracle.jdbc.driver.OracleDriver"/>
> <property name="url" value="jdbc:oracle:thin:@//x.x.x.x:1521/amqdb"/>
> <property name="username" value="xxxxx"/>
> <property name="password" value="xxxxxx"/>
> <property name="poolPreparedStatements" value="true"/>
> </bean>
> <import resource="jetty.xml"/>
> </beans>
> <!-- END SNIPPET: example -->
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
