[
https://issues.apache.org/jira/browse/AMQ-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16685119#comment-16685119
]
Veaceslav Doina commented on AMQ-5238:
--------------------------------------
Hello,
Tried to configure HA using JDBC and Amazon EFS for scheduled messages.
Today it worked fine. On the restart, new master also acquire lock on shared
storage for scheduled messages:
----
[root@broker-1 centos]# cat /opt/activemq/data/activemq.log
2018-11-13 12:10:18,185 | INFO | Attempting to acquire the exclusive lock to
become the Master broker | org.apache.activemq.store.jdbc.DefaultDatabaseLocker
| main
2018-11-13 12:10:18,198 | INFO | Becoming the master on dataSource:
org.apache.commons.dbcp2.BasicDataSource@4a8355dd |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:10:18,200 | INFO |
PListStore:[/activemq-data/broker-1/tmp_storage] started |
org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2018-11-13 12:10:18,358 | INFO | Scheduler Store version 1 loaded |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerKahaDBMetaData | main
2018-11-13 12:10:18,469 | INFO | Recovering from the scheduled job journal
@3:2315632 | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl |
main
2018-11-13 12:10:18,474 | INFO | Recovery replayed 1 operations from the
journal in 0.005 seconds. |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | main
2018-11-13 12:10:18,520 | INFO | JobSchedulerStore: /efs/scheduler started. |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | main
2018-11-13 12:10:18,597 | INFO | Apache ActiveMQ 5.15.7 (broker-1,
ID:broker-1-38641-1542111018366-1:1) is starting |
org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:18,689 | INFO | Listening for connections at:
tcp://broker-1:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-13 12:10:18,691 | INFO | Connector openwire started |
org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,694 | INFO | Listening for connections at:
amqp://broker-1:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-13 12:10:18,696 | INFO | Connector amqp started |
org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,699 | INFO | Listening for connections at:
stomp://broker-1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-13 12:10:18,701 | INFO | Connector stomp started |
org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,704 | INFO | Listening for connections at:
mqtt://broker-1:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-13 12:10:18,705 | INFO | Connector mqtt started |
org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,712 | INFO | Starting Jetty server |
org.apache.activemq.transport.WebTransportServerSupport | main
2018-11-13 12:10:18,742 | INFO | Creating Jetty connector |
org.apache.activemq.transport.WebTransportServerSupport | main
2018-11-13 12:10:18,810 | WARN |
ServletContext@o.e.j.s.ServletContextHandler@4a9f80d3\{/,null,STARTING} has
uncovered http methods for path: / | org.eclipse.jetty.security.SecurityHandler
| main
2018-11-13 12:10:18,845 | INFO | Listening for connections at
ws://broker-1:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600 |
org.apache.activemq.transport.ws.WSTransportServer | main
2018-11-13 12:10:18,850 | INFO | Connector ws started |
org.apache.activemq.broker.TransportConnector | main
2018-11-13 12:10:18,851 | INFO | Apache ActiveMQ 5.15.7 (broker-1,
ID:broker-1-38641-1542111018366-1:1) started |
org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:18,852 | INFO | For help or more information please see:
http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:18,854 | WARN | Store limit is 102400 mb (current store usage
is 0 mb). The data directory: / only has 13135 mb of usable space. - resetting
to maximum available disk space: 13135 mb |
org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:18,855 | WARN | Temporary Store limit is 51200 mb (current
store usage is 0 mb). The data directory: / only has 13135 mb of usable space.
- resetting to maximum available disk space: 13135 mb |
org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:19,338 | INFO | No Spring WebApplicationInitializer types
detected on classpath | /admin | main
2018-11-13 12:10:19,568 | INFO | ActiveMQ WebConsole available at
http://0.0.0.0:8161/ | org.apache.activemq.web.WebConsoleStarter | main
2018-11-13 12:10:19,568 | INFO | ActiveMQ Jolokia REST API available at
http://0.0.0.0:8161/api/jolokia/ | org.apache.activemq.web.WebConsoleStarter |
main
2018-11-13 12:10:19,617 | INFO | Initializing Spring FrameworkServlet
'dispatcher' | /admin | main
2018-11-13 12:10:19,831 | INFO | No Spring WebApplicationInitializer types
detected on classpath | /api | main
2018-11-13 12:10:19,906 | INFO | jolokia-agent: Using policy access restrictor
classpath:/jolokia-access.xml | /api | main
[root@broker-2 centos]# cat /opt/activemq/data/activemq.log
2018-11-13 12:10:17,854 | INFO | Apache ActiveMQ 5.15.7 (broker-2,
ID:broker-2-41696-1542110982743-1:1) is shutting down |
org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2018-11-13 12:10:17,857 | INFO | Connector openwire stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:17,858 | INFO | socketQueue interrupted - stopping |
org.apache.activemq.transport.tcp.TcpTransportServer | ActiveMQ Transport
Server Thread Handler:
tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2018-11-13 12:10:17,859 | INFO | Could not accept connection during shutdown :
{} | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server
Thread Handler:
tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)[:1.8.0_192]
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)[:1.8.0_192]
at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)[:1.8.0_192]
at
org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:494)[activemq-client-5.15.7.jar:5.15.7]
at java.lang.Thread.run(Thread.java:748)[:1.8.0_192]
2018-11-13 12:10:17,862 | INFO | Connector amqp stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:17,863 | INFO | Connector stomp stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:17,871 | INFO | Connector mqtt stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:17,878 | INFO | Connector ws stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ ShutdownHook
2018-11-13 12:10:18,105 | INFO | JobSchedulerStore: /efs/scheduler stopped. |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | ActiveMQ
ShutdownHook
2018-11-13 12:10:18,115 | INFO |
PListStore:[/activemq-data/broker-2/tmp_storage] stopped |
org.apache.activemq.store.kahadb.plist.PListStoreImpl | ActiveMQ ShutdownHook
2018-11-13 12:10:18,123 | INFO | Apache ActiveMQ 5.15.7 (broker-2,
ID:broker-2-41696-1542110982743-1:1) uptime 39.918 seconds |
org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2018-11-13 12:10:18,124 | INFO | Apache ActiveMQ 5.15.7 (broker-2,
ID:broker-2-41696-1542110982743-1:1) is shutdown |
org.apache.activemq.broker.BrokerService | ActiveMQ ShutdownHook
2018-11-13 12:10:18,124 | INFO | Closing
org.apache.activemq.xbean.XBeanBrokerFactory$1@1a968a59: startup date [Tue Nov
13 12:09:37 UTC 2018]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | ActiveMQ ShutdownHook
2018-11-13 12:10:18,131 | INFO | Destroying Spring FrameworkServlet
'dispatcher' | /admin | ActiveMQ ShutdownHook
2018-11-13 12:10:19,693 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@1a968a59: startup date [Tue Nov
13 12:10:19 UTC 2018]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2018-11-13 12:10:20,687 | INFO | JobScheduler using directory: /efs/scheduler |
org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:20,712 | INFO | Using Persistence Adapter:
JDBCPersistenceAdapter(org.apache.commons.dbcp2.BasicDataSource@4a8355dd) |
org.apache.activemq.broker.BrokerService | main
2018-11-13 12:10:21,287 | INFO | Database adapter driver override not found for
: [mysql_connector_j]. Will use default implementation. |
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-13 12:11:12,437 | INFO | Database lock driver override not found for :
[mysql_connector_j]. Will use default implementation. |
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-13 12:11:12,439 | INFO | Attempting to acquire the exclusive lock to
become the Master broker | org.apache.activemq.store.jdbc.DefaultDatabaseLocker
| main
2018-11-13 12:12:03,469 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:13:04,485 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:14:05,500 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:15:06,514 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:16:07,530 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-13 12:17:08,544 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
----
Yesterday have some strange behavior with such configuration, when both brokers
tried to get lock without success and as a resul both dead:
----
[root@broker-1 centos]# cat /opt/activemq/data/activemq.log
2018-11-12 16:40:50,550 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@1a968a59: startup date [Mon Nov
12 16:40:50 UTC 2018]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2018-11-12 16:40:51,641 | INFO | JobScheduler using directory: /efs/scheduler |
org.apache.activemq.broker.BrokerService | main
2018-11-12 16:40:51,664 | INFO | Using Persistence Adapter:
JDBCPersistenceAdapter(org.apache.commons.dbcp2.BasicDataSource@4a8355dd) |
org.apache.activemq.broker.BrokerService | main
2018-11-12 16:40:52,374 | INFO | Database adapter driver override not found for
: [mysql_connector_j]. Will use default implementation. |
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-12 16:41:43,495 | INFO | Database lock driver override not found for :
[mysql_connector_j]. Will use default implementation. |
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-12 16:41:43,497 | INFO | Attempting to acquire the exclusive lock to
become the Master broker | org.apache.activemq.store.jdbc.DefaultDatabaseLocker
| main
2018-11-12 16:42:34,522 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:43:35,535 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:44:36,549 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:45:37,562 | INFO | Failed to acquire lock. Sleeping for 10000
milli(s) before trying again... |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
[root@broker-2 opt]# cat /opt/activemq/data/activemq.log
2018-11-12 16:40:02,638 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@1a968a59: startup date [Mon Nov
12 16:40:02 UTC 2018]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2018-11-12 16:40:03,637 | INFO | JobScheduler using directory: /efs/scheduler |
org.apache.activemq.broker.BrokerService | main
2018-11-12 16:40:03,657 | INFO | Using Persistence Adapter:
JDBCPersistenceAdapter(org.apache.commons.dbcp2.BasicDataSource@4a8355dd) |
org.apache.activemq.broker.BrokerService | main
2018-11-12 16:40:04,299 | INFO | Database adapter driver override not found for
: [mysql_connector_j]. Will use default implementation. |
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-12 16:40:04,414 | INFO | Database lock driver override not found for :
[mysql_connector_j]. Will use default implementation. |
org.apache.activemq.store.jdbc.JDBCPersistenceAdapter | main
2018-11-12 16:40:04,416 | INFO | Attempting to acquire the exclusive lock to
become the Master broker | org.apache.activemq.store.jdbc.DefaultDatabaseLocker
| main
2018-11-12 16:40:04,433 | INFO | Becoming the master on dataSource:
org.apache.commons.dbcp2.BasicDataSource@4a8355dd |
org.apache.activemq.store.jdbc.DefaultDatabaseLocker | main
2018-11-12 16:40:04,436 | INFO |
PListStore:[/activemq-data/broker-2/tmp_storage] started |
org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2018-11-12 16:40:04,450 | INFO | Database /efs/scheduler/lock is locked by
another server. This broker is now in slave mode waiting a lock to be acquired
| org.apache.activemq.store.SharedFileLocker | main
----
> Add JDBC Persistence for Scheduler
> ----------------------------------
>
> Key: AMQ-5238
> URL: https://issues.apache.org/jira/browse/AMQ-5238
> Project: ActiveMQ
> Issue Type: New Feature
> Components: Job Scheduler
> Affects Versions: 5.9.0
> Environment: JBoss A-MQ 6.0
> Reporter: Jason Shepherd
> Priority: Minor
>
> AMQ-3024 added support for non-KahaDB persistence of scheduled jobs, but a
> JDBC alternative has yet to be added.
> If we're using JDBC persistence adapter, we want all persistence for the
> broker to be done for JDBC. Ideally this would also allows us to view
> historical statistics of the scheduler as per AMQ-5192.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
