[
https://issues.apache.org/jira/browse/AMQ-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16678048#comment-16678048
]
Veaceslav Doina edited comment on AMQ-6903 at 11/7/18 11:16 AM:
----------------------------------------------------------------
Hello,
We also trying to configure ActiveMQ 5.15.7 in a cluster using shared storage.
Setup is done on AWS.
# 2 x EC2 instances
# 1 x EFS
Initially, all worked fine. But after some period of time and master restart,
slave stopped to work.
As was mentioned by [~ChakradharDanireddy], removing data directory solve the
issue but we lost all data:
rm -rf /efs/activemq-data/kahadb/
We found a similar issue which was reported as solved:
https://issues.apache.org/jira/browse/AMQ-6441
Can someone confirm that ActiveMQ correctly works with Amazon EFS?
Such a configuration is used to achieve a fault tolerance for Scheduled mesages
using the cluster, as JDBC cluster store such messages only locally:
https://issues.apache.org/jira/browse/AMQ-5238
*Broker custom configuration:*
----
<broker xmlns="http://activemq.apache.org/schema/core"; brokerName="broker-1"
schedulerDirectory="/efs/activemq-data-sheduler" schedulerSupport="true">
<persistenceAdapter>
<kahaDB directory="/efs/activemq-data/kahadb"/>
</persistenceAdapter>
----
*Broker logs:*
----
2018-11-07 11:01:00,417 | INFO | Refreshing
org.apache.activemq.xbean.XBeanBrokerFactory$1@5442a311: startup date [Wed Nov
07 11:01:00 UTC 2018]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2018-11-07 11:01:01,447 | INFO | Using Persistence Adapter:
KahaDBPersistenceAdapter[/efs/activemq-data/kahadb] |
org.apache.activemq.broker.BrokerService | main
2018-11-07 11:01:01,578 | INFO | KahaDB is version 6 |
org.apache.activemq.store.kahadb.MessageDatabase | main
2018-11-07 11:01:01,609 | INFO |
PListStore:[/activemq-data/broker-1/tmp_storage] started |
org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2018-11-07 11:01:01,663 | INFO | Scheduler Store version 1 loaded |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerKahaDBMetaData | main
2018-11-07 11:01:01,768 | INFO | Recovering from the scheduled job journal
@1:9820 | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl |
main
2018-11-07 11:01:01,770 | INFO | Recovery replayed 1 operations from the
journal in 0.002 seconds. |
org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl | main
2018-11-07 11:01:01,771 | INFO | JobSchedulerStore: /efs/activemq-data-sheduler
started. | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl |
main
2018-11-07 11:01:01,850 | INFO | Apache ActiveMQ 5.15.7 (broker-1,
ID:broker-1-43211-1541588461667-1:1) is starting |
org.apache.activemq.broker.BrokerService | main
2018-11-07 11:01:01,922 | INFO | Listening for connections at:
tcp://broker-1:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-07 11:01:01,924 | INFO | Connector openwire started |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:01,928 | INFO | Listening for connections at:
amqp://broker-1:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-07 11:01:01,930 | INFO | Connector amqp started |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:01,934 | INFO | Listening for connections at:
stomp://broker-1:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-07 11:01:01,939 | INFO | Connector stomp started |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:01,942 | INFO | Listening for connections at:
mqtt://broker-1:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600
| org.apache.activemq.transport.TransportServerThreadSupport | main
2018-11-07 11:01:01,944 | INFO | Connector mqtt started |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:01,950 | INFO | Starting Jetty server |
org.apache.activemq.transport.WebTransportServerSupport | main
2018-11-07 11:01:01,983 | INFO | Creating Jetty connector |
org.apache.activemq.transport.WebTransportServerSupport | main
2018-11-07 11:01:02,065 | WARN |
[email protected]@4b40f651\{/,null,STARTING} has
uncovered http methods for path: / | org.eclipse.jetty.security.SecurityHandler
| main
2018-11-07 11:01:02,107 | INFO | Listening for connections at
ws://broker-1:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600 |
org.apache.activemq.transport.ws.WSTransportServer | main
2018-11-07 11:01:02,108 | INFO | Connector ws started |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:02,109 | INFO | Apache ActiveMQ 5.15.7 (broker-1,
ID:broker-1-43211-1541588461667-1:1) started |
org.apache.activemq.broker.BrokerService | main
2018-11-07 11:01:02,110 | INFO | For help or more information please see:
http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2018-11-07 11:01:02,113 | ERROR | File system space reported by:
/efs/activemq-data/kahadb was negative, possibly a huge file system, set a sane
usage.total to provide some guidance | org.apache.activemq.broker.BrokerService
| main
2018-11-07 11:01:02,115 | ERROR | Failed to start Apache ActiveMQ (broker-1,
ID:broker-1-43211-1541588461667-1:1) | org.apache.activemq.broker.BrokerService
| main
org.apache.activemq.ConfigurationException: File system space reported by:
/efs/activemq-data/kahadb was negative, possibly a huge file system, set a sane
usage.total to provide some guidance
at
org.apache.activemq.broker.BrokerService.checkUsageLimit(BrokerService.java:2092)[activemq-broker-5.15.7.jar:5.15.7]
at
org.apache.activemq.broker.BrokerService.checkStoreUsageLimits(BrokerService.java:2029)[activemq-broker-5.15.7.jar:5.15.7]
at
org.apache.activemq.broker.BrokerService.checkStoreSystemUsageLimits(BrokerService.java:2202)[activemq-broker-5.15.7.jar:5.15.7]
at
org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:777)[activemq-broker-5.15.7.jar:5.15.7]
at
org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:733)[activemq-broker-5.15.7.jar:5.15.7]
at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:636)[activemq-broker-5.15.7.jar:5.15.7]
at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.7.jar:5.15.7]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_192]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_192]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_192]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_192]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1763)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1700)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1630)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:312)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:308)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)[spring-beans-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)[spring-context-4.3.18.RELEASE.jar:4.3.18.RELEASE]
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)[xbean-spring-4.2.jar:4.2]
at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)[xbean-spring-4.2.jar:4.2]
at
org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)[activemq-spring-5.15.7.jar:5.15.7]
at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)[activemq-spring-5.15.7.jar:5.15.7]
at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)[activemq-spring-5.15.7.jar:5.15.7]
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)[activemq-broker-5.15.7.jar:5.15.7]
at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)[activemq-broker-5.15.7.jar:5.15.7]
at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)[activemq-console-5.15.7.jar:5.15.7]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.7.jar:5.15.7]
at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.7.jar:5.15.7]
at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.7.jar:5.15.7]
at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.7.jar:5.15.7]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_192]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_192]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_192]
at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_192]
at
org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.7]
at org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.7]
2018-11-07 11:01:02,120 | INFO | Apache ActiveMQ 5.15.7 (broker-1,
ID:broker-1-43211-1541588461667-1:1) is shutting down |
org.apache.activemq.broker.BrokerService | main
2018-11-07 11:01:02,122 | INFO | Connector openwire stopped |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:02,124 | INFO | Connector amqp stopped |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:02,125 | INFO | Connector stomp stopped |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:02,127 | INFO | Connector mqtt stopped |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:02,142 | INFO | Connector ws stopped |
org.apache.activemq.broker.TransportConnector | main
2018-11-07 11:01:02,307 | INFO | JobSchedulerStore: /efs/activemq-data-sheduler
stopped. | org.apache.activemq.store.kahadb.scheduler.JobSchedulerStoreImpl |
main
2018-11-07 11:01:02,314 | INFO |
PListStore:[/activemq-data/broker-1/tmp_storage] stopped |
org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2018-11-07 11:01:02,314 | INFO | Stopping async queue tasks |
org.apache.activemq.store.kahadb.KahaDBStore | main
2018-11-07 11:01:02,315 | INFO | Stopping async topic tasks |
org.apache.activemq.store.kahadb.KahaDBStore | main
2018-11-07 11:01:02,320 | INFO | Stopped KahaDB |
org.apache.activemq.store.kahadb.KahaDBStore | main
2018-11-07 11:01:02,404 | INFO | Apache ActiveMQ 5.15.7 (broker-1,
ID:broker-1-43211-1541588461667-1:1) uptime 0.972 seconds |
org.apache.activemq.broker.BrokerService | main
2018-11-07 11:01:02,405 | INFO | Apache ActiveMQ 5.15.7 (broker-1,
ID:broker-1-43211-1541588461667-1:1) is shutdown |
org.apache.activemq.broker.BrokerService | main
2018-11-07 11:01:02,406 | INFO | Closing
org.apache.activemq.xbean.XBeanBrokerFactory$1@5442a311: startup date [Wed Nov
07 11:01:00 UTC 2018]; root of context hierarchy |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
2018-11-07 11:01:02,409 | WARN | Exception encountered during context
initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'org.apache.activemq.xbean.XBeanBrokerService#0' defined in class
path resource [activemq.xml]: Invocation of init method failed; nested
exception is org.apache.activemq.ConfigurationException: File system space
reported by: /efs/activemq-data/kahadb was negative, possibly a huge file
system, set a sane usage.total to provide some guidance |
org.apache.activemq.xbean.XBeanBrokerFactory$1 | main
----
Thank you!
Slava.
was (Author: veaceslavdoina):
Hello,
We also trying to configure ActiveMQ 5.15.7 in a cluster using shared storage.
Setup is done on AWS.
# 2 x EC2 instances
# 1 x EFS
Initially, all worked fine. But after some period of time and master restart,
slave stopped to work.
As was mentioned by [~ChakradharDanireddy], removing data directory solve the
issue but we lost all data:
rm -rf /efs/activemq-data/kahadb/
> Issue With Master and Slave Mode
> --------------------------------
>
> Key: AMQ-6903
> URL: https://issues.apache.org/jira/browse/AMQ-6903
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.15.2
> Reporter: CHAKRADHAR REDDY
> Priority: Critical
>
> Hi All,
> We have installed Apache AMQ in master slave mode we could see that slave is
> not becoming up immediately after master is shutdown and we could see below
> exception.
>
> FYI: We have many messages in queues and this slave is not up till we clear
> data in kahadb location and once we have cleaned data in kahadb location
> there no messages in queues any longer.
> we have lost all messages from queues is there any option we can get messages
> from db.log etc.. files from kahadb location
> We have received below transport Errors this is reason we have shutdown
> master to make slave up
> 2018-01-13 04:33:04,032 | ERROR | Could not accept connection from tcp://IP
> of server containing interfaces accessing AMQ server:52808 : {} |
> org.apache.activemq.broker.TransportConnector | ActiveMQ BrokerServ
> ice[ip] Task-15549
> java.lang.IllegalStateException: Timer already cancelled.
> at java.util.Timer.sched(Timer.java:397)[:1.8.0_151]
> at java.util.Timer.schedule(Timer.java:193)[:1.8.0_151]
> at
> org.apache.activemq.transport.AbstractInactivityMonitor.startConnectCheckTask(AbstractInactivityMonitor.java:425)[activemq-client-5.15.2.jar:5.15.2]
> at org.apache.activemq.transport.AbstractInactivityMonitor.startConne
> After shutdown of master slave is not up with below error and we lost all
> data in queues.
> 2018-02-21 20:09:18,135 | ERROR | File system space reported by:
> /DBA/EFS/activemq/kahadb was negative, possibly a huge file system, set a
> sane usage.total to provide some guidance |
> org.apache.activemq.broker.BrokerService | main
> 2018-02-21 20:09:18,139 | ERROR | Failed to start Apache ActiveMQ (IP, ID:DNS
> of activemq server) | org.apache.activemq.broker.BrokerService | main
> org.apache.activemq.ConfigurationException: File system space reported by:
> /DBA/EFS/activemq/kahadb was negative, possibly a huge file system, set a
> sane usage.total to provide some guidance
> at
> org.apache.activemq.broker.BrokerService.checkUsageLimit(BrokerService.java:2092)[activemq-broker-5.15.2.jar:5.15.2]
> at
> org.apache.activemq.broker.BrokerService.checkStoreUsageLimits(BrokerService.java:2029)[activemq-broker-5.15.2.jar:5.15.2]
> at
> org.apache.activemq.broker.BrokerService.checkStoreSystemUsageLimits(BrokerService.java:2202)[activemq-broker-5.15.2.jar:5.15.2]
> at
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:777)[activemq-broker-5.15.2.jar:5.15.2]
> at
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:733)[activemq-broker-5.15.2.jar:5.15.2]
> at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:636)[activemq-broker-5.15.2.jar:5.15.2]
> at
> org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)[activemq-spring-5.15.2.jar:5.15.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_151]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_151]
> :
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
