Any ideas? Or, is there a way to export these messages so that I can recreate the broker data directory and import them?
On Fri, Aug 7, 2020 at 6:39 AM Ramesh Venkitaswaran <[email protected]> wrote: > Understood, that LevelDB is deprecated, and we are trying to get this team > to move to Artemis. However we still need this broker to run while the > migration is going on. > > On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon < > [email protected]> wrote: > >> LevelDB support has been deprecated for many years now and is going to be >> removed in the next version of ActiveMQ (5.17.0) so I would suggest trying >> to migrate over to KahaDB. >> >> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran < >> [email protected]> >> wrote: >> >> > Hello >> > We have a 3 node activemq 5.15 cluster with zookeeper using Level DB. >> It's >> > a very old system. However last night we started to get these errors and >> > the broker did not come up. Any ideas on how to fix this would be >> > appreciated. I can provide further log messages as needed, if sufficient >> > direction is provided. Thank you! >> > >> > Here are the logs >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, >> > >> > >> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) >> > is starting | >> > org.apache.activemq.broker.BrokerService | main >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: >> > >> > >> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat >> > .maxFrameSize=104857600 | >> > org.apache.activemq.transport.TransportServerThreadSupport | main >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | >> > org.apache.activemq.broker.TransportConnector | main >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: >> > >> > >> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat >> > at >> > >> > >> org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0] >> > at >> > >> > >> org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0] >> > at >> > >> > >> org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0] >> > at >> > >> > >> org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0] >> > at >> > >> > >> org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0] >> > at >> > >> > >> org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0] >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > Method)[:1.8.0_232] >> > at >> > >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232] >> > at >> > >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232] >> > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232] >> > at >> > >> > >> org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0] >> > at >> > >> org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0] >> > 2020-08-07 02:44:46,669 | INFO | Refreshing >> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date >> [Fri >> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy | >> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main >> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: Replicated >> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181, >> 10.184.11.9:2181, >> > 10.184.11.7:2181//activemq/ap >> > ps/mkt/lpf/prod-usc1a/BROKER01P] | >> org.apache.activemq.broker.BrokerService >> > | main >> > 2020-08-07 02:44:51,492 | INFO | Client >> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 >> GMT >> > | org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,494 | INFO | Client >> > environment:host.name >> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal >> > | org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,496 | INFO | Client >> environment:java.version=1.8.0_232 >> > | org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,499 | INFO | Client environment:java.vendor=Oracle >> > Corporation | org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,500 | INFO | Client >> > >> > >> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre >> > | org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,504 | INFO | Client >> > >> > >> environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar >> > | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,505 | INFO | Client >> > >> > >> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib >> > | org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,506 | INFO | Client >> > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,507 | INFO | Client environment:java.compiler=<NA> >> | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,509 | INFO | Client >> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name=activemq >> | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,510 | INFO | Client >> > environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper | >> > main >> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection, >> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181 >> > sessionTimeout=2000 watcher=org.apache.activem >> > q.leveldb.replicated.groups.ZKClient@36060e | >> > org.apache.zookeeper.ZooKeeper | main >> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher | >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state >> > change dispatcher thread >> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed: >> > javax.security.auth.login.LoginException: No JAAS configuration section >> > named 'Client' was found in specified JAAS >> > configuration file: >> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will continue >> > connection to Zookeeper server without SASL authentication, if Zookeeper >> > server a >> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread( >> > 10.184.11.9:2181) >> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to server >> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn | >> > main-SendThread(10.184.11.9:2181) >> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed | >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | >> main-EventThread >> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to >> > 10.184.11.9/10.184.11.9:2181, initiating session | >> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184. >> > 11.9:2181) >> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on >> server >> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, negotiated >> > timeout = 4000 | org.apache. >> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181) >> > 2020-08-07 02:44:51,946 | INFO | Promoted to master | >> > org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread >> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to >> process >> > the events | >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport | >> > main-EventThread >> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to >> process >> > the events | >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport | >> > main-EventThread >> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB >> > implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ >> > BrokerService[BROKER01P] Task-1 >> > 2020-08-07 02:44:54,349 | INFO | Master started: >> > >> > >> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218 >> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ >> > BrokerService[BROKER01P] Task-2 >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, >> > >> > >> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) >> > is starting | org.apache.activemq.broker.BrokerService | main >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: >> > >> > >> tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 >> > | org.apache.activemq.transport.TransportServerThreadSupport | main >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | >> > org.apache.activemq.broker.TransportConnector | main >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: >> > >> > >> ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true >> > | org.apache.activemq.transport.TransportServerThreadSupport | main >> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started | >> > org.apache.activemq.broker.TransportConnector | main >> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, >> > >> > >> ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) >> > started | org.apache.activemq.broker.BrokerService | main >> > 2020-08-07 02:44:54,827 | INFO | For help or more information please >> see: >> > http://activemq.apache.org | org.apache.activemq.broker.BrokerService | >> > main >> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current >> store >> > usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb >> only >> > has 42345 mb of usable space. - resetting to maximum available disk >> space: >> > 42345 mb | org.apache.activemq.broker.BrokerService | main >> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb >> > (current store usage is 25900 mb). The data directory: >> > /var/lib/activemq/data only has 42345 mb of usable space. - resetting to >> > maximum available disk space: 42345 mb | >> > org.apache.activemq.broker.BrokerService | main >> > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] due >> to >> > exception, java.io.EOFException: File >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 >> | >> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException >> > handler. >> > java.io.EOFException: File >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> > >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> > >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at org.apache.activemq.console.Main.runTaskClass(Main.java:262) >> > at org.apache.activemq.console.Main.main(Main.java:115) >> > >> >
