LevelDB support was removed in large part due to the absence of developers willing to maintain the code and volunteers willing to provide support for it on this mailing list. In the 5+ years I've been involved in this mailing list, I don't recall ever seeing a LevelDB question answered (and I've definitely seen them asked).
When Chris says that LevelDB is unsupported, he means that it's UNSUPPORTED. If you're not prepared to provide your own support for whatever problems you encounter (leveraging the open-source codebase), you shouldn't be using it, and should switch to KahaDB (or Artemis, as you said, but KahaDB gives you a short-term solution for minimal effort). I'm sorry to be the bearer of bad news, especially since you might not have been aware of LevelDB's status until now, but you're on your own for LevelDB issues. Good luck. Tim On Fri, Aug 7, 2020, 8:24 AM Ramesh Venkitaswaran <[email protected]> wrote: > Any ideas? Or, is there a way to export these messages so that I can > recreate the broker data directory and import them? > > On Fri, Aug 7, 2020 at 6:39 AM Ramesh Venkitaswaran < > [email protected]> > wrote: > > > Understood, that LevelDB is deprecated, and we are trying to get this > team > > to move to Artemis. However we still need this broker to run while the > > migration is going on. > > > > On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon < > > [email protected]> wrote: > > > >> LevelDB support has been deprecated for many years now and is going to > be > >> removed in the next version of ActiveMQ (5.17.0) so I would suggest > trying > >> to migrate over to KahaDB. > >> > >> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran < > >> [email protected]> > >> wrote: > >> > >> > Hello > >> > We have a 3 node activemq 5.15 cluster with zookeeper using Level DB. > >> It's > >> > a very old system. However last night we started to get these errors > and > >> > the broker did not come up. Any ideas on how to fix this would be > >> > appreciated. I can provide further log messages as needed, if > sufficient > >> > direction is provided. Thank you! > >> > > >> > Here are the logs > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > >> > > >> > > >> > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > >> > is starting | > >> > org.apache.activemq.broker.BrokerService | main > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > >> > > >> > > >> > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat > >> > .maxFrameSize=104857600 | > >> > org.apache.activemq.transport.TransportServerThreadSupport | main > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > >> > org.apache.activemq.broker.TransportConnector | main > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > >> > > >> > > >> > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat > >> > at > >> > > >> > > >> > org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0] > >> > at > >> > > >> > > >> > org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0] > >> > at > >> > > >> > > >> > org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0] > >> > at > >> > > >> > > >> > org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0] > >> > at > >> > > >> > > >> > org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0] > >> > at > >> > > >> > > >> > org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0] > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >> > Method)[:1.8.0_232] > >> > at > >> > > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232] > >> > at > >> > > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232] > >> > at > java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232] > >> > at > >> > > >> > > >> > org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0] > >> > at > >> > > >> > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0] > >> > 2020-08-07 02:44:46,669 | INFO | Refreshing > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date > >> [Fri > >> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy | > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main > >> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: > Replicated > >> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181, > >> 10.184.11.9:2181, > >> > 10.184.11.7:2181//activemq/ap > >> > ps/mkt/lpf/prod-usc1a/BROKER01P] | > >> org.apache.activemq.broker.BrokerService > >> > | main > >> > 2020-08-07 02:44:51,492 | INFO | Client > >> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 > >> GMT > >> > | org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,494 | INFO | Client > >> > environment:host.name > >> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal > >> > | org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,496 | INFO | Client > >> environment:java.version=1.8.0_232 > >> > | org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,499 | INFO | Client > environment:java.vendor=Oracle > >> > Corporation | org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,500 | INFO | Client > >> > > >> > > >> > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre > >> > | org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,504 | INFO | Client > >> > > >> > > >> > environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar > >> > | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,505 | INFO | Client > >> > > >> > > >> > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > >> > | org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,506 | INFO | Client > >> > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,507 | INFO | Client > environment:java.compiler=<NA> > >> | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,509 | INFO | Client > >> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name > =activemq > >> | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,510 | INFO | Client > >> > environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper > | > >> > main > >> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection, > >> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181 > >> > sessionTimeout=2000 watcher=org.apache.activem > >> > q.leveldb.replicated.groups.ZKClient@36060e | > >> > org.apache.zookeeper.ZooKeeper | main > >> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher | > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper > state > >> > change dispatcher thread > >> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed: > >> > javax.security.auth.login.LoginException: No JAAS configuration > section > >> > named 'Client' was found in specified JAAS > >> > configuration file: > >> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will > continue > >> > connection to Zookeeper server without SASL authentication, if > Zookeeper > >> > server a > >> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread( > >> > 10.184.11.9:2181) > >> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to server > >> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn | > >> > main-SendThread(10.184.11.9:2181) > >> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed > | > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | > >> main-EventThread > >> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to > >> > 10.184.11.9/10.184.11.9:2181, initiating session | > >> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184. > >> > 11.9:2181) > >> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on > >> server > >> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, > negotiated > >> > timeout = 4000 | org.apache. > >> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181) > >> > 2020-08-07 02:44:51,946 | INFO | Promoted to master | > >> > org.apache.activemq.leveldb.replicated.MasterElector | > main-EventThread > >> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to > >> process > >> > the events | > >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport | > >> > main-EventThread > >> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to > >> process > >> > the events | > >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport | > >> > main-EventThread > >> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB > >> > implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ > >> > BrokerService[BROKER01P] Task-1 > >> > 2020-08-07 02:44:54,349 | INFO | Master started: > >> > > >> > > >> > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218 > >> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ > >> > BrokerService[BROKER01P] Task-2 > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > >> > > >> > > >> > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > >> > is starting | org.apache.activemq.broker.BrokerService | main > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > >> > > >> > > >> > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 > >> > | org.apache.activemq.transport.TransportServerThreadSupport | main > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > >> > org.apache.activemq.broker.TransportConnector | main > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > >> > > >> > > >> > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true > >> > | org.apache.activemq.transport.TransportServerThreadSupport | main > >> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started | > >> > org.apache.activemq.broker.TransportConnector | main > >> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > >> > > >> > > >> > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > >> > started | org.apache.activemq.broker.BrokerService | main > >> > 2020-08-07 02:44:54,827 | INFO | For help or more information please > >> see: > >> > http://activemq.apache.org | > org.apache.activemq.broker.BrokerService | > >> > main > >> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current > >> store > >> > usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb > >> only > >> > has 42345 mb of usable space. - resetting to maximum available disk > >> space: > >> > 42345 mb | org.apache.activemq.broker.BrokerService | main > >> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb > >> > (current store usage is 25900 mb). The data directory: > >> > /var/lib/activemq/data only has 42345 mb of usable space. - resetting > to > >> > maximum available disk space: 42345 mb | > >> > org.apache.activemq.broker.BrokerService | main > >> > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] > due > >> to > >> > exception, java.io.EOFException: File > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: > 273453365 > >> | > >> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB > IOException > >> > handler. > >> > java.io.EOFException: File > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: > 273453365 > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> > at > >> > > >> > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> > at > >> > > >> > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> > at java.lang.reflect.Method.invoke(Method.java:498) > >> > at org.apache.activemq.console.Main.runTaskClass(Main.java:262) > >> > at org.apache.activemq.console.Main.main(Main.java:115) > >> > > >> > > >
