In terms of exporting I am not sure. I personally have never looked at LevelDB. In terms of the error itself it seems like some of the files got corrupt so not sure if it's possible to recover. If you want to export the data you would probably need to write your own export tool to parse the LevelDB structure. Or if you manage to get the broker running again and want to migrate you could write a client to move data to a new broker.
Tim is right on the reasoning for deprecation and upcoming removal. LevelDB has been deprecated for 4 years now and as far as I'm aware no one still around really knows anything about it and no one is going to answer any questions on it. There was no community support or knowledge (it was written in Scala which was one major issue) and the people who wrote it have long stopped contributing to the project so it didn't make sense to continue supporting it if no one was around to do so. On Fri, Aug 7, 2020 at 11:20 AM Tim Bain <tb...@alumni.duke.edu> wrote: > LevelDB support was removed in large part due to the absence of developers > willing to maintain the code and volunteers willing to provide support for > it on this mailing list. In the 5+ years I've been involved in this mailing > list, I don't recall ever seeing a LevelDB question answered (and I've > definitely seen them asked). > > When Chris says that LevelDB is unsupported, he means that it's > UNSUPPORTED. If you're not prepared to provide your own support for > whatever problems you encounter (leveraging the open-source codebase), you > shouldn't be using it, and should switch to KahaDB (or Artemis, as you > said, but KahaDB gives you a short-term solution for minimal effort). I'm > sorry to be the bearer of bad news, especially since you might not have > been aware of LevelDB's status until now, but you're on your own for > LevelDB issues. > > Good luck. > > Tim > > On Fri, Aug 7, 2020, 8:24 AM Ramesh Venkitaswaran <venkitaswa...@gmail.com > > > wrote: > > > Any ideas? Or, is there a way to export these messages so that I can > > recreate the broker data directory and import them? > > > > On Fri, Aug 7, 2020 at 6:39 AM Ramesh Venkitaswaran < > > venkitaswa...@gmail.com> > > wrote: > > > > > Understood, that LevelDB is deprecated, and we are trying to get this > > team > > > to move to Artemis. However we still need this broker to run while the > > > migration is going on. > > > > > > On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon < > > > christopher.l.shan...@gmail.com> wrote: > > > > > >> LevelDB support has been deprecated for many years now and is going to > > be > > >> removed in the next version of ActiveMQ (5.17.0) so I would suggest > > trying > > >> to migrate over to KahaDB. > > >> > > >> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran < > > >> venkitaswa...@gmail.com> > > >> wrote: > > >> > > >> > Hello > > >> > We have a 3 node activemq 5.15 cluster with zookeeper using Level > DB. > > >> It's > > >> > a very old system. However last night we started to get these errors > > and > > >> > the broker did not come up. Any ideas on how to fix this would be > > >> > appreciated. I can provide further log messages as needed, if > > sufficient > > >> > direction is provided. Thank you! > > >> > > > >> > Here are the logs > > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > >> > > > >> > > > >> > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > >> > is starting | > > >> > org.apache.activemq.broker.BrokerService | main > > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > > >> > > > >> > > > >> > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat > > >> > .maxFrameSize=104857600 | > > >> > org.apache.activemq.transport.TransportServerThreadSupport | main > > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > > >> > org.apache.activemq.broker.TransportConnector | main > > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > > >> > > > >> > > > >> > > > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat > > >> > at > > >> > > > >> > > > >> > > > org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0] > > >> > at > > >> > > > >> > > > >> > > > org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0] > > >> > at > > >> > > > >> > > > >> > > > org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0] > > >> > at > > >> > > > >> > > > >> > > > org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0] > > >> > at > > >> > > > >> > > > >> > > > org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0] > > >> > at > > >> > > > >> > > > >> > > > org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0] > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > >> > Method)[:1.8.0_232] > > >> > at > > >> > > > >> > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232] > > >> > at > > >> > > > >> > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232] > > >> > at > > java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232] > > >> > at > > >> > > > >> > > > >> > > > org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0] > > >> > at > > >> > > > >> > > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0] > > >> > 2020-08-07 02:44:46,669 | INFO | Refreshing > > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup > date > > >> [Fri > > >> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy | > > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main > > >> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: > > Replicated > > >> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181, > > >> 10.184.11.9:2181, > > >> > 10.184.11.7:2181//activemq/ap > > >> > ps/mkt/lpf/prod-usc1a/BROKER01P] | > > >> org.apache.activemq.broker.BrokerService > > >> > | main > > >> > 2020-08-07 02:44:51,492 | INFO | Client > > >> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 > 09:09 > > >> GMT > > >> > | org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,494 | INFO | Client > > >> > environment:host.name > > >> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal > > >> > | org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,496 | INFO | Client > > >> environment:java.version=1.8.0_232 > > >> > | org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,499 | INFO | Client > > environment:java.vendor=Oracle > > >> > Corporation | org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,500 | INFO | Client > > >> > > > >> > > > >> > > > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre > > >> > | org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,504 | INFO | Client > > >> > > > >> > > > >> > > > environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar > > >> > | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,505 | INFO | Client > > >> > > > >> > > > >> > > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > >> > | org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,506 | INFO | Client > > >> > > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,507 | INFO | Client > > environment:java.compiler=<NA> > > >> | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux > | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,509 | INFO | Client > > >> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name > > =activemq > > >> | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,510 | INFO | Client > > >> > environment:user.home=/home/activemq | > org.apache.zookeeper.ZooKeeper > > | > > >> > main > > >> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection, > > >> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181 > > >> > sessionTimeout=2000 watcher=org.apache.activem > > >> > q.leveldb.replicated.groups.ZKClient@36060e | > > >> > org.apache.zookeeper.ZooKeeper | main > > >> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher | > > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper > > state > > >> > change dispatcher thread > > >> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed: > > >> > javax.security.auth.login.LoginException: No JAAS configuration > > section > > >> > named 'Client' was found in specified JAAS > > >> > configuration file: > > >> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will > > continue > > >> > connection to Zookeeper server without SASL authentication, if > > Zookeeper > > >> > server a > > >> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread( > > >> > 10.184.11.9:2181) > > >> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to > server > > >> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn | > > >> > main-SendThread(10.184.11.9:2181) > > >> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: > AuthFailed > > | > > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | > > >> main-EventThread > > >> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to > > >> > 10.184.11.9/10.184.11.9:2181, initiating session | > > >> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184. > > >> > 11.9:2181) > > >> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on > > >> server > > >> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, > > negotiated > > >> > timeout = 4000 | org.apache. > > >> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181) > > >> > 2020-08-07 02:44:51,946 | INFO | Promoted to master | > > >> > org.apache.activemq.leveldb.replicated.MasterElector | > > main-EventThread > > >> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to > > >> process > > >> > the events | > > >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport > | > > >> > main-EventThread > > >> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to > > >> process > > >> > the events | > > >> > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport > | > > >> > main-EventThread > > >> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB > > >> > implementation. | org.apache.activemq.leveldb.LevelDBClient | > ActiveMQ > > >> > BrokerService[BROKER01P] Task-1 > > >> > 2020-08-07 02:44:54,349 | INFO | Master started: > > >> > > > >> > > > >> > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218 > > >> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ > > >> > BrokerService[BROKER01P] Task-2 > > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > >> > > > >> > > > >> > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > >> > is starting | org.apache.activemq.broker.BrokerService | main > > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > > >> > > > >> > > > >> > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 > > >> > | org.apache.activemq.transport.TransportServerThreadSupport | main > > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > > >> > org.apache.activemq.broker.TransportConnector | main > > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > > >> > > > >> > > > >> > > > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true > > >> > | org.apache.activemq.transport.TransportServerThreadSupport | main > > >> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started | > > >> > org.apache.activemq.broker.TransportConnector | main > > >> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > >> > > > >> > > > >> > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > >> > started | org.apache.activemq.broker.BrokerService | main > > >> > 2020-08-07 02:44:54,827 | INFO | For help or more information > please > > >> see: > > >> > http://activemq.apache.org | > > org.apache.activemq.broker.BrokerService | > > >> > main > > >> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current > > >> store > > >> > usage is 25900 mb). The data directory: > /var/lib/activemq/data/leveldb > > >> only > > >> > has 42345 mb of usable space. - resetting to maximum available disk > > >> space: > > >> > 42345 mb | org.apache.activemq.broker.BrokerService | main > > >> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb > > >> > (current store usage is 25900 mb). The data directory: > > >> > /var/lib/activemq/data only has 42345 mb of usable space. - > resetting > > to > > >> > maximum available disk space: 42345 mb | > > >> > org.apache.activemq.broker.BrokerService | main > > >> > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] > > due > > >> to > > >> > exception, java.io.EOFException: File > > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: > > 273453365 > > >> | > > >> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB > > IOException > > >> > handler. > > >> > java.io.EOFException: File > > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: > > 273453365 > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> > at > > >> > > > >> > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > >> > at > > >> > > > >> > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > >> > at java.lang.reflect.Method.invoke(Method.java:498) > > >> > at org.apache.activemq.console.Main.runTaskClass(Main.java:262) > > >> > at org.apache.activemq.console.Main.main(Main.java:115) > > >> > > > >> > > > > > >
