Thanks everyone. I think this is what I would need to convince the team to really get out of this. I appreciate the time you put into this and guiding me appropriately.
Thanks again. On Fri, Aug 7, 2020 at 10:24 AM Christopher Shannon < [email protected]> wrote: > In terms of exporting I am not sure. I personally have never looked at > LevelDB. In terms of the error itself it seems like some of the files got > corrupt so not sure if it's possible to recover. If you want to export the > data you would probably need to write your own export tool to parse the > LevelDB structure. Or if you manage to get the broker running again and > want to migrate you could write a client to move data to a new broker. > > Tim is right on the reasoning for deprecation and upcoming removal. > LevelDB has been deprecated for 4 years now and as far as I'm aware no one > still around really knows anything about it and no one is going to answer > any questions on it. There was no community support or knowledge (it was > written in Scala which was one major issue) and the people who wrote it > have long stopped contributing to the project so it didn't make sense to > continue supporting it if no one was around to do so. > > > On Fri, Aug 7, 2020 at 11:20 AM Tim Bain <[email protected]> wrote: > > > LevelDB support was removed in large part due to the absence of > developers > > willing to maintain the code and volunteers willing to provide support > for > > it on this mailing list. In the 5+ years I've been involved in this > mailing > > list, I don't recall ever seeing a LevelDB question answered (and I've > > definitely seen them asked). > > > > When Chris says that LevelDB is unsupported, he means that it's > > UNSUPPORTED. If you're not prepared to provide your own support for > > whatever problems you encounter (leveraging the open-source codebase), > you > > shouldn't be using it, and should switch to KahaDB (or Artemis, as you > > said, but KahaDB gives you a short-term solution for minimal effort). I'm > > sorry to be the bearer of bad news, especially since you might not have > > been aware of LevelDB's status until now, but you're on your own for > > LevelDB issues. > > > > Good luck. > > > > Tim > > > > On Fri, Aug 7, 2020, 8:24 AM Ramesh Venkitaswaran < > [email protected] > > > > > wrote: > > > > > Any ideas? Or, is there a way to export these messages so that I can > > > recreate the broker data directory and import them? > > > > > > On Fri, Aug 7, 2020 at 6:39 AM Ramesh Venkitaswaran < > > > [email protected]> > > > wrote: > > > > > > > Understood, that LevelDB is deprecated, and we are trying to get this > > > team > > > > to move to Artemis. However we still need this broker to run while > the > > > > migration is going on. > > > > > > > > On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon < > > > > [email protected]> wrote: > > > > > > > >> LevelDB support has been deprecated for many years now and is going > to > > > be > > > >> removed in the next version of ActiveMQ (5.17.0) so I would suggest > > > trying > > > >> to migrate over to KahaDB. > > > >> > > > >> On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran < > > > >> [email protected]> > > > >> wrote: > > > >> > > > >> > Hello > > > >> > We have a 3 node activemq 5.15 cluster with zookeeper using Level > > DB. > > > >> It's > > > >> > a very old system. However last night we started to get these > errors > > > and > > > >> > the broker did not come up. Any ideas on how to fix this would be > > > >> > appreciated. I can provide further log messages as needed, if > > > sufficient > > > >> > direction is provided. Thank you! > > > >> > > > > >> > Here are the logs > > > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 > (BROKER01P, > > > >> > > > > >> > > > > >> > > > > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > > >> > is starting | > > > >> > org.apache.activemq.broker.BrokerService | main > > > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > > > >> > > > > >> > > > > >> > > > > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat > > > >> > .maxFrameSize=104857600 | > > > >> > org.apache.activemq.transport.TransportServerThreadSupport | main > > > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > > > >> > org.apache.activemq.broker.TransportConnector | main > > > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > > > >> > > > > >> > > > > >> > > > > > > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0] > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0] > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0] > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0] > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0] > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0] > > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > > >> > Method)[:1.8.0_232] > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232] > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232] > > > >> > at > > > java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232] > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0] > > > >> > at > > > >> > > > > >> > > > > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0] > > > >> > 2020-08-07 02:44:46,669 | INFO | Refreshing > > > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup > > date > > > >> [Fri > > > >> > Aug 07 02:44:46 CDT 2020]; root of context hierarchy | > > > >> > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main > > > >> > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: > > > Replicated > > > >> > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181, > > > >> 10.184.11.9:2181, > > > >> > 10.184.11.7:2181//activemq/ap > > > >> > ps/mkt/lpf/prod-usc1a/BROKER01P] | > > > >> org.apache.activemq.broker.BrokerService > > > >> > | main > > > >> > 2020-08-07 02:44:51,492 | INFO | Client > > > >> > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 > > 09:09 > > > >> GMT > > > >> > | org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,494 | INFO | Client > > > >> > environment:host.name > > > >> > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal > > > >> > | org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,496 | INFO | Client > > > >> environment:java.version=1.8.0_232 > > > >> > | org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,499 | INFO | Client > > > environment:java.vendor=Oracle > > > >> > Corporation | org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,500 | INFO | Client > > > >> > > > > >> > > > > >> > > > > > > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre > > > >> > | org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,504 | INFO | Client > > > >> > > > > >> > > > > >> > > > > > > environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar > > > >> > | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,505 | INFO | Client > > > >> > > > > >> > > > > >> > > > > > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > > >> > | org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,506 | INFO | Client > > > >> > > > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,507 | INFO | Client > > > environment:java.compiler=<NA> > > > >> | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name > =Linux > > | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,508 | INFO | Client > environment:os.arch=amd64 | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,509 | INFO | Client > > > >> > environment:os.version=3.10.0-1062.4.1.el7.x86_64 | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name > > > =activemq > > > >> | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,510 | INFO | Client > > > >> > environment:user.home=/home/activemq | > > org.apache.zookeeper.ZooKeeper > > > | > > > >> > main > > > >> > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,514 | INFO | Initiating client connection, > > > >> > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181 > > > >> > sessionTimeout=2000 watcher=org.apache.activem > > > >> > q.leveldb.replicated.groups.ZKClient@36060e | > > > >> > org.apache.zookeeper.ZooKeeper | main > > > >> > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher | > > > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper > > > state > > > >> > change dispatcher thread > > > >> > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed: > > > >> > javax.security.auth.login.LoginException: No JAAS configuration > > > section > > > >> > named 'Client' was found in specified JAAS > > > >> > configuration file: > > > >> > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will > > > continue > > > >> > connection to Zookeeper server without SASL authentication, if > > > Zookeeper > > > >> > server a > > > >> > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread( > > > >> > 10.184.11.9:2181) > > > >> > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to > > server > > > >> > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn | > > > >> > main-SendThread(10.184.11.9:2181) > > > >> > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: > > AuthFailed > > > | > > > >> > org.apache.activemq.leveldb.replicated.groups.ZKClient | > > > >> main-EventThread > > > >> > 2020-08-07 02:44:51,606 | INFO | Socket connection established to > > > >> > 10.184.11.9/10.184.11.9:2181, initiating session | > > > >> > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184. > > > >> > 11.9:2181) > > > >> > 2020-08-07 02:44:51,622 | INFO | Session establishment complete > on > > > >> server > > > >> > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, > > > negotiated > > > >> > timeout = 4000 | org.apache. > > > >> > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181) > > > >> > 2020-08-07 02:44:51,946 | INFO | Promoted to master | > > > >> > org.apache.activemq.leveldb.replicated.MasterElector | > > > main-EventThread > > > >> > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to > > > >> process > > > >> > the events | > > > >> > > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport > > | > > > >> > main-EventThread > > > >> > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to > > > >> process > > > >> > the events | > > > >> > > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport > > | > > > >> > main-EventThread > > > >> > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB > > > >> > implementation. | org.apache.activemq.leveldb.LevelDBClient | > > ActiveMQ > > > >> > BrokerService[BROKER01P] Task-1 > > > >> > 2020-08-07 02:44:54,349 | INFO | Master started: > > > >> > > > > >> > > > > >> > > > > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218 > > > >> > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ > > > >> > BrokerService[BROKER01P] Task-2 > > > >> > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 > (BROKER01P, > > > >> > > > > >> > > > > >> > > > > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > > >> > is starting | org.apache.activemq.broker.BrokerService | main > > > >> > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > > > >> > > > > >> > > > > >> > > > > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 > > > >> > | org.apache.activemq.transport.TransportServerThreadSupport | > main > > > >> > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > > > >> > org.apache.activemq.broker.TransportConnector | main > > > >> > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > > > >> > > > > >> > > > > >> > > > > > > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true > > > >> > | org.apache.activemq.transport.TransportServerThreadSupport | > main > > > >> > 2020-08-07 02:44:54,825 | INFO | Connector ssl started | > > > >> > org.apache.activemq.broker.TransportConnector | main > > > >> > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 > (BROKER01P, > > > >> > > > > >> > > > > >> > > > > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > > >> > started | org.apache.activemq.broker.BrokerService | main > > > >> > 2020-08-07 02:44:54,827 | INFO | For help or more information > > please > > > >> see: > > > >> > http://activemq.apache.org | > > > org.apache.activemq.broker.BrokerService | > > > >> > main > > > >> > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb > (current > > > >> store > > > >> > usage is 25900 mb). The data directory: > > /var/lib/activemq/data/leveldb > > > >> only > > > >> > has 42345 mb of usable space. - resetting to maximum available > disk > > > >> space: > > > >> > 42345 mb | org.apache.activemq.broker.BrokerService | main > > > >> > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 > mb > > > >> > (current store usage is 25900 mb). The data directory: > > > >> > /var/lib/activemq/data only has 42345 mb of usable space. - > > resetting > > > to > > > >> > maximum available disk space: 42345 mb | > > > >> > org.apache.activemq.broker.BrokerService | main > > > >> > 2020-08-07 02:44:55,677 | INFO | Stopping > BrokerService[BROKER01P] > > > due > > > >> to > > > >> > exception, java.io.EOFException: File > > > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: > > > 273453365 > > > >> | > > > >> > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB > > > IOException > > > >> > handler. > > > >> > java.io.EOFException: File > > > >> > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: > > > 273453365 > > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > >> > at > > > >> > > > > >> > > > > >> > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > >> > at java.lang.reflect.Method.invoke(Method.java:498) > > > >> > at > org.apache.activemq.console.Main.runTaskClass(Main.java:262) > > > >> > at org.apache.activemq.console.Main.main(Main.java:115) > > > >> > > > > >> > > > > > > > > > >
