Understood, that LevelDB is deprecated, and we are trying to get this team to move to Artemis. However we still need this broker to run while the migration is going on.
On Fri, Aug 7, 2020 at 5:48 AM Christopher Shannon < [email protected]> wrote: > LevelDB support has been deprecated for many years now and is going to be > removed in the next version of ActiveMQ (5.17.0) so I would suggest trying > to migrate over to KahaDB. > > On Fri, Aug 7, 2020 at 4:08 AM Ramesh Venkitaswaran < > [email protected]> > wrote: > > > Hello > > We have a 3 node activemq 5.15 cluster with zookeeper using Level DB. > It's > > a very old system. However last night we started to get these errors and > > the broker did not come up. Any ideas on how to fix this would be > > appreciated. I can provide further log messages as needed, if sufficient > > direction is provided. Thank you! > > > > Here are the logs > > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > is starting | > > org.apache.activemq.broker.BrokerService | main > > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > > > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat > > .maxFrameSize=104857600 | > > org.apache.activemq.transport.TransportServerThreadSupport | main > > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > > org.apache.activemq.broker.TransportConnector | main > > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > > > > > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat > > at > > > > > org.apache.activemq.util.IntrospectionSupport.setProperty(IntrospectionSupport.java:184)[activemq-client-5.15.0.jar:5.15.0] > > at > > > > > org.apache.activemq.console.command.store.amq.CommandLineSupport.setOptions(CommandLineSupport.java:81)[activemq-console-5.15.0.jar:5.15.0] > > at > > > > > org.apache.activemq.console.command.StoreExportCommand.execute(StoreExportCommand.java:51)[activemq-console-5.15.0.jar:5.15.0] > > at > > > > > org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:154)[activemq-console-5.15.0.jar:5.15.0] > > at > > > > > org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:63)[activemq-console-5.15.0.jar:5.15.0] > > at > > > > > org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)[activemq-console-5.15.0.jar:5.15.0] > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method)[:1.8.0_232] > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_232] > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_232] > > at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_232] > > at > > > > > org.apache.activemq.console.Main.runTaskClass(Main.java:262)[activemq.jar:5.15.0] > > at > > org.apache.activemq.console.Main.main(Main.java:115)[activemq.jar:5.15.0] > > 2020-08-07 02:44:46,669 | INFO | Refreshing > > org.apache.activemq.xbean.XBeanBrokerFactory$1@3159c4b8: startup date > [Fri > > Aug 07 02:44:46 CDT 2020]; root of context hierarchy | > > org.apache.activemq.xbean.XBeanBrokerFactory$1 | main > > 2020-08-07 02:44:49,462 | INFO | Using Persistence Adapter: Replicated > > LevelDB[/var/lib/activemq/data/leveldb, 10.184.11.8:2181, > 10.184.11.9:2181, > > 10.184.11.7:2181//activemq/ap > > ps/mkt/lpf/prod-usc1a/BROKER01P] | > org.apache.activemq.broker.BrokerService > > | main > > 2020-08-07 02:44:51,492 | INFO | Client > > environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 > GMT > > | org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,494 | INFO | Client > > environment:host.name > > =mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal > > | org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,496 | INFO | Client > environment:java.version=1.8.0_232 > > | org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,499 | INFO | Client environment:java.vendor=Oracle > > Corporation | org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,500 | INFO | Client > > > > > environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.232.b09-0.el7_7.x86_64/jre > > | org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,504 | INFO | Client > > > > > environment:java.class.path=/opt/activemq/apache-activemq-5.15.0//bin/activemq.jar:/srv/prop/cavisson/netdiagnostics/lib/ndmain.jar > > | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,505 | INFO | Client > > > > > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > > | org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,506 | INFO | Client > > environment:java.io.tmpdir=/opt/activemq/apache-activemq-5.15.0//tmp | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,507 | INFO | Client environment:java.compiler=<NA> | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,507 | INFO | Client environment:os.name=Linux | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,508 | INFO | Client environment:os.arch=amd64 | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,509 | INFO | Client > > environment:os.version=3.10.0-1062.4.1.el7.x86_64 | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,509 | INFO | Client environment:user.name=activemq > | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,510 | INFO | Client > > environment:user.home=/home/activemq | org.apache.zookeeper.ZooKeeper | > > main > > 2020-08-07 02:44:51,511 | INFO | Client environment:user.dir=/ | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,514 | INFO | Initiating client connection, > > connectString=10.184.11.8:2181,10.184.11.9:2181,10.184.11.7:2181 > > sessionTimeout=2000 watcher=org.apache.activem > > q.leveldb.replicated.groups.ZKClient@36060e | > > org.apache.zookeeper.ZooKeeper | main > > 2020-08-07 02:44:51,499 | INFO | Starting StateChangeDispatcher | > > org.apache.activemq.leveldb.replicated.groups.ZKClient | ZooKeeper state > > change dispatcher thread > > 2020-08-07 02:44:51,573 | WARN | SASL configuration failed: > > javax.security.auth.login.LoginException: No JAAS configuration section > > named 'Client' was found in specified JAAS > > configuration file: > > '/opt/activemq/apache-activemq-5.15.0//conf/login.config'. Will continue > > connection to Zookeeper server without SASL authentication, if Zookeeper > > server a > > llows it. | org.apache.zookeeper.ClientCnxn | main-SendThread( > > 10.184.11.9:2181) > > 2020-08-07 02:44:51,577 | INFO | Opening socket connection to server > > 10.184.11.9/10.184.11.9:2181 | org.apache.zookeeper.ClientCnxn | > > main-SendThread(10.184.11.9:2181) > > 2020-08-07 02:44:51,587 | WARN | unprocessed event state: AuthFailed | > > org.apache.activemq.leveldb.replicated.groups.ZKClient | main-EventThread > > 2020-08-07 02:44:51,606 | INFO | Socket connection established to > > 10.184.11.9/10.184.11.9:2181, initiating session | > > org.apache.zookeeper.ClientCnxn | main-SendThread(10.184. > > 11.9:2181) > > 2020-08-07 02:44:51,622 | INFO | Session establishment complete on > server > > 10.184.11.9/10.184.11.9:2181, sessionid = 0x373c7b2a5960001, negotiated > > timeout = 4000 | org.apache. > > zookeeper.ClientCnxn | main-SendThread(10.184.11.9:2181) > > 2020-08-07 02:44:51,946 | INFO | Promoted to master | > > org.apache.activemq.leveldb.replicated.MasterElector | main-EventThread > > 2020-08-07 02:44:52,110 | WARN | listeners are taking too long to > process > > the events | > > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport | > > main-EventThread > > 2020-08-07 02:44:52,111 | WARN | listeners are taking too long to > process > > the events | > > org.apache.activemq.leveldb.replicated.groups.ChangeListenerSupport | > > main-EventThread > > 2020-08-07 02:44:52,159 | INFO | Using the pure java LevelDB > > implementation. | org.apache.activemq.leveldb.LevelDBClient | ActiveMQ > > BrokerService[BROKER01P] Task-1 > > 2020-08-07 02:44:54,349 | INFO | Master started: > > > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:41218 > > | org.apache.activemq.leveldb.replicated.MasterElector | ActiveMQ > > BrokerService[BROKER01P] Task-2 > > 2020-08-07 02:44:54,502 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > is starting | org.apache.activemq.broker.BrokerService | main > > 2020-08-07 02:44:54,732 | INFO | Listening for connections at: > > > > > tcp://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 > > | org.apache.activemq.transport.TransportServerThreadSupport | main > > 2020-08-07 02:44:54,733 | INFO | Connector openwire started | > > org.apache.activemq.broker.TransportConnector | main > > 2020-08-07 02:44:54,823 | INFO | Listening for connections at: > > > > > ssl://mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal:61617?maximumConnections=1000&wireFormat.maxFrameSize=104857600&needClientAuth=true > > | org.apache.activemq.transport.TransportServerThreadSupport | main > > 2020-08-07 02:44:54,825 | INFO | Connector ssl started | > > org.apache.activemq.broker.TransportConnector | main > > 2020-08-07 02:44:54,826 | INFO | Apache ActiveMQ 5.15.0 (BROKER01P, > > > > > ID:mkt-lpf-prod-usc1a-activemq-broker01p-1.c.xxxxx-mkt-prd.internal-35573-1596786294369-0:1) > > started | org.apache.activemq.broker.BrokerService | main > > 2020-08-07 02:44:54,827 | INFO | For help or more information please > see: > > http://activemq.apache.org | org.apache.activemq.broker.BrokerService | > > main > > 2020-08-07 02:44:54,828 | WARN | Store limit is 102400 mb (current store > > usage is 25900 mb). The data directory: /var/lib/activemq/data/leveldb > only > > has 42345 mb of usable space. - resetting to maximum available disk > space: > > 42345 mb | org.apache.activemq.broker.BrokerService | main > > 2020-08-07 02:44:54,830 | WARN | Temporary Store limit is 51200 mb > > (current store usage is 25900 mb). The data directory: > > /var/lib/activemq/data only has 42345 mb of usable space. - resetting to > > maximum available disk space: 42345 mb | > > org.apache.activemq.broker.BrokerService | main > > 2020-08-07 02:44:55,677 | INFO | Stopping BrokerService[BROKER01P] due > to > > exception, java.io.EOFException: File > > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 | > > org.apache.activemq.util.DefaultIOExceptionHandler | LevelDB IOException > > handler. > > java.io.EOFException: File > > '/var/lib/activemq/data/leveldb/0000000d1063f5cc.log' offset: 273453365 > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at org.apache.activemq.console.Main.runTaskClass(Main.java:262) > > at org.apache.activemq.console.Main.main(Main.java:115) > > >
