Klaus Pittig created AMQ-6011:
---------------------------------
Summary: create how-to repair/recover corrupt LevelDB
Key: AMQ-6011
URL: https://issues.apache.org/jira/browse/AMQ-6011
Project: ActiveMQ
Issue Type: Improvement
Components: Message Store
Affects Versions: 5.11.2
Environment: Windows x64 7,8; Oracle Java 8 (u60)
Reporter: Klaus Pittig
With a single AMQ 5.11.2 instance, Java8u60 and Windows 7+ we can produce
LevelDB corruption by putting the system to sleep (sometimes by simply closing
a notebook).
Irrespective of whether this bug should have been recovered by the LevelDB
store automatically or not, how can this situation been recovered _without_
losing the persistent queue content? The KahaDB recovery docs were enough in
most situations, and something near it with LevelDB could help not to switch
back to KahaDB or other Persistence Adapters.
The above mentioned scenario produces the following log output, how can this be
fixed reliably with a minimum of data loss?
{code}
jvm 1 | ERROR: org.springframework.beans.factory.BeanCreationException:
Error creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0'
defined in class path resource [activemq.xml]: Invocation of init method
failed; nested exception is
org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException:
java.lang.RuntimeException: Could not open table 11
jvm 1 | org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'org.apache.activemq.xbean.XBeanBrokerService#0'
defined in class path resource [activemq.xml]: Invocation of init method
failed; nested exception is
org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException:
java.lang.RuntimeException: Could not open table 11
jvm 1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1514)
jvm 1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:521)
jvm 1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:458)
jvm 1 | at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
jvm 1 | at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
jvm 1 | at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
jvm 1 | at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
jvm 1 | at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:636)
jvm 1 | at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:934)
jvm 1 | at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
jvm 1 | at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
jvm 1 | at
org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
jvm 1 | at
org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:104)
jvm 1 | at
org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:104)
jvm 1 | at
org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:67)
jvm 1 | at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
jvm 1 | at
org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
jvm 1 | at
org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:87)
jvm 1 | at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
jvm 1 | at
org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:150)
jvm 1 | at
org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
jvm 1 | at
org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:104)
jvm 1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1 | at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
jvm 1 | at java.lang.reflect.Method.invoke(Unknown Source)
jvm 1 | at org.apache.activemq.console.Main.runTaskClass(Main.java:262)
jvm 1 | at org.apache.activemq.console.Main.main(Main.java:115)
jvm 1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1 | at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
jvm 1 | at java.lang.reflect.Method.invoke(Unknown Source)
jvm 1 | at
org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:240)
jvm 1 | at java.lang.Thread.run(Unknown Source)
jvm 1 | Caused by:
org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException:
java.lang.RuntimeException: Could not open table 11
jvm 1 | at
org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:411)
jvm 1 | at
org.iq80.leveldb.impl.DbImpl.createWriteBatch(DbImpl.java:707)
jvm 1 | at
org.apache.activemq.leveldb.LevelDBClient.replay_from(LevelDBClient.scala:704)
jvm 1 | at
org.apache.activemq.leveldb.LevelDBClient.start(LevelDBClient.scala:572)
jvm 1 | at
org.apache.activemq.leveldb.DBManager.start(DBManager.scala:648)
jvm 1 | at
org.apache.activemq.leveldb.LevelDBStore.doStart(LevelDBStore.scala:312)
jvm 1 | at
org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)
jvm 1 | at
org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:653)
jvm 1 | at
org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:642)
jvm 1 | at
org.apache.activemq.broker.BrokerService.start(BrokerService.java:607)
jvm 1 | at
org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:73)
jvm 1 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1 | at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source)
jvm 1 | at java.lang.reflect.Method.invoke(Unknown Source)
jvm 1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1640)
jvm 1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1581)
jvm 1 | at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1511)
jvm 1 | ... 33 more
jvm 1 | Caused by: java.lang.RuntimeException: Could not open table 11
jvm 1 | at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:87)
jvm 1 | at
org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:69)
jvm 1 | at
org.iq80.leveldb.impl.TableCache.newIterator(TableCache.java:64)
jvm 1 | at
org.iq80.leveldb.util.Level0Iterator.<init>(Level0Iterator.java:29)
jvm 1 | at
org.iq80.leveldb.impl.VersionSet.makeInputIterator(VersionSet.java:205)
jvm 1 | at
org.iq80.leveldb.impl.DbImpl.doCompactionWork(DbImpl.java:1009)
jvm 1 | at
org.iq80.leveldb.impl.DbImpl.backgroundCompaction(DbImpl.java:478)
jvm 1 | at org.iq80.leveldb.impl.DbImpl.backgroundCall(DbImpl.java:426)
jvm 1 | at org.iq80.leveldb.impl.DbImpl.access$100(DbImpl.java:83)
jvm 1 | at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:396)
jvm 1 | at org.iq80.leveldb.impl.DbImpl$2.call(DbImpl.java:390)
jvm 1 | at java.util.concurrent.FutureTask.run(Unknown Source)
jvm 1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
jvm 1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
jvm 1 | ... 1 more
jvm 1 | Caused by: java.io.FileNotFoundException:
..\data\leveldb\dirty.index\000011.sst (Das System kann die angegebene Datei
nicht finden)
jvm 1 | at java.io.FileInputStream.open0(Native Method)
jvm 1 | at java.io.FileInputStream.open(Unknown Source)
jvm 1 | at java.io.FileInputStream.<init>(Unknown Source)
jvm 1 | at
org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:112)
jvm 1 | at
org.iq80.leveldb.impl.TableCache$TableAndFile.<init>(TableCache.java:102)
jvm 1 | at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:57)
jvm 1 | at org.iq80.leveldb.impl.TableCache$1.load(TableCache.java:54)
jvm 1 | at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3579)
jvm 1 | at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2372)
jvm 1 | at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2335)
jvm 1 | at
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2250)
jvm 1 | at com.google.common.cache.LocalCache.get(LocalCache.java:3980)
jvm 1 | at
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3984)
jvm 1 | at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4868)
jvm 1 | at org.iq80.leveldb.impl.TableCache.getTable(TableCache.java:80)
jvm 1 | ... 14 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
