Dag, Thanks for the quick response. The db still fails to boot with the non-debug jar, so the consistency check is unfortunately a no-go. So it's seeming more and more like a derby bug. Also double checked the disk on the machine where this failed, seemed healthy.
I've filed DERBY-5248 and attached the db to it. Thanks, Tim Wu On May 26, 2011, at 3:53 PM, Dag H. Wanvik wrote: > > Hi Tim, > > you could file a new JIRA issue and attach the database as a zip file if > it's not too large. > > http://db.apache.org/derby/DerbyBugGuidelines.html > > I see you use the debug version of Derby. In this particular case, it > would be interesting to know what happens if you run Derby with the > non-debug version, since it would run without the assert check that hit > you. If you are able to boot then (please save the database first), you > should run a full consistency check on the database, see > http://wiki.apache.org/db-derby/DatabaseConsistencyCheck. > > Should this work out OK, it could mean that the assert was wrong. Maybe > not very likely, I am not familiar with this code, but worth a try. If > it crashes or fails the consistency checking it sounds like a Derby bug > to me. > > Thanks, > Dag > > > > Tim Wu <[email protected]> writes: > >> Hi, >> >> During some crash tests for our product with embedded derby, we hit a >> situation where after the crash, derby's db appears to have become >> corrupted. There aren't logs as to what was going on at the exact >> moment of the crash, but the db fails to boot with the following >> error. This test was done using Derby 10.8.1.2, and we were unable to >> reproduce the exact problem again. Seems like it could be a problem if >> it were possible to cause Derby's DB to become corrupted just by >> bringing down the process. Is this a Derby bug? >> >> I do have the offending db zipped up, just not quite sure how to share >> it with the mailing list. Let me know the preferred method, and I'll >> send it across. >> >> org.apache.derby.shared.common.sanity.AssertFailure: ASSERT FAILED >> incorrect deleted row count. Should be: 903, instead got: 902, maxSlot >> = 1255, recordCount = 1255 >> at >> org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162) >> at >> org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147) >> at >> org.apache.derby.impl.store.raw.data.BasePage.internalNonDeletedRecordCount(BasePage.java:1432) >> at >> org.apache.derby.impl.store.raw.data.CachedPage.releaseExclusive(CachedPage.java:488) >> >> at >> org.apache.derby.impl.store.raw.data.StoredPage.releaseExclusive(StoredPage.java:1066) >> >> at org.apache.derby.impl.store.raw.data.BasePage.unlatch(BasePage.java:1370) >> at >> org.apache.derby.impl.store.raw.data.PageBasicOperation.releaseResource(PageBasicOperation.java:195) >> at >> org.apache.derby.impl.store.raw.data.PhysicalUndoOperation.releaseResource(PhysicalUndoOperation.java:177) >> at org.apache.derby.impl.store.raw.log.FileLogger.undo(FileLogger.java:1055) >> at org.apache.derby.impl.store.raw.xact.Xact.abort(Xact.java:952) >> at >> org.apache.derby.impl.store.raw.xact.XactFactory.rollbackAllTransactions(XactFactory.java:547) >> at >> org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:1229) >> at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:339) >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> at >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:531) >> >> at >> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) >> >> at >> org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> at >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:531) >> >> at >> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) >> >> at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:749) >> at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:177) >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> at >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1816) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1682) >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1560) >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:979) >> at >> org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:550) >> at >> org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2697) >> >> at >> org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:385) >> at >> org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) >> >> at >> org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) >> >> at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) >> at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) >> at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) >> at java.sql.DriverManager.getConnection(DriverManager.java:582) >> at java.sql.DriverManager.getConnection(DriverManager.java:154) >> at OpenDerby.main(OpenDerby.java:17) >> --------------- >> Stack traces for all live threads: >> Thread name=main id=1 priority=5 state=RUNNABLE isdaemon=false >> java.lang.Thread.dumpThreads(Native Method) >> java.lang.Thread.getAllStackTraces(Thread.java:1554) >> org.apache.derby.shared.common.sanity.ThreadDump.getStackDumpString(ThreadDump.java:34) >> >> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> java.lang.reflect.Method.invoke(Method.java:597) >> org.apache.derby.shared.common.sanity.AssertFailure$1.run(AssertFailure.java:165) >> >> java.security.AccessController.doPrivileged(Native Method) >> org.apache.derby.shared.common.sanity.AssertFailure.dumpThreads(AssertFailure.java:159) >> >> org.apache.derby.shared.common.sanity.AssertFailure.<init>(AssertFailure.java:72) >> >> org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162) >> >> org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147) >> >> org.apache.derby.impl.store.raw.data.BasePage.internalNonDeletedRecordCount(BasePage.java:1432) >> org.apache.derby.impl.store.raw.data.CachedPage.releaseExclusive(CachedPage.java:488) >> >> org.apache.derby.impl.store.raw.data.StoredPage.releaseExclusive(StoredPage.java:1066) >> >> org.apache.derby.impl.store.raw.data.BasePage.unlatch(BasePage.java:1370) >> org.apache.derby.impl.store.raw.data.PageBasicOperation.releaseResource(PageBasicOperation.java:195) >> org.apache.derby.impl.store.raw.data.PhysicalUndoOperation.releaseResource(PhysicalUndoOperation.java:177) >> org.apache.derby.impl.store.raw.log.FileLogger.undo(FileLogger.java:1055) >> org.apache.derby.impl.store.raw.xact.Xact.abort(Xact.java:952) >> org.apache.derby.impl.store.raw.xact.XactFactory.rollbackAllTransactions(XactFactory.java:547) >> org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:1229) >> org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:339) >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:531) >> >> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) >> >> org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019) >> >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:531) >> >> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) >> >> org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:749) >> org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:177) >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1816) >> >> org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1682) >> org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1560) >> org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:979) >> org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:550) >> >> org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2697) >> >> org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:385) >> org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) >> >> org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) >> >> org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) >> org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) >> org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) >> java.sql.DriverManager.getConnection(DriverManager.java:582) >> java.sql.DriverManager.getConnection(DriverManager.java:154) >> OpenDerby.main(OpenDerby.java:17) >> >> Thread name=derby.rawStoreDaemon id=12 priority=5 state=TIMED_WAITING >> isdaemon=true >> java.lang.Object.wait(Native Method) >> org.apache.derby.impl.services.daemon.BasicDaemon.rest(BasicDaemon.java:576) >> org.apache.derby.impl.services.daemon.BasicDaemon.run(BasicDaemon.java:390) >> java.lang.Thread.run(Thread.java:680) >> >> Thread name=Finalizer id=3 priority=8 state=WAITING isdaemon=true >> java.lang.Object.wait(Native Method) >> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) >> java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) >> java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) >> >> Thread name=Reference Handler id=2 priority=10 state=WAITING isdaemon=true >> java.lang.Object.wait(Native Method) >> java.lang.Object.wait(Object.java:485) >> java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) >> >> Thread name=Signal Dispatcher id=6 priority=9 state=RUNNABLE isdaemon=true >> >> Thread name=Timer-0 id=10 priority=5 state=WAITING isdaemon=true >> java.lang.Object.wait(Native Method) >> java.lang.Object.wait(Object.java:485) >> java.util.TimerThread.mainLoop(Timer.java:483) >> java.util.TimerThread.run(Timer.java:462) >> >> >> --------------- >> >> Exception in thread "main" java.sql.SQLException: Failed to start >> database >> /Users/tim/Downloads/LinkedBlockingQueueCrashTest/objectdb/datadb' >> with class loader sun.misc.Launcher$AppClassLoader@40affc70, see the >> next exception for details. >> at >> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:98) >> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) >> at org.apache.derby.impl.jdbc.Util.seeNextException(Util.java:278) >> at >> org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2736) >> >> at >> org.apache.derby.impl.jdbc.EmbedConnection.<init>(EmbedConnection.java:385) >> at >> org.apache.derby.impl.jdbc.EmbedConnection30.<init>(EmbedConnection30.java:73) >> >> at >> org.apache.derby.impl.jdbc.EmbedConnection40.<init>(EmbedConnection40.java:51) >> >> at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Driver40.java:70) >> at org.apache.derby.jdbc.InternalDriver.connect(InternalDriver.java:248) >> at org.apache.derby.jdbc.AutoloadedDriver.connect(AutoloadedDriver.java:144) >> at java.sql.DriverManager.getConnection(DriverManager.java:582) >> at java.sql.DriverManager.getConnection(DriverManager.java:154) >> at OpenDerby.main(OpenDerby.java:17) >> Caused by: java.sql.SQLException: Failed to start database >> /Users/tim/Downloads/LinkedBlockingQueueCrashTest/objectdb/datadb' >> with class loader sun.misc.Launcher$AppClassLoader@40affc70, see the >> next exception for details. >> at >> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) >> at >> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) >> at >> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) >> ... 12 more >> Caused by: java.sql.SQLException: Java exception: 'ASSERT FAILED >> incorrect deleted row count. Should be: 903, instead got: 902, maxSlot >> = 1255, recordCount = 1255: >> org.apache.derby.shared.common.sanity.AssertFailure'. >> at >> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(SQLExceptionFactory.java:45) >> at >> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(SQLExceptionFactory40.java:122) >> at >> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(SQLExceptionFactory40.java:71) >> at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Util.java:142) >> at org.apache.derby.impl.jdbc.Util.javaException(Util.java:299) >> at >> org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2732) >> >> ... 9 more >> Caused by: org.apache.derby.shared.common.sanity.AssertFailure: ASSERT >> FAILED incorrect deleted row count. Should be: 903, instead got: 902, >> maxSlot = 1255, recordCount = 1255 >> at >> org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:162) >> at >> org.apache.derby.shared.common.sanity.SanityManager.THROWASSERT(SanityManager.java:147) >> at >> org.apache.derby.impl.store.raw.data.BasePage.internalNonDeletedRecordCount(BasePage.java:1432) >> at >> org.apache.derby.impl.store.raw.data.CachedPage.releaseExclusive(CachedPage.java:488) >> >> at >> org.apache.derby.impl.store.raw.data.StoredPage.releaseExclusive(StoredPage.java:1066) >> >> at org.apache.derby.impl.store.raw.data.BasePage.unlatch(BasePage.java:1370) >> at >> org.apache.derby.impl.store.raw.data.PageBasicOperation.releaseResource(PageBasicOperation.java:195) >> at >> org.apache.derby.impl.store.raw.data.PhysicalUndoOperation.releaseResource(PhysicalUndoOperation.java:177) >> at org.apache.derby.impl.store.raw.log.FileLogger.undo(FileLogger.java:1055) >> at org.apache.derby.impl.store.raw.xact.Xact.abort(Xact.java:952) >> at >> org.apache.derby.impl.store.raw.xact.XactFactory.rollbackAllTransactions(XactFactory.java:547) >> at >> org.apache.derby.impl.store.raw.log.LogToFile.recover(LogToFile.java:1229) >> at org.apache.derby.impl.store.raw.RawStore.boot(RawStore.java:339) >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> at >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:531) >> >> at >> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) >> >> at >> org.apache.derby.impl.store.access.RAMAccessManager.boot(RAMAccessManager.java:1019) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> at >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.startModule(BaseMonitor.java:531) >> >> at >> org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Monitor.java:427) >> >> at org.apache.derby.impl.db.BasicDatabase.bootStore(BasicDatabase.java:749) >> at org.apache.derby.impl.db.BasicDatabase.boot(BasicDatabase.java:177) >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.boot(BaseMonitor.java:1982) >> >> at >> org.apache.derby.impl.services.monitor.TopService.bootModule(TopService.java:334) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.bootService(BaseMonitor.java:1816) >> >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(BaseMonitor.java:1682) >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(BaseMonitor.java:1560) >> at >> org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(BaseMonitor.java:979) >> at >> org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Monitor.java:550) >> at >> org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(EmbedConnection.java:2697) >> >> ... 9 more >> >> >> Thanks, >> Tim Wu
