Greetings,
We are using ApacheDS as LDAP provider for one of our Solutions (for user
authentication).
Recently one of our ApacheDS Servers experienced a database corruption - we are
not sure why.
The ApacheDS server booted up correctly, but when some client tried to retrieve
a list of all users (for instance the Apache Directory Studio), the Server
throws a exception and closes the socket to the client:
[05:40:05] WARN [org.apache.directory.server.ldap.LdapProtocolHandler] -
Unexpected exception forcing session to close: sending disconnect notice to
client.
java.lang.Error: ERR_546 CRITICAL: page header magic for block 64 not OK 0
at jdbm.recman.PageHeader.<init>(PageHeader.java:95)
at jdbm.recman.PageHeader.getView(PageHeader.java:124)
at jdbm.recman.PageManager.getNext(PageManager.java:234)
at jdbm.recman.PageCursor.next(PageCursor.java:104)
at jdbm.recman.PhysicalRowIdManager.fetch(PhysicalRowIdManager.java:158)
at jdbm.recman.BaseRecordManager.fetch(BaseRecordManager.java:324)
at jdbm.recman.CacheRecordManager.fetch(CacheRecordManager.java:262)
at jdbm.btree.BPage.loadBPage(BPage.java:899)
at jdbm.btree.BPage.childBPage(BPage.java:890)
at jdbm.btree.BPage.find(BPage.java:284)
at jdbm.btree.BTree.find(BTree.java:408)
at
org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmTable.get(JdbmTable.java:395)
at
org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmMasterTable.get(JdbmMasterTable.java:155)
at
org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmStore.lookup(JdbmStore.java:1332)
at
org.apache.directory.server.core.partition.impl.btree.jdbm.JdbmStore.lookup(JdbmStore.java:70)
at
org.apache.directory.server.xdbm.AbstractXdbmPartition.lookup(AbstractXdbmPartition.java:327)
at
org.apache.directory.server.core.partition.impl.btree.ServerEntryCursorAdaptor.get(ServerEntryCursorAdaptor.java:139)
at
org.apache.directory.server.core.partition.impl.btree.ServerEntryCursorAdaptor.get(ServerEntryCursorAdaptor.java:39)
at
org.apache.directory.server.core.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:503)
at
org.apache.directory.server.ldap.handlers.SearchHandler.readResults(SearchHandler.java:314)
at
org.apache.directory.server.ldap.handlers.SearchHandler.doSimpleSearch(SearchHandler.java:749)
at
org.apache.directory.server.ldap.handlers.SearchHandler.handleIgnoringReferrals(SearchHandler.java:978)
at
org.apache.directory.server.ldap.handlers.SearchHandler.handleWithReferrals(SearchHandler.java:1054)
at
org.apache.directory.server.ldap.handlers.SearchHandler.handleWithReferrals(SearchHandler.java:78)
at
org.apache.directory.server.ldap.handlers.ReferralAwareRequestHandler.handle(ReferralAwareRequestHandler.java:94)
at
org.apache.directory.server.ldap.handlers.ReferralAwareRequestHandler.handle(ReferralAwareRequestHandler.java:57)
at
org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:208)
at
org.apache.directory.server.ldap.handlers.LdapRequestHandler.handleMessage(LdapRequestHandler.java:58)
at
org.apache.mina.handler.demux.DemuxingIoHandler.messageReceived(DemuxingIoHandler.java:232)
at
org.apache.directory.server.ldap.LdapProtocolHandler.messageReceived(LdapProtocolHandler.java:193)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:713)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:434)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:46)
at
org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:793)
at
org.apache.mina.core.filterchain.IoFilterEvent.fire(IoFilterEvent.java:71)
at org.apache.mina.core.session.IoEvent.run(IoEvent.java:63)
at
org.apache.mina.filter.executor.UnorderedThreadPoolExecutor$Worker.runTask(UnorderedThreadPoolExecutor.java:480)
at
org.apache.mina.filter.executor.UnorderedThreadPoolExecutor$Worker.run(UnorderedThreadPoolExecutor.java:434)
at java.lang.Thread.run(Thread.java:662)
[05:40:05] WARN [org.apache.directory.server.ldap.LdapProtocolHandler] -
Unexpected exception forcing session to close: sending disconnect notice to
client.
org.apache.mina.core.write.WriteToClosedSessionException
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.clearWriteRequestQueue(AbstractPollingIoProcessor.java:573)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.removeNow(AbstractPollingIoProcessor.java:525)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.removeSessions(AbstractPollingIoProcessor.java:497)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:61)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:974)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
We do not think that there is a way to repair this corruption, therefore we
tried to restore a backup (we are using Tivoli for backup of our SAN storage).
We were able to determine the Point in Time when the corruption took place, as
we use a oracle database job which fails in case the LDAP query fails.
After trying to restore to the point in time when the Database was still
working we experienced a new Problem:
The ApacheDS Server fails to boot up - it crashes with the following Stack
trace:
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version
information : assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version
information : assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version
information : assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version
information : assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version
information : assuming version: 1
[11:27:58] WARN [org.apache.directory.shared.ldap.ldif.LdifReader] - No version
information : assuming version: 1
[11:27:59] ERROR [org.apache.directory.daemon.Bootstrapper] - Failed on
null.init(InstallationLayout, String[])
java.lang.NullPointerException
at
org.apache.directory.server.core.entry.ClonedServerEntry.<init>(ClonedServerEntry.java:67)
at
org.apache.directory.server.xdbm.AbstractXdbmPartition.lookup(AbstractXdbmPartition.java:327)
at
org.apache.directory.server.core.partition.impl.btree.ServerEntryCursorAdaptor.get(ServerEntryCursorAdaptor.java:139)
at
org.apache.directory.server.core.partition.impl.btree.ServerEntryCursorAdaptor.get(ServerEntryCursorAdaptor.java:39)
at
org.apache.directory.server.core.filtering.BaseEntryFilteringCursor.next(BaseEntryFilteringCursor.java:503)
at
org.apache.directory.server.core.authz.GroupCache.initialize(GroupCache.java:147)
at
org.apache.directory.server.core.authz.GroupCache.<init>(GroupCache.java:111)
at
org.apache.directory.server.core.authz.AciAuthorizationInterceptor.init(AciAuthorizationInterceptor.java:202)
at
org.apache.directory.server.core.interceptor.InterceptorChain.register0(InterceptorChain.java:442)
at
org.apache.directory.server.core.interceptor.InterceptorChain.register(InterceptorChain.java:398)
at
org.apache.directory.server.core.interceptor.InterceptorChain.init(InterceptorChain.java:258)
at
org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1447)
at
org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:907)
at
org.apache.directory.server.configuration.ApacheDS.startup(ApacheDS.java:163)
at org.apache.directory.server.Service.initLdap(Service.java:136)
at org.apache.directory.server.Service.init(Service.java:77)
at
org.apache.directory.daemon.Bootstrapper.callInit(Bootstrapper.java:154)
at
org.apache.directory.daemon.TanukiBootstrapper.start(TanukiBootstrapper.java:54)
at
org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)
I failed to archive any information in the internet about this szenario.
Does someone know what the problem is?
May the Tivoli backup restore some inconsistent state?
Is there any chance to rescue the data of this LDAP server?
Good thing it was no Productive environment this time - only QA, but it would
be great to solve this in order to know how to solve it the next time.. ;-) (if
there is a way to solve it)
Some additional information about our Environment: We are using Red Head
Enterprise Linux release 5.8 (x86_64) on the impacted server.
ApacheDS is running on the embedded Java Virtual Machine
Mit freundlichen Grüßen / Best regards
Daniel van Maren
Bosch Software Innovations GmbH
Stuttgarter Str. 130,
71332 Waiblingen
GERMANY
www.bosch-si.de
daniel.vanma...@bosch-si.com<mailto:nikolaj.lang...@bosch-si.com>
Registered office: Immenstaad, Register court: Amtsgericht Ulm, HRB 631888;
Executives: Heinz Derenbach; Erica Fölsche, Thomas Cotic, Michael Hahn, Klaus
Hüftle
