[
https://issues.apache.org/jira/browse/DIRSERVER-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347048#comment-14347048
]
Emmanuel Lecharny commented on DIRSERVER-2053:
----------------------------------------------
I just found a cycle in the free page list, there may be more cycle elsewhere.
I suspect that is due to some concurrent writes, while loading the data in the
server.
Let me explain how all this works.
The {{RecordManager}} is resonspible for writing pages on disk. It also manages
{{B-trees}} (in your case, there are 18 {{B-trees}}). There are a few meta-data
that are used for that :
- the {{RecordManager Header}}
- the {{B-tree of B-trees}} aka {{BoB}}
- the {{Copied Pages B-tree}} aka {{CPB}} (not currently used)
- the free page lists
Each {{B-tree}} is using some meta-data too :
- the {{BtreeHeader}}
- the {{BtreeInfo}}
Let's get back to the {{RecordManager Header}}, which is the very first page of
the file. It contains :
{code}
* +---------------------+
* | PageSize | 4 bytes : The size of a physical page (default
to 4096)
* +---------------------+
* | NbTree | 4 bytes : The number of managed B-trees (at
least 1)
* +---------------------+
* | FirstFree | 8 bytes : The offset of the first free page
* +---------------------+
* | current BoB offset | 8 bytes : The offset of the current B-tree of
B-trees
* +---------------------+
* | previous BoB offset | 8 bytes : The offset of the previous B-tree of
B-trees
* +---------------------+
* | current CP offset | 8 bytes : The offset of the current CopiedPages
B-tree
* +---------------------+
* | previous CP offset | 8 bytes : The offset of the previous CopiedPages
B-tree
* +---------------------+
{code}
When we start the system, we will read this page. This is also the page that
gets updated (see {{RecordManager.updateRecordManagerHeader}}) every time we
write something in any B-tree (as it's a single page, it will always be written
completely on disk,so it will always reflect a consistent state. Still, we may
have intermediate states, that is the reason why we have current and previous
offsets for the BOB and CPB).
The {{FreePageList}} links all the PageIOs that can be reused when we need to
write some data. They are all linked together, newly freed pages are added at
the beginning of the list, the new first free page offset is written in the
{{RecordManager Header}}, when it's done (see {{RecordManager.free()}}).
The {{BoB}} B-tree is the 'directory' of all the existing B-trees we manage. It
contains all the existing B-trees, with all the existing revisions. Ie, if we
have a B-tree with 3 revisions, then we will have 3 elements stored in the
{{BoB}} B-tree.
This B-tree can grow quite quickly, if we don't remove older revisions that are
not anymore in use. Anyway, associated with a Tuple<Btree name, revision> is a
{{BTreeHeader}}.
The {{BTreeHeader}} data structure is simple : it's a page containing the
following informations :
{code}
* +------------+
* | revision | The B-tree revision
* +------------+
* | nbElems | The B-tree number of elements
* +------------+
* | rootPage | The root page offset
* +------------+
* | BtreeInfo | The B-tree info offset
* +------------+
{code}
The {{BtreeInfo}} is not really interesting, it contains static info about a
given B-tree, like its name, the key and value serializers FQCN and such
things. In any case, it's written once (when a B-tree is added), and read once
(when we initialize a given B-tree), but never ever modified.
The {{RootPage}} is the top level page of a B-tree. It can be a {{Node}} or a
{{Leaf}}. If it's a {{Node}}, then obviously it has children.
Let's go a bit deeper now. The file is split in pages which size is fixed
(default to 512bytes). That means we can only write only 512 bytes in it, so
when we have to write more, then we have to use more than one page. We have the
possibility to link pages together to make it possible to store more than
512bytes. The {{PageIO}} class is used to store the physical pages, the first 8
bytes is a offset to the next page, or -1 ({{NO_PAGE}}) if this is the last
page. The very first page also contain the size of the data that has been put
in all those linked pages (from 0 to 2G bytes of possible data...).
> Sometimes apacheds will throw OutOfMemoryException
> --------------------------------------------------
>
> Key: DIRSERVER-2053
> URL: https://issues.apache.org/jira/browse/DIRSERVER-2053
> Project: Directory ApacheDS
> Issue Type: Bug
> Components: ldap
> Affects Versions: 2.0.0-M19
> Environment: CentOS 6.5 with apacheds M19 installed.
> Reporter: linzhao
> Priority: Blocker
>
> After injecting some data into apacheds and restart it. Sometimes the
> apacheds can't be started, it will throw OutOfMemoryException. The data
> entries will less than 600. Exception list below:
> INFO | jvm 1 | 2015/03/04 23:52:09 | Dumping heap to
> /opt/polycom/apacheds-dumps/java_pid24024.hprof ...
> INFO | jvm 1 | 2015/03/04 23:52:19 | Heap dump file created [2094548383
> bytes in 9.597 secs]
> INFO | jvm 1 | 2015/03/04 23:52:19 | Error in WrapperListener.start
> callback. java.lang.OutOfMemoryError: GC overhead limit exceeded
> INFO | jvm 1 | 2015/03/04 23:52:19 | java.lang.OutOfMemoryError: GC
> overhead limit exceeded
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.mavibot.btree.RecordManager.fetchPage(RecordManager.java:3045)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.mavibot.btree.RecordManager.readPageIOs(RecordManager.java:797)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.mavibot.btree.RecordManager.deserialize(RecordManager.java:987)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.mavibot.btree.PersistedPageHolder.fetchElement(PersistedPageHolder.java:133)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.mavibot.btree.PersistedPageHolder.getValue(PersistedPageHolder.java:113)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.mavibot.btree.AbstractPage.get(AbstractPage.java:252)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.mavibot.btree.AbstractBTree.get(AbstractBTree.java:505)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.mavibot.btree.PersistedBTree.get(PersistedBTree.java:43)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.partition.impl.btree.mavibot.MavibotTable.get(MavibotTable.java:317)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.partition.impl.btree.mavibot.MavibotIndex.forwardLookup(MavibotIndex.java:305)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.partition.impl.btree.mavibot.MavibotIndex.forwardLookup(MavibotIndex.java:58)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.getEntryId(AbstractBTreePartition.java:2473)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.xdbm.search.impl.DefaultSearchEngine.computeResult(DefaultSearchEngine.java:123)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.partition.impl.btree.AbstractBTreePartition.search(AbstractBTreePartition.java:1141)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.shared.partition.DefaultPartitionNexus.search(DefaultPartitionNexus.java:624)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.shared.ReferralManagerImpl.init(ReferralManagerImpl.java:178)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.shared.ReferralManagerImpl.<init>(ReferralManagerImpl.java:86)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.referral.ReferralInterceptor.init(ReferralInterceptor.java:213)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.DefaultDirectoryService.initInterceptors(DefaultDirectoryService.java:685)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.DefaultDirectoryService.initialize(DefaultDirectoryService.java:1818)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.core.DefaultDirectoryService.startup(DefaultDirectoryService.java:1244)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.ApacheDsService.initDirectoryService(ApacheDsService.java:323)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.ApacheDsService.start(ApacheDsService.java:182)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.apache.directory.server.wrapper.ApacheDsTanukiWrapper.start(ApacheDsTanukiWrapper.java:72)
> INFO | jvm 1 | 2015/03/04 23:52:19 | at
> org.tanukisoftware.wrapper.WrapperManager$12.run(WrapperManager.java:2788)
> STATUS | wrapper | 2015/03/04 23:52:20 | <-- Wrapper Stopped
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
