[
https://issues.apache.org/jira/browse/DIRSERVER-1459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804233#action_12804233
]
Alex Karasulu commented on DIRSERVER-1459:
------------------------------------------
I've been working on this issue for some time and have found out exactly why it
is resulting. First this is a general problem that occurs irregardless of
uniqueMember attributes. It's occurring due to the inability of JDBM to
reclaim blocks used by secondary BTree objects for storing large numbers of
values for duplicate keys. The JdbmTable wrapper is designed to switch at some
configurable threshold to begin using embedded BTrees to store values instead
of serializing a TreeSet/ArraySet container data structure.
When this happens and a nested BTree is used, the space used cannot be
reclaimed by JDBM when the BTree is deleted. Plus modify operations for some
reason are deleting and adding the values causing the data to be rewritten.
This might be due to yet another issue all together in the design of the modify
operation. This will need to be investigated. However the need to free blocks
used by embedded secondary BTrees used for duplicate key value storage is a
must.
I am looking at JDBM right now to see how we can make sure this does in fact
happen properly. This is not a small matter and is critical for proper
operation. I will keep you informed as this progresses.
> Adding members to a groupOfNames results in polynomial increase in JDBM
> partition size
> --------------------------------------------------------------------------------------
>
> Key: DIRSERVER-1459
> URL: https://issues.apache.org/jira/browse/DIRSERVER-1459
> Project: Directory ApacheDS
> Issue Type: Bug
> Affects Versions: 1.5.5
> Environment: Any (tested on Linux and Mac OS X)
> Reporter: Ben Hoyt
> Assignee: Alex Karasulu
> Priority: Blocker
> Fix For: 1.5.6
>
> Attachments: DIRSERVER-1459.tar.gz, screenshot-1.jpg
>
>
> I noticed a polynomial increase JDBM partition size and therefore disk usage
> when adding users to groups in my ApacheDS instance. The vast majority of
> the usage (95+% once you hit a couple thousand users) is in
> workingDirectory/partitionId/master.db
> Further testing showed that simply adding a user is linear, as one would
> expect, and as 'apacheds-tools capacity' confirms. It is only when a user is
> made a member of a group that the JDBM partition size shoots up.
> Example statistics:
> Add 16,000 users - JDBM partition size = ~70 megabytes
> Now add those same 16,000 users to a single group (all in the same group) -
> JDBM partition size = ~19 GIGABYTES
> I'll work to attach a test case and some more numbers from my tests
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
