[
https://issues.apache.org/jira/browse/LUCENE-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766430#comment-13766430
]
caviler commented on LUCENE-3425:
---------------------------------
Because inside IndexSearcher.search method, it use thread pool to execute
Segment.search in every segments,
so, if some segment is too big, it will causes this big segment's searcher too
slow, eventually entire search method will too slow.
example, we have 2G index files.
=========================================
use AverageMergePolicy, IndexSearcher.search spent time = 1s
segment size segment.search spent time
1 500M 1s
2 500M 1s
3 500M 1s
4 500M 1s
=========================================
use other MergePolicy, IndexSearcher.search spent time = 5s
segment size segment.search spent time
1 2000M 5s
=========================================
Why not use LogByteSizeMergePolicy but AverageMergePolicy?
Because:
1. I want every semgent as small as possible!
2. I want semgent as more as possible!
we don't known how big of one segment size when entire index is growing, so
can't use LogByteSizeMergePolicy.
if use LogByteSizeMergePolicy and setMaxMergeMB(200M):
index = 200M, LogByteSizeMergePolicy = 1 segment(per 200M),
AverageMergePolicy = 4 segments(per 50M)
index = 2000M, LogByteSizeMergePolicy = 10 segment(per 200M),
AverageMergePolicy = 4 segments(per 500M)
> NRT Caching Dir to allow for exact memory usage, better buffer allocation and
> "global" cross indices control
> ------------------------------------------------------------------------------------------------------------
>
> Key: LUCENE-3425
> URL: https://issues.apache.org/jira/browse/LUCENE-3425
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 3.4, 4.0-ALPHA
> Reporter: Shay Banon
> Fix For: 5.0, 4.5
>
>
> A discussion on IRC raised several improvements that can be made to NRT
> caching dir. Some of the problems it currently has are:
> 1. Not explicitly controlling the memory usage, which can result in overusing
> memory (for example, large new segments being committed because refreshing is
> too far behind).
> 2. Heap fragmentation because of constant allocation of (probably promoted to
> old gen) byte buffers.
> 3. Not being able to control the memory usage across indices for multi index
> usage within a single JVM.
> A suggested solution (which still needs to be ironed out) is to have a
> BufferAllocator that controls allocation of byte[], and allow to return
> unused byte[] to it. It will have a cap on the size of memory it allows to be
> allocated.
> The NRT caching dir will use the allocator, which can either be provided (for
> usage across several indices) or created internally. The caching dir will
> also create a wrapped IndexOutput, that will flush to the main dir if the
> allocator can no longer provide byte[] (exhausted).
> When a file is "flushed" from the cache to the main directory, it will return
> all the currently allocated byte[] to the BufferAllocator to be reused by
> other "files".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
