[
https://issues.apache.org/jira/browse/LUCENE-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13033080#comment-13033080
]
Simon Willnauer commented on LUCENE-3090:
-----------------------------------------
bq. But shouldn't stallControl kick in in that case? Ie, we stall all indexing
if the number of flush-pending DWPTs is >= the number of active DWPTs, I think?
Right so lets say we have two active thread states:
1. thread 1 starts indexing (max ram is 16M) it indexes n docs and has 15.9 MB
ram used. Now n+1 doc comes in has 5MB (active mem= 20.9M flush Mem: 0M)
2. take it out for flush (active mem=0M flush Mem: 20.9M)
3. thread 2 starts indexing and fills ram quickly ending up with 18M memory
(active mem=18M flush Mem: 20.9M)
4. take thread 2 out for flush (active mem=0M flush Mem: 38.9M)
5. thread 3 has already started indexing and reaches the RAM threshold (16M) so
we have: (active mem=16M flush Mem: 38.9M)
6. take it out for flushing (now we stall currently) (active mem=0M flush Mem:
54.9M) - this is more than 3x max ram buffer.
we currently stall at flush-pending DWPTs is > (num active DWPT + 1) we can
reduce that though but maybe we should swap back to ram based stalling?
> DWFlushControl does not take active DWPT out of the loop on fullFlush
> ---------------------------------------------------------------------
>
> Key: LUCENE-3090
> URL: https://issues.apache.org/jira/browse/LUCENE-3090
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 4.0
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Critical
> Fix For: 4.0
>
> Attachments: LUCENE-3090.patch, LUCENE-3090.patch
>
>
> We have seen several OOM on TestNRTThreads and all of them are caused by
> DWFlushControl missing DWPT that are set as flushPending but can't full due
> to a full flush going on. Yet that means that those DWPT are filling up in
> the background while they should actually be checked out and blocked until
> the full flush finishes. Even further we currently stall on the
> maxNumThreadStates while we should stall on the num of active thread states.
> I will attach a patch tomorrow.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
