[
https://issues.apache.org/jira/browse/LUCENE-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791039#action_12791039
]
Michael McCandless commented on LUCENE-2164:
--------------------------------------------
{quote}
I think we discussed IO throttling at the directory level
before. A controller could be passed to createOutput that has a
setPriority method. The priority level would be set based on the
demand for IO from IndexWriter. SegmentMerger could know if it's
creating a large segment from the sizes of the source segments,
and set the IO priority accordingly?
{quote}
Yeah we talked about this for the more general case (throttling
merging when searches are running). That's a pretty big change
though.
I think for this case we could handle the throttling entirely in CMS,
by wrapping the OneMerge and overriding the checkAborted method to
pause the large merge when smaller merge(s) are running. This would
force the long merges to stop so that the smaller merges could
complete without fighting for IO.
To get a sense of how important this would be, I think we need to
measure how long it takes a merge to complete when it's alone vs when
another merge is running at the same time, on a machine with enough
cores, for a "typical" IO system. If they do cause one another to
thrash (seems likely, though, I've also seen evidence that merging is
CPU bound, at least for merging the postings...) then we should only
allow 1 to run at once...
> Make CMS smarter about thread priorities
> ----------------------------------------
>
> Key: LUCENE-2164
> URL: https://issues.apache.org/jira/browse/LUCENE-2164
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2164.patch
>
>
> Spinoff from LUCENE-2161...
> The hard throttling CMS does (blocking the incoming thread that wants
> to launch a new merge) can be devastating when it strikes during NRT
> reopen.
> It can easily happen if a huge merge is off and running, but then a
> tiny merge is needed to clean up recently created segments due to
> frequent reopens.
> I think a small change to CMS, whereby it assigns a higher thread
> priority to tiny merges than big merges, should allow us to increase
> the max merge thread count again, and greatly reduce the chance that
> NRT's reopen would hit this.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
