Michael McCandless created LUCENE-7868:
------------------------------------------
Summary: Use multiple threads to apply deletes and DV updates
Key: LUCENE-7868
URL: https://issues.apache.org/jira/browse/LUCENE-7868
Project: Lucene - Core
Issue Type: Improvement
Reporter: Michael McCandless
Assignee: Michael McCandless
Fix For: master (7.0)
Today, when users delete documents or apply doc values updates, IndexWriter
buffers them up into frozen packets and then eventually uses a single thread
(BufferedUpdatesStream.applyDeletesAndUpdates) to resolve delete/update terms
to docids. This thread also holds IW's monitor lock, so it also blocks
refresh, merges starting/finishing, commits, etc.
We have heavily optimized this part of Lucene over time, e.g. LUCENE-6161,
LUCENE-2897, LUCENE-2680, LUCENE-3342, but still, it's a single thread so it
can't use multiple CPU cores commonly available now.
This doesn't affect append-only usage, but for update-heavy users (me!) this
can be a big bottleneck, and causes long stop-the-world hangs during indexing.
I have an initial exploratory patch to make these lookups concurrent, without
holding IW's lock, so that when a new packet of deletes is pushed, which
happens when we flush a new segment, we immediately use that same indexing
thread to and resolve the deletions.
This is analogous to when we made segment flushing concurrent (LUCENE-3023),
just for deletes and DV updates as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
