[
https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850754#action_12850754
]
Jason Rutherglen commented on LUCENE-2324:
------------------------------------------
{quote}I don't think we should delete in FG - I suspect this'll
give net/net worse performance, due to loss of locality. It also
means you must always keep readers available, which is an
unnecessary cost for non-NRT apps. {quote}
Right, I agree it's best to buffer the deleted terms/queries...
The comments below touch on why this came up as a thought.
{quote}deletesInRAM are those deletes done during the current
segment. deletesFlushed absorbs deletesInRAM on successful
segment flush. We have to segregate the two for proper recovery
if we fail to flush the RAM buffer, eg say you hit a disk full
while flushing a new segment, and then you close your IW
successfully. We have to make sure in that case that
deletesInRAM are discarded. {quote}
Ok, this helps...
The design issue I'm running into is the buffered deletes "num
limit" which seems to be the highest doc id of DW at the time
the delete docs method is called? Then when apply deletes is
called, deletes are made only up to "num limit"? This is to
insure that documents (with for example a del term) that are
added after the delete docs call, are not deleted when apply
deletes is called, unless another call to del docs is made again
(at which point the "num limit" is set to the current DW max doc
id).
If the above is true, in order to not remap deletes on each
merge, would we need to maintain this "num limit" variable per
DW? I don't think the global remap is useful with per thread DWs
because a DW could fail, and we wouldn't know the order in
segment infos it could/would've been placed? Whereas with a
single DW, we know the new segment will be placed last in
segment infos.
We could maintain a global deleted terms/queries queue, but then
we'd also need to maintain a term -> "num limit" map per DW? It
seems a bit redundant, but maybe it's ok?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
