[
https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850864#action_12850864
]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
bq. Mike, can you explain what the advantages of this kind of thread affinity
are? I was always wondering why the DocumentsWriter code currently makes
efforts to assign a ThreadState always to the same Thread? Is that being done
for performance reasons?
It's for performance. I expect there are apps where a given
thread/pool indexes certain kind of docs, ie, the app threads
themselves have "affinity" for docs with similar term distributions.
In which case, it's best (most RAM efficient) if those docs w/
presumably similar term stats are sent back to the same DW. If you
mix in different term stats into one buffer you get worse RAM
efficiency.
Also, for better RAM efficiency you want *fewer* DWs... because we get
more RAM efficiency the higher the freq of the terms... but of course
you want more DWs for better CPU efficiency whenever that many threads
are running at once.
Net/net CPU efficiency should trump RAM efficiency, I think, so if
there is a conflict we should favor CPU efficiency.
Though, thread affinity doesn't seem that CPU costly to implement?
Lookup the DW your thread first used... if it's free, seize it. If
it's not, fallback to any DW that's free.
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
