[
https://issues.apache.org/jira/browse/LUCENE-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12857381#action_12857381
]
Michael McCandless commented on LUCENE-2324:
--------------------------------------------
{quote}
i would love to be able to explicitly define a "segment" affinity for documents
i'm feeding
this would then allow me to say:
all docs from table a has affinity 1
all docs from table b has affinity 2
{quote}
Right, this is exactly what affinity would be good for -- so IW would
try to send "table a" docs their own DW(s) and "table b" docs to their
own DW(s), which should give faster indexing than randomly binding to
DWs.
But:
bq. this would ideally result in indexing documents from each table into a
different segment (obviously, i would then need to be able to have segment
merging be affinity aware so optimize/merging would only merge segments that
share an affinity)
This part I was not proposing :)
The affinity would just be an optimization hint in creating the
initial flushed segments, so IW can speed up indexing.
Probably if you really want to keep the segments segregated like that,
you should in fact index to separate indices?
> Per thread DocumentsWriters that write their own private segments
> -----------------------------------------------------------------
>
> Key: LUCENE-2324
> URL: https://issues.apache.org/jira/browse/LUCENE-2324
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Index
> Reporter: Michael Busch
> Assignee: Michael Busch
> Priority: Minor
> Fix For: 3.1
>
> Attachments: lucene-2324.patch, LUCENE-2324.patch
>
>
> See LUCENE-2293 for motivation and more details.
> I'm copying here Mike's summary he posted on 2293:
> Change the approach for how we buffer in RAM to a more isolated
> approach, whereby IW has N fully independent RAM segments
> in-process and when a doc needs to be indexed it's added to one of
> them. Each segment would also write its own doc stores and
> "normal" segment merging (not the inefficient merge we now do on
> flush) would merge them. This should be a good simplification in
> the chain (eg maybe we can remove the *PerThread classes). The
> segments can flush independently, letting us make much better
> concurrent use of IO & CPU.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-dev-h...@lucene.apache.org
