[
https://issues.apache.org/jira/browse/LUCENE-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984818#comment-15984818
]
ASF subversion and git services commented on LUCENE-7792:
---------------------------------------------------------
Commit 25f1dd2f9b2a984cc59202fe56f7a8860f514657 in lucene-solr's branch
refs/heads/master from Mike McCandless
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=25f1dd2 ]
LUCENE-7792: add try/finally to make sure semaphore is released on exceptions
> Add optional concurrency to OfflineSorter
> -----------------------------------------
>
> Key: LUCENE-7792
> URL: https://issues.apache.org/jira/browse/LUCENE-7792
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael McCandless
> Assignee: Michael McCandless
> Fix For: master (7.0), 6.6
>
> Attachments: LUCENE-7792.patch, LUCENE-7792.patch, LUCENE-7792.patch
>
>
> OfflineSorter is a heavy operation and is really an embarrassingly concurrent
> problem at heart, and if you have enough hardware concurrency (e.g. fast
> SSDs, multiple CPU cores) it can be a big speedup.
> E.g., after reading a partition from the input, one thread can sort and write
> it, while another thread reads the next partition, etc. Merging partitions
> can also be done in the background. Some things still cannot be concurrent,
> e.g. the initial read from the input must be a single thread, as well as the
> final merge and writing to the final output.
> I think I found a fairly non-invasive way to add optional concurrency to this
> class, by adding an optional ExecutorService to OfflineSorter's ctor (similar
> to IndexSearcher) and using futures to represent each partition as we sort,
> and creating Callable classes for sorting and merging partitions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
