[
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16697846#comment-16697846
]
Toke Eskildsen commented on SOLR-13013:
---------------------------------------
I have uploaded a proof of concept for the idea in the issue description. The
structure that collects and holds the temporary values is made by mashing the
keyboard until it worked and the performance test is Frankensteined from
existing unit-test code in TestExportWriter. Nevertheless unit-tests in
TestExportWriter passes and a performance test can be executed with
{{TES_SIZES="1000,10000,100000,200000,300000" ant -Dtests.heapsize=5g
-Dtests.codec=Lucene80 -Dtestmethod=testExportSpeed -Dtestcase=TestExportWriter
test | grep "TES:"}}
It takes a 10+ minutes and writes a summary at the end, For a quicker test, use
TES_SIZES="1000,10000" or something like that. For my desktop the result was
{{[junit4] 1> TES: Concatenated output:}}
{{ [junit4] 1> TES: Test 1/5: 1000 documents, trie: 11098 / 7525 docs/sec (
147%), points: 7639 / 11552 docs/sec ( 66%)}}
{{ [junit4] 1> TES: Test 2/5: 1000 documents, trie: 15135 / 9269 docs/sec (
163%), points: 27769 / 15986 docs/sec ( 174%)}}
{{ [junit4] 1> TES: Test 3/5: 1000 documents, trie: 11505 / 9593 docs/sec (
120%), points: 37643 / 13584 docs/sec ( 277%)}}
{{ [junit4] 1> TES: Test 4/5: 1000 documents, trie: 17495 / 9730 docs/sec (
180%), points: 39103 / 18222 docs/sec ( 215%)}}
{{ [junit4] 1> TES: Test 5/5: 1000 documents, trie: 17657 / 10331 docs/sec (
171%), points: 37633 / 19104 docs/sec ( 197%)}}
{{ [junit4] 1> TES: ------------------}}
{{ [junit4] 1> TES: Test 1/5: 10000 documents, trie: 17018 / 7901 docs/sec (
215%), points: 38606 / 12381 docs/sec ( 312%)}}
{{ [junit4] 1> TES: Test 2/5: 10000 documents, trie: 17191 / 7879 docs/sec (
218%), points: 39920 / 12404 docs/sec ( 322%)}}
{{ [junit4] 1> TES: Test 3/5: 10000 documents, trie: 17218 / 7881 docs/sec (
218%), points: 41696 / 12410 docs/sec ( 336%)}}
{{ [junit4] 1> TES: Test 4/5: 10000 documents, trie: 17451 / 7884 docs/sec (
221%), points: 41719 / 12360 docs/sec ( 338%)}}
{{ [junit4] 1> TES: Test 5/5: 10000 documents, trie: 17227 / 7855 docs/sec (
219%), points: 41879 / 12436 docs/sec ( 337%)}}
{{ [junit4] 1> TES: ------------------}}
{{ [junit4] 1> TES: Test 1/5: 100000 documents, trie: 15849 / 3718 docs/sec (
426%), points: 36037 / 4841 docs/sec ( 744%)}}
{{ [junit4] 1> TES: Test 2/5: 100000 documents, trie: 16348 / 3717 docs/sec (
440%), points: 37994 / 4858 docs/sec ( 782%)}}
{{ [junit4] 1> TES: Test 3/5: 100000 documents, trie: 15378 / 3718 docs/sec (
414%), points: 38831 / 4872 docs/sec ( 797%)}}
{{ [junit4] 1> TES: Test 4/5: 100000 documents, trie: 16042 / 3710 docs/sec (
432%), points: 39084 / 4876 docs/sec ( 802%)}}
{{ [junit4] 1> TES: Test 5/5: 100000 documents, trie: 16009 / 3713 docs/sec (
431%), points: 39503 / 4865 docs/sec ( 812%)}}
{{ [junit4] 1> TES: ------------------}}
{{ [junit4] 1> TES: Test 1/5: 200000 documents, trie: 15403 / 3031 docs/sec (
508%), points: 37349 / 3531 docs/sec (1058%)}}
{{ [junit4] 1> TES: Test 2/5: 200000 documents, trie: 15853 / 3018 docs/sec (
525%), points: 37509 / 3544 docs/sec (1058%)}}
{{ [junit4] 1> TES: Test 3/5: 200000 documents, trie: 14993 / 3018 docs/sec (
497%), points: 38468 / 3547 docs/sec (1084%)}}
{{ [junit4] 1> TES: Test 4/5: 200000 documents, trie: 15191 / 3023 docs/sec (
502%), points: 38684 / 3538 docs/sec (1093%)}}
{{ [junit4] 1> TES: Test 5/5: 200000 documents, trie: 15678 / 3035 docs/sec (
517%), points: 38729 / 3542 docs/sec (1093%)}}
{{ [junit4] 1> TES: ------------------}}
{{ [junit4] 1> TES: Test 1/5: 300000 documents, trie: 15529 / 2834 docs/sec (
548%), points: 36911 / 3652 docs/sec (1011%)}}
{{ [junit4] 1> TES: Test 2/5: 300000 documents, trie: 15455 / 2846 docs/sec (
543%), points: 37705 / 3630 docs/sec (1039%)}}
{{ [junit4] 1> TES: Test 3/5: 300000 documents, trie: 15805 / 2866 docs/sec (
551%), points: 37583 / 3660 docs/sec (1027%)}}
{{ [junit4] 1> TES: Test 4/5: 300000 documents, trie: 15653 / 2883 docs/sec (
543%), points: 39365 / 3591 docs/sec (1096%)}}
{{ [junit4] 1> TES: Test 5/5: 300000 documents, trie: 15736 / 2895 docs/sec (
543%), points: 38606 / 3667 docs/sec (1053%)}}
The two numbers for trie and points are sorted followed by non_sorted. The
numbers in the parentheses are sorted/non_sorted. As can be seen, non_sorted
export performance degrades as index size (measured in number of documents)
goes up. Also, as can be seen from the percentages, reusing the
DocValues-iterators and ensuring docID order improved the speed significantly,
The patch is not at alll production-ready. See it as a "is this idea worth
exploring?". Ping to [~joel.bernstein], as I expect he will be interested in
this.
> Change export to extract DocValues in docID order
> -------------------------------------------------
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Export Writer
> Affects Versions: 7.5, master (8.0)
> Reporter: Toke Eskildsen
> Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for
> paging through the result set in a given sort order. Each time a window has
> been calculated, the values for the export fields are retrieved from the
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support
> random access. The current export implementation bypasses this by creating a
> new DocValues-iterator for each individual value to retrieve. This slows down
> export as the iterator has to seek to the given docID from start for each
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An
> alternative is to extract the DocValues in docID-order, with re-use of
> DocValues-iterators. The idea is as follows:
> # Change the FieldWriters for export to re-use the DocValues-iterators if
> subsequent requests are for docIDs higher than the previous ones
> # Calculate the sliding window of SortDocs as usual
> # Take a note of the order of the SortDocs in the sliding window
> # Re-sort the SortDocs in docID-order
> # Extract the DocValues to a temporary on-heap structure
> # Re-sort the extracted values to the original sliding window order
> Deliver the values
> One big difference from the current export code is of course the need to hold
> the whole sliding window scaled result set in memory. This might well be a
> showstopper as there is no real limit to how large this partial result set
> can be. Maybe such an optimization could be requested explicitly if the user
> knows that there is enough memory?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
