[
https://issues.apache.org/jira/browse/LUCENE-6868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley updated LUCENE-6868:
---------------------------------
Description:
ParallelLeafReader has a getTermVectors(docId) implementation that loops over
each field it has in a loop and calls getTermVector(docId,fieldName). But the
implementation of that will load all term vectors for all fields in that
reader, yet ParallelLeafReader only wants one. The effect is an O(n^2) where
'n' is the number of fields, when we could get O\(n) if we do it right. PLR
should call getTermVectors(docId) (not referring to a specific field) for each
of it's readers and then aggregate them.
This wouldn't be such a problem if our term vector API/Codec was improved to
not load all term vectors for all fields from disk at once.
Found via randomized-testing of IndexWriter auto-picking ParallelAtomicReader
along with a test I have that asserts TVs aren't fetched for a doc more than
once.
was:
ParallelLeafReader has a getTermVectors(docId) implementation that loops over
each field it has in a loop and calls getTermVector(docId,fieldName). But the
implementation of that will load all term vectors for all fields in that
reader, yet ParallelLeafReader only wants one. The effect is an O(n^2) where
'n' is the number of fields, when we could get O(n) if we do it right. PLR
should call getTermVectors(docId) (not referring to a specific field) for each
of it's readers and then aggregate them.
This wouldn't be such a problem if our term vector API/Codec was improved to
not load all term vectors for all fields from disk at once.
Found via randomized-testing of IndexWriter auto-picking ParallelAtomicReader
along with a test I have that asserts TVs aren't fetched for a doc more than
once.
> ParallelLeafReader.getTermVectors can indirectly load TVs multiple times
> ------------------------------------------------------------------------
>
> Key: LUCENE-6868
> URL: https://issues.apache.org/jira/browse/LUCENE-6868
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index, core/termvectors
> Reporter: David Smiley
>
> ParallelLeafReader has a getTermVectors(docId) implementation that loops over
> each field it has in a loop and calls getTermVector(docId,fieldName). But
> the implementation of that will load all term vectors for all fields in that
> reader, yet ParallelLeafReader only wants one. The effect is an O(n^2) where
> 'n' is the number of fields, when we could get O\(n) if we do it right. PLR
> should call getTermVectors(docId) (not referring to a specific field) for
> each of it's readers and then aggregate them.
> This wouldn't be such a problem if our term vector API/Codec was improved to
> not load all term vectors for all fields from disk at once.
> Found via randomized-testing of IndexWriter auto-picking ParallelAtomicReader
> along with a test I have that asserts TVs aren't fetched for a doc more than
> once.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
