[
https://issues.apache.org/jira/browse/LUCENE-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547298#comment-16547298
]
David Smiley commented on LUCENE-8403:
--------------------------------------
{quote}Doesn't the matches API remove the need for that feature? It should
allow to retrieve offsets from the inverted index directly?
{quote}
I assume you refer to
{{org.apache.lucene.search.uhighlight.UnifiedHighlighter.OffsetSource#POSTINGS_WITH_TERM_VECTORS}}
No, it's definitely not made obsolete; I don't think the new Weight.Matches
is related at all. But good question though! Consider the case of an index
that only has offsets in postings (in the inverted index), there are no term
vectors, and you're asked to highlight a MultiTermQuery. Without the benefit
of term vectors for the document you want to highlight, you'd have to set an
MTQ loose on the entire index to find potentially thousands of terms (maybe
only one is in this document?) which could be very high overhead. At least the
worse case would be terrible. The UH (and PostingsHighlighter before it) sees
an MTQ and switches over to Analysis. Perhaps it isn't necessarily always more
costly than term vectors or analysis – like if the MTQ matches a small number
of terms. I suppose a smarter highlighter could check first. CC [~romseygeek]
> Support 'filtered' term vectors - don't require all terms to be present
> -----------------------------------------------------------------------
>
> Key: LUCENE-8403
> URL: https://issues.apache.org/jira/browse/LUCENE-8403
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Michael Braun
> Priority: Minor
>
> The genesis of this was a conversation and idea from [~dsmiley] several years
> ago.
> In order to optimize term vector storage, we may not actually need all tokens
> to be present in the term vectors - and if so, ideally our codec could just
> opt not to store them.
> I attempted to fork the standard codec and override the TermVectorsFormat and
> TermVectorsWriter to ignore storing certain Terms within a field. This
> worked, however, CheckIndex checks that the terms present in the standard
> postings are also present in the TVs, if TVs enabled. So this then doesn't
> work as 'valid' according to CheckIndex.
> Can the TermVectorsFormat be made in such a way to support configuration of
> tokens that should not be stored (benefits: less storage, more optimal
> retrieval per doc)? Is this valuable to the wider community? Is there a way
> we can design this to not break CheckIndex's contract while at the same time
> lessening storage for unneeded tokens?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
