[
https://issues.apache.org/jira/browse/LUCENE-10023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17377639#comment-17377639
]
Robert Muir commented on LUCENE-10023:
--------------------------------------
{quote}
In contrast, the current PR only runs analysis once, and avoids buffering
BytesRefs (i.e., via deepCopy() – also avoids creating disposable one-off
SortedSetDocValuesField objects), and instead sends BytesRefs directly to the
docValuesWriter.
{quote}
I don't view this as an advantage. In fact it seems to only matter for exactly
the trappy cases that I am concerned about. If docs are so large, and you dont
have enough ram to index a single doc into SortedSetDocValues itself, well,
that says everything :)
Also it isn't clear the current PR works in all cases (e.g. indexed=false, but
Field passed with a Tokenstream)... There are a lot of corner cases, but things
use these features (e.g. solr preanalyzedfield). I am concerned about the
increase in complexity with really no benefit, except making it easier for
someone to shoot themselves in the foot.
> Multi-token post-analysis DocValues
> -----------------------------------
>
> Key: LUCENE-10023
> URL: https://issues.apache.org/jira/browse/LUCENE-10023
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Reporter: Michael Gibney
> Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The single-token case for post-analysis DocValues is accounted for by
> {{Analyzer.normalize(...)}} (and formerly {{MultiTermAwareComponent}}); but
> there are cases where it would be desirable to have post-analysis DocValues
> based on multi-token fields.
> The main use cases that I can think of are variants of faceting/terms
> aggregation. I understand that this could be viewed as "trappy" for the naive
> "Moby Dick word cloud" case; but:
> # I think this can be supported fairly cleanly in Lucene
> # Explicit user configuration of this option would help prevent people
> shooting themselves in the foot
> # The current situation is arguably "trappy" as well; it just offloads the
> trappiness onto Lucene-external workarounds for systems/users that want to
> support this kind of behavior
> # Integrating this functionality directly in Lucene would afford consistency
> guarantees that present opportunities for future optimizations (e.g., shared
> Terms dictionary between indexed terms and DocValues).
> This issue proposes adding support for multi-token post-analysis DocValues
> directly to {{IndexingChain}}. The initial proposal involves extending the
> API to include {{IndexableFieldType.tokenDocValuesType()}} (in addition to
> existing {{IndexableFieldType.docValuesType()}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
