[
https://issues.apache.org/jira/browse/LUCENE-8610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16723164#comment-16723164
]
Michael Gibney commented on LUCENE-8610:
----------------------------------------
Thanks, [~romseygeek]! Created SOLR-13077.
I do see that the [TokenStream
API|https://lucene.apache.org/core/7_6_0/core/org/apache/lucene/analysis/TokenStream.html]
says:
{quote}To make sure that filters and consumers know which attributes are
available, the attributes must be added during instantiation.
{quote}
Is this a requirement, or a recommendation? It reads a bit like a requirement;
but you could also read it as being a recommendation and a warning ("if you
*do* add attributes after instantiation, filters and consumers might not know
which attributes are available, so proceed at your own risk")
> NPE in TermsHashPerField.add() for TokenStreams with lazily instantiated
> token Attributes
> -----------------------------------------------------------------------------------------
>
> Key: LUCENE-8610
> URL: https://issues.apache.org/jira/browse/LUCENE-8610
> Project: Lucene - Core
> Issue Type: Wish
> Components: core/index
> Affects Versions: 7.4, master (8.0)
> Reporter: Michael Gibney
> Priority: Minor
> Attachments: LUCENE-8610.patch
>
>
> {{DefaultIndexingChain.invert(...)}} calls
> {{invertState.setAttributeSource(stream)}} before {{stream.incrementToken()}}
> is called.
> For instances of {{stream}} that lazily instantiate token attributes (e.g.,
> as {{solr.PreAnalyzedField$PreAnalyzedTokenizer}} does upon the first call to
> {{incrementToken()}} that returns {{true}}), this can result in caching a
> {{null}} value in {{invertState.termAttribute}} for a given {{stream}}
> instance.
> Subsequent calls that reuse the same {{stream}} instance (reusing
> {{TokenStreamComponents}}) for field values with at least 1 token will call
> {{termHashPerField.start(...)}} which sets {{termsHashPerField.termAtt}} from
> the {{null}} value cached in the {{FieldInvertState.termAttribute}}. An NPE
> would be thrown when {{termsHashPerField.add()}} reasonably but incorrectly
> assumes a non-null value for {{termAtt}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
