Michael Gibney created SOLR-13077:
-------------------------------------
Summary: PreAnalyzedField TokenStreamComponents should be reusable
Key: SOLR-13077
URL: https://issues.apache.org/jira/browse/SOLR-13077
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: Schema and Analysis
Reporter: Michael Gibney
{{TokenStreamComponents}} for {{PreAnalyzedField}} is currently recreated from
scratch for every field value.
This is necessary at the moment because the current implementation has no a
priori knowledge about the schema/TokenStream that it's deserializing --
Attributes are implicit in the serialized token stream, and token Attributes
are lazily instantiated in {{incrementToken()}}.
Reuse of {{TokenStreamComponents}} with the current implementation would at a
minimum cause problems at index time, when Attributes are cached in indexing
components (e.g., {{FieldInvertState}}), keyed per {{AttributeSource}}. For
instance, if the first field encountered has no value specified for
{{PayloadAttribute}}, a {{null}} value would be cached for that
{{PayloadAttribute}} for the corresponding {{AttributeSource}}. If that
{{AttributeSource}} were to be reused for a field that _does_ specify a
{{PayloadAttribute}}, indexing components would "consult" the cached {{null}}
value, and the payload (and all subsequent payloads) would be silently ignored
(not indexed).
This is not exactly _broken_ currently, but I gather it's an unorthodox
implementation of {{TokenStream}}, and the current workaround of disabling
{{TokenStreamComponents}} reuse necessarily adds to object creation and GC
overhead.
For reference, the [TokenStream
API|https://lucene.apache.org/core/7_5_0/core/org/apache/lucene/analysis/TokenStream.html]
says:
bq.To make sure that filters and consumers know which attributes are available,
the attributes must be added during instantiation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
