[
https://issues.apache.org/jira/browse/SOLR-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14296079#comment-14296079
]
Steve Rowe commented on SOLR-7058:
----------------------------------
One idea:
We can make the {{AddSchemaFieldsUpdateProcessorFactory}}'s default field type
{{text_general}} instead of {{string}}, and then add a new kind of type mapping
constraint that can be configured with a max value length. Then one of these
new type mappings can be mapped to the {{string}} field type and placed after
all the other type mappings.
That way if no other type mapping matches, and a field value is not longer than
the configured max value length, the field will be created with type
{{string}}; otherwise, the field will be created with type {{text_general}}.
Multiple text field values (i.e., those that don't trigger any other type
mappings) should fall back to the default field type if they are not all short
enough.
> Data-driven schema needs to index large text fields as text and not as string
> -----------------------------------------------------------------------------
>
> Key: SOLR-7058
> URL: https://issues.apache.org/jira/browse/SOLR-7058
> Project: Solr
> Issue Type: Improvement
> Components: Data-driven Schema
> Reporter: Timothy Potter
>
> While using the SimplePostTool to index some freebase articles into a core
> that uses our data-driven configs, I ran into the following gem:
> {code}
> Caused by: java.lang.IllegalArgumentException: Document contains at least one
> immense term in field="xml_data" (whose UTF8 encoding is longer than the max
> length 32766), all of which were skipped. Please correct the analyzer to not
> produce such terms. The prefix of the first immense term is: '[60, 63, 120,
> 109, 108, 32, 118, 101, 114, 115, 105, 111, 110, 61, 34, 49, 46, 48, 34, 32,
> 101, 110, 99, 111, 100, 105, 110, 103, 61, 34]...', original message: bytes
> can be at most 32766 in length; got 173684
> at
> org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:667)
> at
> org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
> at
> org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
> at
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:231)
> at
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:449)
> at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1415)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:242)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:166)
> {code}
> Ideally, the data-driven configs would index large text fields containing
> multiple tokens (whitespace delimited) as text and not a string. However,
> this obviously poses an issue if the first doc has a short text value that
> looks like a string and then the next doc has a large one. Not sure what the
> right solution looks like yet, but wanted to capture the issue so we can
> discuss options.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
