[
https://issues.apache.org/jira/browse/SOLR-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510855#comment-13510855
]
Shawn Heisey commented on SOLR-4148:
------------------------------------
Thanks Robert. As I was putting the issue together, I did wonder if it was a
known limitation. On 4x I'm using ICUTokenizerFactory on most fields, only one
field left with the whitespace tokenizer. That field is for mime-type, which
should never have long unstructured data in it.
If nobody objects, this issue can hang around until someone decides to take a
crack at fixing known limitations.
> WhiteSpaceTokenizer splits long no-whitespace sequences every 255 characters
> ----------------------------------------------------------------------------
>
> Key: SOLR-4148
> URL: https://issues.apache.org/jira/browse/SOLR-4148
> Project: Solr
> Issue Type: Bug
> Components: Schema and Analysis
> Affects Versions: 3.5, 4.0
> Environment: for branch_4x:
> Linux bigindy5 2.6.32-279.14.1.el6.centos.plus.x86_64 #1 SMP Wed Nov 7
> 00:40:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.7.0_09"
> Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)
> for 3.5:
> Linux idxa1 2.6.32-279.14.1.el6.centos.plus.x86_64 #1 SMP Wed Nov 7 00:40:45
> UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
> Reporter: Shawn Heisey
> Fix For: 4.1, 5.0
>
>
> I have the following text input in a field with an analysis chain that starts
> with WhiteSpaceTokenizerFactory.
> Auto,DEU,Drogendezernat,Drogenfahnder,Drogenfahndung,Drug,Duesseldorf,Ermittler,Ermittlung,Fahnder,Fahndung,Fahrzeug,Germany,Innere,Kriminalitaet,Kriminalpolizei,Kriminalpolizisten,Kripo,Oeffentliche,Ordnung,Polizei,Polizeikelle,Polizist,Polizistin,Rauschgift,Rauschgiftdezernat,Sicherheit,Sicherheitsbehoerden,Sicherheitskraefte,Uniform,Verbrechen,Waffen,Zivilbeamte,Zivilfahnder,Zivilpolizisten,anhalten,car,crime,criminal,drug,enforcement,investigators,law,officers,police,policeman,policemen,policeofficer,public,saftey,security,squad,stop,stoppen,stopping,suspect,suspicious,ueberpruefen,verdaechtig
> This input has no whitespace. WhiteSpaceTokenizer is taking this and
> breaking it into tokens after every 255th character. It happens on both 3.5
> and a recent branch_4x checkout. I am using ICUTokenizerFactory on some
> fields in branch_4x, it doesn't have this problem. I have not tried the
> ICUTokenizerFactory on 3.5.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
