Shawn Heisey created SOLR-4148:
----------------------------------
Summary: WhiteSpaceTokenizer splits long no-whitespace sequences
every 255 characters
Key: SOLR-4148
URL: https://issues.apache.org/jira/browse/SOLR-4148
Project: Solr
Issue Type: Bug
Components: Schema and Analysis
Affects Versions: 4.0, 3.5
Environment: for branch_4x:
Linux bigindy5 2.6.32-279.14.1.el6.centos.plus.x86_64 #1 SMP Wed Nov 7 00:40:45
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
java version "1.7.0_09"
Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)
for 3.5:
Linux idxa1 2.6.32-279.14.1.el6.centos.plus.x86_64 #1 SMP Wed Nov 7 00:40:45
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
Reporter: Shawn Heisey
Fix For: 4.1, 5.0
I have the following text input in a field with an analysis chain that starts
with WhiteSpaceTokenizerFactory.
Auto,DEU,Drogendezernat,Drogenfahnder,Drogenfahndung,Drug,Duesseldorf,Ermittler,Ermittlung,Fahnder,Fahndung,Fahrzeug,Germany,Innere,Kriminalitaet,Kriminalpolizei,Kriminalpolizisten,Kripo,Oeffentliche,Ordnung,Polizei,Polizeikelle,Polizist,Polizistin,Rauschgift,Rauschgiftdezernat,Sicherheit,Sicherheitsbehoerden,Sicherheitskraefte,Uniform,Verbrechen,Waffen,Zivilbeamte,Zivilfahnder,Zivilpolizisten,anhalten,car,crime,criminal,drug,enforcement,investigators,law,officers,police,policeman,policemen,policeofficer,public,saftey,security,squad,stop,stoppen,stopping,suspect,suspicious,ueberpruefen,verdaechtig
This input has no whitespace. WhiteSpaceTokenizer is taking this and breaking
it into tokens after every 255th character. It happens on both 3.5 and a
recent branch_4x checkout. I am using ICUTokenizerFactory on some fields in
branch_4x, it doesn't have this problem. I have not tried the
ICUTokenizerFactory on 3.5.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
