Hi Colm, Thanks for bringing the issue up. The behavior you mention is expected, though - see below for details.
In the future, you should use the java-user@l.a.o mailing list for questions about Lucene *usage* -- this mailing list (dev@l.a.o), by contrast, is intended for development discussions. <rant> FYI, Nabble.com stripped out all of your code and examples before sending your message to the mailing list. My suggestion: stop using Nabble. (I've described this problem to their support people a couple of times, and they apparently just don't care, since it still persists, years later.) </rant> StandardTokenizer, the tokenizer included in StandardAnalyzer, implements the Word Break rules from Unicode 6.0.0 UAX#29 <http://www.unicode.org/reports/tr29/tr29-17.html>. These rules are international in nature. The UAX#29 Word Break rules that prohibit breaking around commas when surrounded by digits are: Do not break within sequences, such as "3.2" or "3,456.789". WB11. Numeric (MidNum | MidNumLet) × Numeric WB12. Numeric × (MidNum | MidNumLet) Numeric (In these rules, "×" means "do not break".) >From the table "Word_Break property values" at ><http://www.unicode.org/reports/tr29/tr29-17.html#Default_Word_Boundaries>, >you can see that the set of characters that are assigned the Word_Break:MidNum >property value is a superset of the characters assigned the >Line_Break:Infix_Numeric property value, which includes the comma character. >You can see the full set of characters that are assigned the >Line_Break:Infix_Number property value here: ><http://unicode.org/cldr/utility/list-unicodeset.jsp?a=%5B%3ALine_Break+%3D+Infix_Numeric%3A%5D> > (note that because this utility refers to Unicode 6.1.0, the results may >differ slightly from Lucene v3.5.0 StandardTokenizer, since it uses Unicode >6.0.0). Steve -----Original Message----- From: colm.mchugh [mailto:colm.mch...@mapflow.com] Sent: Thursday, March 22, 2012 9:23 AM To: dev@lucene.apache.org Subject: Case where StandardAnalyzer doesn't remove punctuation I'm using Lucene to search address data, and came across an interesting case where StandardAnalyzer appears not to remove punctuation (a comma). To illustrate, the following code snippet uses StandardAnalyzer to analyze an address, printing out each analyzed token. The output of the code snippet is: If the code is altered slightly so the String text is initialized as follows: (there's a space between the first comma and the building number) then the output is as follows:I would expect the output to be the same in both cases based on my understanding. Is this a known issue? Or am I off on my understanding? It's not a biggie. It caught my attention because I have a unit test that asserts token text is all lower case or alphanumeric. It can be easily got around, but I thought it worth posting about. -- View this message in context: http://lucene.472066.n3.nabble.com/Case-where-StandardAnalyzer-doesn-t-remove-punctuation-tp3848460p3848460.html Sent from the Lucene - Java Developer mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
