[
https://issues.apache.org/jira/browse/LUCENE-2763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steven Rowe updated LUCENE-2763:
--------------------------------
Attachment: LUCENE-2763.patch
Patch to perform the switch on trunk.
> Swap URL+Email recognizing StandardTokenizer and UAX29Tokenizer
> ---------------------------------------------------------------
>
> Key: LUCENE-2763
> URL: https://issues.apache.org/jira/browse/LUCENE-2763
> Project: Lucene - Java
> Issue Type: Improvement
> Components: Analysis
> Affects Versions: 3.1, 4.0
> Reporter: Steven Rowe
> Assignee: Steven Rowe
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2763.patch
>
>
> Currently, in addition to implementing the UAX#29 word boundary rules,
> StandardTokenizer recognizes email adresses and URLs, but doesn't provide a
> way to turn this behavior off and/or provide overlapping tokens with the
> components (username from email address, hostname from URL, etc.).
> UAX29Tokenizer should become StandardTokenizer, and current StandardTokenizer
> should be renamed to something like UAX29TokenizerPlusPlus (or something like
> that).
> For rationale, see [the discussion at the reopened
> LUCENE-2167|https://issues.apache.org/jira/browse/LUCENE-2167?focusedCommentId=12929325&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12929325].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
