[
https://issues.apache.org/jira/browse/LUCENE-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shai Erera updated LUCENE-1068:
-------------------------------
Attachment: StandardTokenizerImpl-3.patch
The previous patch I put was incorrect since it would still break existing
applications. The current patch does:
1. Introduces a new type ACRONYM_DEP which is deprecated and recognizes the old
ACRONYM format.
2. Fixes ACRONYM to recognize LETTER + "." (LETTER + ".")+.
3. Added a public member to StandardTokenizer and StandardAnalyzer
replaceDepAcronym which can be set if the application would like the deprecated
acronym format to be treated as ACRONYM or HOST. The default behavior, if not
set is to recognize the old ACRONYM as HOST.
This is how it should be used:
public static void main(String[] args) throws Exception {
parse("www.abc.com.", false);
parse("www.abc.com.", true);
parse("www.abc.com", true);
parse("I.B.M.", true);
}
public static void parse(String text, boolean replaceDepAcronym) throws
Exception {
StandardAnalyzer analyzer = new StandardAnalyzer();
analyzer.replaceDepAcronym = replaceDepAcronym;
TokenStream ts = analyzer.tokenStream("content", new
StringReader(text));
Token t;
while ((t = ts.next()) != null) {
System.out.println(t);
}
}
And here is the output:
(wwwabccom,0,12,type=<ACRONYM>)
(www.abc.com.,0,12,type=<HOST>)
(www.abc.com,0,11,type=<HOST>)
(ibm,0,6,type=<ACRONYM>)
The member is marked deprecated so we can remove it in the next release.
Applications that would like to new behavior need to do nothing, and therefore
will not be impacted once we remove that member. Applications that want the old
behavior need to explicitly set it and in the next major release remove it.
I think that solves it. How should I proceed?
> Invalid behavior of StandardTokenizerImpl
> -----------------------------------------
>
> Key: LUCENE-1068
> URL: https://issues.apache.org/jira/browse/LUCENE-1068
> Project: Lucene - Java
> Issue Type: Bug
> Components: Analysis
> Reporter: Shai Erera
> Attachments: StandardTokenizerImpl-2.patch,
> StandardTokenizerImpl-3.patch, standardTokenizerImpl.jflex.patch,
> standardTokenizerImpl.patch
>
>
> The following code prints the output of StandardAnalyzer:
> Analyzer analyzer = new StandardAnalyzer();
> TokenStream ts = analyzer.tokenStream("content", new
> StringReader("<some text>"));
> Token t;
> while ((t = ts.next()) != null) {
> System.out.println(t);
> }
> If you pass "www.abc.com", the output is (www.abc.com,0,11,type=<HOST>)
> (which is correct in my opinion).
> However, if you pass "www.abc.com." (notice the extra '.' at the end), the
> output is (wwwabccom,0,12,type=<ACRONYM>).
> I think the behavior in the second case is incorrect for several reasons:
> 1. It recognizes the string incorrectly (no argue on that).
> 2. It kind of prevents you from putting URLs at the end of a sentence, which
> is perfectly legal.
> 3. An ACRONYM, at least to the best of my understanding, is of the form
> A.B.C. and not ABC.DEF.
> I looked at StandardTokenizerImpl.jflex and I think the problem comes from
> this definition:
> // acronyms: U.S.A., I.B.M., etc.
> // use a post-filter to remove dots
> ACRONYM = {ALPHA} "." ({ALPHA} ".")+
> Notice how the comment relates to acronym as U.S.A., I.B.M. and not something
> else. I changed the definition to
> ACRONYM = {LETTER} "." ({LETTER} ".")+
> and it solved the problem.
> This was also reported here:
> http://www.nabble.com/Inconsistent-StandardTokenizer-behaviour-tf596059.html#a1593383
> http://www.nabble.com/Standard-Analyzer---Host-and-Acronym-tf3620533.html#a10109926
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
