[
https://issues.apache.org/jira/browse/LUCENE-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dingjin updated LUCENE-3834:
----------------------------
Priority: Minor (was: Major)
Description:
That is because the field input in class SentenceTokenizer isn't reset after we
call the method reset().
They are two input field,one is from Tokenizer and another is from
TokenFilter,if we need to reset a tokenstream created by SmartChineseAnalyzer,
both them need reset.This bug is because of the author forget reset input
field in class SentenceTokenizer .
class path : org.apache.lucene.analysis.cn.smart.SentenceTokenizer
oringal code
public final class SentenceTokenizer extends Tokenizer {
....
@Override
public void reset() throws IOException {
super.reset();
tokenStart = tokenEnd = 0;
}
...
}
this method should changes as follow
public void reset() throws IOException {
super.reset();
/*should reset input*/
if (input.markSupported())
input.reset();
tokenStart = tokenEnd = 0;
}
was:
That is because the field input in class SentenceTokenizer isn't reset after we
call the method reset().
They are two input field,one is from Tokenizer and another is from
TokenFilter,if we need to reset a tokenstream created by SmartChineseAnalyzer,
both them need reset.This bug is because of the author forget reset input
field in class SentenceTokenizer .
class path : org.apache.lucene.analysis.cn.smart.SentenceTokenizer
oringal code
public final class SentenceTokenizer extends Tokenizer {
....
@Override
public void reset() throws IOException {
super.reset();
tokenStart = tokenEnd = 0;
}
...
}
this method should changes as follow
public void reset() throws IOException {
super.reset();
/*should reset input*/
input.reset();
tokenStart = tokenEnd = 0;
}
> The tokenstream create by SmartChineseAnalyzer can't reset
> ----------------------------------------------------------
>
> Key: LUCENE-3834
> URL: https://issues.apache.org/jira/browse/LUCENE-3834
> Project: Lucene - Java
> Issue Type: Bug
> Components: modules/analysis
> Affects Versions: 3.5
> Reporter: dingjin
> Priority: Minor
>
> That is because the field input in class SentenceTokenizer isn't reset after
> we call the method reset().
> They are two input field,one is from Tokenizer and another is from
> TokenFilter,if we need to reset a tokenstream created by
> SmartChineseAnalyzer, both them need reset.This bug is because of the author
> forget reset input field in class SentenceTokenizer .
> class path : org.apache.lucene.analysis.cn.smart.SentenceTokenizer
> oringal code
> public final class SentenceTokenizer extends Tokenizer {
> ....
> @Override
> public void reset() throws IOException {
> super.reset();
> tokenStart = tokenEnd = 0;
> }
> ...
> }
> this method should changes as follow
>
> public void reset() throws IOException {
> super.reset();
> /*should reset input*/
> if (input.markSupported())
> input.reset();
> tokenStart = tokenEnd = 0;
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
