ehatcher 2003/12/05 06:30:12
Modified: src/java/org/apache/lucene/analysis StopFilter.java
. CHANGES.txt
Log:
revert position increment change due to conflict with PhraseQuery
Revision Changes Path
1.6 +4 -15
jakarta-lucene/src/java/org/apache/lucene/analysis/StopFilter.java
Index: StopFilter.java
===================================================================
RCS file:
/home/cvs/jakarta-lucene/src/java/org/apache/lucene/analysis/StopFilter.java,v
retrieving revision 1.5
retrieving revision 1.6
diff -u -r1.5 -r1.6
--- StopFilter.java 1 Dec 2003 18:23:47 -0000 1.5
+++ StopFilter.java 5 Dec 2003 14:30:12 -0000 1.6
@@ -57,12 +57,8 @@
import java.io.IOException;
import java.util.Hashtable;
-/**
- * Removes stop words from a token stream. Position increments
- * on tokens emitted are adjusted to account for words
- * removed. Exact phrase queries will not match across holes left
- * by stop word removal, but sloppy phrase queries may match.
- */
+/** Removes stop words from a token stream. */
+
public final class StopFilter extends TokenFilter {
private Hashtable table;
@@ -93,17 +89,10 @@
/** Returns the next input Token whose termText() is not a stop word. */
public final Token next() throws IOException {
- int position = 1;
-
// return the first non-stop word found
- for (Token token = input.next(); token != null; token = input.next()) {
- if (table.get(token.termText) == null) {
- token.setPositionIncrement(position);
+ for (Token token = input.next(); token != null; token = input.next())
+ if (table.get(token.termText) == null)
return token;
- }
-
- position++;
- }
// reached EOS -- return null
return null;
}
1.61 +1 -7 jakarta-lucene/CHANGES.txt
Index: CHANGES.txt
===================================================================
RCS file: /home/cvs/jakarta-lucene/CHANGES.txt,v
retrieving revision 1.60
retrieving revision 1.61
diff -u -r1.60 -r1.61
--- CHANGES.txt 28 Nov 2003 02:03:14 -0000 1.60
+++ CHANGES.txt 5 Dec 2003 14:30:12 -0000 1.61
@@ -7,12 +7,6 @@
1. Added catch of BooleanQuery$TooManyClauses in QueryParser to
throw ParseException instead. (Erik Hatcher)
- 2. Modified StopFilter to increment positions to account for
- stop words removed. This prevents exact phrase queries from
- matching erroneously (use slop factor to account for missing
- stop words). StopFilter is used by StopAnalyzer, StandardAnalyzer
- and some others. (Erik Hatcher)
-
1.3 RC3
1. Added minMergeDocs in IndexWriter. This can be raised to speed
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
