kz930 commented on code in PR #6801:
URL: https://github.com/apache/texera/pull/6801#discussion_r3634655495
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/keywordSearch/CaseSensitiveAnalyzer.scala:
##########
@@ -19,16 +19,17 @@
package org.apache.texera.amber.operator.keywordSearch
import org.apache.lucene.analysis.{Analyzer, TokenStream}
-import org.apache.lucene.analysis.core.WhitespaceTokenizer
+import org.apache.lucene.analysis.standard.StandardTokenizer
import org.apache.lucene.analysis.CharArraySet
import org.apache.lucene.analysis.StopFilter
import org.apache.lucene.analysis.Analyzer.TokenStreamComponents
-// Achieves case sensitivity by skipping the lowercasing and normalization
-// pipeline used in StandardAnalyzer.
+// Mirrors StandardAnalyzer but omits the LowerCaseFilter, so tokens keep their
+// case while still splitting on Unicode word boundaries. A bare
WhitespaceTokenizer
+// would instead glue punctuation to tokens (e.g. "perfect." would not match
"perfect").
class CaseSensitiveAnalyzer extends Analyzer {
override protected def createComponents(fieldName: String):
TokenStreamComponents = {
- val tokenizer = new WhitespaceTokenizer()
+ val tokenizer = new StandardTokenizer()
val stream: TokenStream = new StopFilter(tokenizer, CharArraySet.EMPTY_SET)
new TokenStreamComponents(tokenizer, stream)
Review Comment:
Good catch — fixed in f245bb7c0.
You're right that `CaseSensitiveAnalyzerSpec` pins the old
`WhitespaceTokenizer` behavior and would break. I updated it to assert the new
`StandardTokenizer` word-boundary behavior instead: `"abc,def"` → `["abc",
"def"]` and `"Hello, world!"` → `["Hello", "world"]` (punctuation is now a
boundary and stripped, not kept attached), and relabeled the tokenizer sections
accordingly. Verified locally — `CaseSensitiveAnalyzerSpec` is green (13/13).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
