Copilot commented on code in PR #6801:
URL: https://github.com/apache/texera/pull/6801#discussion_r3634635569
##########
common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/keywordSearch/CaseSensitiveAnalyzer.scala:
##########
@@ -19,16 +19,17 @@
package org.apache.texera.amber.operator.keywordSearch
import org.apache.lucene.analysis.{Analyzer, TokenStream}
-import org.apache.lucene.analysis.core.WhitespaceTokenizer
+import org.apache.lucene.analysis.standard.StandardTokenizer
import org.apache.lucene.analysis.CharArraySet
import org.apache.lucene.analysis.StopFilter
import org.apache.lucene.analysis.Analyzer.TokenStreamComponents
-// Achieves case sensitivity by skipping the lowercasing and normalization
-// pipeline used in StandardAnalyzer.
+// Mirrors StandardAnalyzer but omits the LowerCaseFilter, so tokens keep their
+// case while still splitting on Unicode word boundaries. A bare
WhitespaceTokenizer
+// would instead glue punctuation to tokens (e.g. "perfect." would not match
"perfect").
class CaseSensitiveAnalyzer extends Analyzer {
override protected def createComponents(fieldName: String):
TokenStreamComponents = {
- val tokenizer = new WhitespaceTokenizer()
+ val tokenizer = new StandardTokenizer()
val stream: TokenStream = new StopFilter(tokenizer, CharArraySet.EMPTY_SET)
new TokenStreamComponents(tokenizer, stream)
Review Comment:
Switching from WhitespaceTokenizer to StandardTokenizer changes the emitted
tokens (punctuation is no longer attached, and token boundaries are no longer
purely whitespace). There is an existing unit test `CaseSensitiveAnalyzerSpec`
that explicitly asserts WhitespaceTokenizer behavior and punctuation attachment
(e.g. it expects `"abc,def"` to remain a single token). With this change, that
spec will fail and needs to be updated to assert the new
StandardTokenizer-based behavior instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
