kamthorn commented on code in PR #16722:
URL: https://github.com/apache/lucene/pull/16722#discussion_r4117439387
##########
lucene/analysis/common/src/java/org/apache/lucene/analysis/th/ThaiTokenizer.java:
##########
@@ -61,17 +70,55 @@ public class ThaiTokenizer extends SegmentingTokenizerBase {
/** Creates a new ThaiTokenizer */
public ThaiTokenizer() {
- this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY);
+ this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, null);
+ }
+
+ /**
+ * Creates a new ThaiTokenizer with a user dictionary.
+ *
+ * @param userDictionary custom dictionary of words, or null for default
dictionary only
+ */
+ public ThaiTokenizer(CharArraySet userDictionary) {
+ this(DEFAULT_TOKEN_ATTRIBUTE_FACTORY, userDictionary);
}
/** Creates a new ThaiTokenizer, supplying the AttributeFactory */
public ThaiTokenizer(AttributeFactory factory) {
+ this(factory, null);
+ }
+
+ /**
+ * Creates a new ThaiTokenizer, supplying the AttributeFactory and user
dictionary.
+ *
+ * @param factory AttributeFactory to use
+ * @param userDictionary custom dictionary of words, or null for default
dictionary only
+ */
+ public ThaiTokenizer(AttributeFactory factory, CharArraySet userDictionary) {
super(factory, (BreakIterator) sentenceProto.clone());
if (!DBBI_AVAILABLE) {
throw new UnsupportedOperationException(
"This JRE does not have support for Thai segmentation");
}
wordBreaker = (BreakIterator) proto.clone();
+ this.userDictionary = userDictionary;
+ if (userDictionary != null && !userDictionary.isEmpty()) {
+ int minLen = Integer.MAX_VALUE;
+ int maxLen = 0;
+ for (Object obj : userDictionary) {
+ int length = (obj instanceof char[]) ? ((char[]) obj).length :
obj.toString().length();
+ if (length < minLen) {
+ minLen = length;
+ }
+ if (length > maxLen) {
+ maxLen = length;
+ }
Review Comment:
Thanks @rmuir! Updated and simplified as suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
