rzo1 commented on code in PR #829:
URL: https://github.com/apache/opennlp/pull/829#discussion_r2248003500
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/SentenceDetectorME.java:
##########
@@ -340,15 +340,27 @@ protected boolean isAcceptableBreak(CharSequence s, int
fromIndex, int candidate
return true;
for (StringList abb : abbDict) {
- String token = abb.getToken(0);
+ final String token = abb.getToken(0);
+ final int tokenPosition = s.toString().indexOf(token, fromIndex);
+ if (tokenPosition == -1) {
+ continue; // skip fast
+ }
+ final char prevChar = s.charAt(tokenPosition - 1);
int tokenLength = token.length();
- int tokenPosition = s.toString().indexOf(token, fromIndex);
- if (tokenPosition + tokenLength < candidateIndex || tokenPosition >
candidateIndex)
- continue;
+ if (tokenPosition + tokenLength < candidateIndex || tokenPosition >
candidateIndex ||
+ /*
+ * Note:
+ * Skip abbreviation candidate if regular characters exist directly
before it,
+ * That is, any letter or digit except: a whitespace, an apostrophe,
or an opening round bracket.
+ * This prevents mismatches from overlaps close to an actual sentence
end.
+ */
+ !(prevChar == ' ' || prevChar == '\'' || prevChar == '`' || prevChar
== 'ยด' || prevChar == '(')) {
Review Comment:
Would it be beneficial to check for `Character.isWhitespace(...)` instead of
`' '` to cover other whitespace derivations?
##########
opennlp-core/opennlp-runtime/src/main/java/opennlp/tools/sentdetect/SentenceDetectorME.java:
##########
@@ -340,15 +340,27 @@ protected boolean isAcceptableBreak(CharSequence s, int
fromIndex, int candidate
return true;
for (StringList abb : abbDict) {
- String token = abb.getToken(0);
+ final String token = abb.getToken(0);
+ final int tokenPosition = s.toString().indexOf(token, fromIndex);
+ if (tokenPosition == -1) {
+ continue; // skip fast
+ }
+ final char prevChar = s.charAt(tokenPosition - 1);
int tokenLength = token.length();
- int tokenPosition = s.toString().indexOf(token, fromIndex);
- if (tokenPosition + tokenLength < candidateIndex || tokenPosition >
candidateIndex)
- continue;
+ if (tokenPosition + tokenLength < candidateIndex || tokenPosition >
candidateIndex ||
+ /*
+ * Note:
+ * Skip abbreviation candidate if regular characters exist directly
before it,
+ * That is, any letter or digit except: a whitespace, an apostrophe,
or an opening round bracket.
Review Comment:
Comment misses `\`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@opennlp.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
