[
https://issues.apache.org/jira/browse/OPENNLP-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18120945#comment-18120945
]
ASF GitHub Bot commented on OPENNLP-1929:
-----------------------------------------
krickert commented on PR #1280:
URL: https://github.com/apache/opennlp/pull/1280#issuecomment-5902908145
Yeh, the head I pushed had the wider check and neither the README hunk nor
the table.
Now only a backslash is rejected, so `|`, `.` and `[,;]` are plain
separators, with rows in the literal separator test.
Your README hunk is applied, the manual has the table of 2.x separators and
what to pass now, the test comment and the end-of-file newline change are gone,
and the branch is rebased onto main. Retitled as you had suggested.
> RegEx removal part 7: BasicContextGenerator splits on the separator as a
> regular expression
> -------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1929
> URL: https://issues.apache.org/jira/browse/OPENNLP-1929
> Project: OpenNLP
> Issue Type: Bug
> Components: Machine Learning
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> {{BasicContextGenerator(String sep)}} documents its argument as a "separator
> character", but {{getContext}} calls {{String.split(separator)}}, which
> compiles the argument as a regular expression on every call.
> Consequences for a caller who follows the Javadoc:
> * {{"|"}}, {{"."}}, {{"+"}}, {{"*"}}, {{"?"}} split on every character or on
> nothing.
> * {{"("}} or {{"["}} throw {{PatternSyntaxException}} at prediction time, not
> at construction.
> * An empty separator splits the input into single characters.
> * If an application takes the separator from configuration, a crafted pattern
> is re-evaluated on every context string, which is a backtracking exposure.
> No code in the repository calls the separator constructor; it arrived with
> the maxent modularization (OPENNLP-1740) and has no test.
> Fix: split on the literal separator with a cursor scan, keeping
> {{String.split}} trailing-empty semantics; reject an empty separator with
> {{IllegalArgumentException}} at construction; state in the Javadoc that the
> separator is literal. Tests: default space, a metacharacter separator, a
> multi-character separator, leading and trailing separators, empty and null
> separator.
> Compatibility: external code that intentionally passes a pattern such as
> {{\s+}} relied on undocumented behavior; 3.0 is the release to correct it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
