[
https://issues.apache.org/jira/browse/OPENNLP-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner resolved OPENNLP-1875.
-------------------------------------
Resolution: Delivered
> Align whitespace handling with the Unicode White_Space property
> ---------------------------------------------------------------
>
> Key: OPENNLP-1875
> URL: https://issues.apache.org/jira/browse/OPENNLP-1875
> Project: OpenNLP
> Issue Type: Improvement
> Affects Versions: 3.0.0-M4
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 3.0.0-M5
>
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> h2. Summary
> OPENNLP-1850 established standards-sourced character classes as a design goal
> (the {{CharClass}} engine reads the UCD {{White_Space}} and {{Dash}} sets,
> not {{Character.isWhitespace}} or a regex class), but the rest of the
> codebase has not caught up: {{StringUtil}} still uses the JVM
> {{Character.isWhitespace}} predicate, and several call sites split or scan
> user text with {{{}split("{}}}{{{}s+"){}}} or {{{}Character.isWhitespace{}}}.
> This ticket migrates the whitespace predicate in {{StringUtil}} to the UCD
> {{White_Space}} set (or documents the intentional exception where JVM
> semantics are genuinely wanted), and audits and converts the remaining
> user-text call sites. This is a behavior change and carries an explicit
> release note, because the UCD {{White_Space}} set and
> {{Character.isWhitespace}} disagree on real code points: {{White_Space}}
> includes the no-break space U+00A0 and the figure and narrow spaces that
> {{Character.isWhitespace}} excludes, and excludes the U+001C to U+001F
> information separators that {{Character.isWhitespace}} includes. OPENNLP-205
> already made exactly this correction in the sentence-detector mapping, so
> this generalizes that fix.
> h2. Scope
> * Audit {{{}split("{}}}{{{}s+"){}}}, {{{}Character.isWhitespace{}}}, and
> {{Character.isSpaceChar}} call sites across the codebase outside the
> OPENNLP-1850 engine. Classify each as user text (migrate to the UCD set) or
> config and non-user-text (leave; regex and JVM predicates are acceptable
> there per the design goals).
> * Migrate the whitespace handling in {{StringUtil}} to the UCD-backed
> {{White_Space}} API delivered in OPENNLP-1850, or document the intentional
> JVM exception with a rationale where a caller genuinely needs JVM semantics
> (for example a format-fixed parser).
> * Add a release note in the 3.0 migration section describing the delta: NBSP
> and the Unicode space separators now count as whitespace; the U+001C to
> U+001F information separators no longer do.
> h2. Acceptance criteria
> * The {{StringUtil}} whitespace predicate is UCD-backed, or the JVM
> exception is documented with a rationale.
> * The call-site audit is complete: user-text paths use the UCD set, and
> every retained regex or JVM predicate on a text path is justified in a
> comment.
> * A release note documents the behavior delta at the boundary code points
> (NBSP, the space separators, the information separators).
> * Tests pin the delta at those boundary code points.
> h2. Out of scope
> * The OPENNLP-1850 engine itself, which is already UCD-backed.
> * Regex in config parsing and other non-user-text utilities, which the
> design goals permit.
> * The legacy {{CharSequenceNormalizer}} regex migration, which is a separate
> ticket.
> * Changing tokenizer or sentence-detector feature generation (model
> stability).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
