[
https://issues.apache.org/jira/browse/OPENNLP-1931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18119509#comment-18119509
]
ASF GitHub Bot commented on OPENNLP-1931:
-----------------------------------------
rzo1 commented on PR #1277:
URL: https://github.com/apache/opennlp/pull/1277#issuecomment-5838118747
@krickert Please link the eval build here.
> RegEx removal part 3: JSON vocabulary and id2label scrape in opennlp-dl
> -----------------------------------------------------------------------
>
> Key: OPENNLP-1931
> URL: https://issues.apache.org/jira/browse/OPENNLP-1931
> Project: OpenNLP
> Issue Type: Improvement
> Components: Deep Learning
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Fix For: 2.5.13, 3.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {{AbstractDL.loadJsonVocab}} and {{DocumentCategorizerConfig.fromJson}}
> located string keys and values with regular expressions. Both now use a small
> cursor helper, {{opennlp.dl.JsonScan}} (public, marked internal), that finds
> string literals honoring backslash escapes, skips ASCII whitespace, and reads
> digit runs.
> Behavior is unchanged, including the quirks of the old patterns, which the
> tests pin: a value with a decimal fraction yields its integer prefix, a
> negative id is skipped, the id2label object is cut at the first closing brace
> even inside a nested value, and a key spanning a newline is accepted while a
> value with a newline is skipped.
> Verification: the old patterns and the new scan were compared over the real
> vocabulary and configuration resources, 57 hand-written edge cases, and
> 400,000 generated inputs with zero differences. {{JsonScanTest}} adds 71
> cases, {{LoadVocabTest}} 27, {{DocumentCategorizerConfigTest}} 24.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
