[
https://issues.apache.org/jira/browse/OPENNLP-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18123432#comment-18123432
]
ASF GitHub Bot commented on OPENNLP-1988:
-----------------------------------------
krickert commented on code in PR #1319:
URL: https://github.com/apache/opennlp/pull/1319#discussion_r4186302803
##########
opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java:
##########
@@ -203,14 +229,38 @@ public Map<String, Integer> loadVocab(
static Map<String, Integer> loadVocabFile(
final File vocabFile) throws IOException {
+ return readVocabFile(vocabFile).ids();
+ }
+
+ /**
+ * A vocabulary read from a file or JSON text.
+ *
+ * @param ids The map of tokens to IDs.
+ * @param lowercase The {@code normalizer.lowercase} setting of a {@code
tokenizer.json}, or
+ * {@code null} if the vocabulary does not carry that setting.
+ */
+ record Vocabulary(Map<String, Integer> ids, Boolean lowercase) {
+ }
+
+ /**
+ * Reads a vocabulary file as {@link #loadVocab(File)} does, keeping the
settings of a
+ * {@code tokenizer.json} that a component must agree with.
+ *
+ * @param vocabFile The vocabulary file.
+ * @return The vocabulary.
+ * @throws IOException Thrown if the vocabulary file cannot be opened or
read.
+ * @throws InvalidFormatException Thrown if a JSON vocabulary is malformed,
has a layout that
+ * is not supported, or contains a value that is not a non-negative
integer.
+ */
+ static Vocabulary readVocabFile(final File vocabFile) throws IOException {
Review Comment:
Removed static. I kept it package-private so the tests can call it directly
(Vocabulary is also package-private)
> Add support for Hugging Face tokenizer.json files whose tokenizer
> configuration can be represented by OpenNLP's WordPiece encoder.
> ----------------------------------------------------------------------------------------------------------------------------------
>
> Key: OPENNLP-1988
> URL: https://issues.apache.org/jira/browse/OPENNLP-1988
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This work is separated from OPENNLP-1931, which replaces regex-based
> vocabulary
> and configuration parsing. That PR will support vocab.txt and flat vocab.json
> files; tokenizer.json support belongs in this follow-up.
> Reading model.vocab alone is insufficient. AbstractDL constructs a
> WordpieceEncoder, but tokenizer.json also represents BPE and Unigram models.
> Accepting a BPE vocabulary and applying WordPiece can produce incorrect model
> inputs without reporting an error. Even WordPiece files can specify settings
> that differ from the encoder's defaults.
> Scope
> - Require model.type to be WordPiece and model.vocab to be an object mapping
> tokens to non-negative integer IDs. Preserve the supplied IDs.
> - Reject BPE, Unigram, missing or unknown model types, and unsupported layouts
> with an error identifying the unsupported setting.
> - Define the supported WordPiece configuration, including
> continuing_subword_prefix, unk_token, max_input_chars_per_word,
> normalization,
> pre-tokenization, and special-token processing. Honor supported settings and
> reject incompatible settings instead of silently using encoder defaults.
> - Define how tokenizer settings interact with explicit InferenceOptions,
> particularly lowerCase. Document defaults and conflict handling.
> - Define handling of added_tokens, including tokens absent from model.vocab,
> conflicting IDs, and special-token matching. Reject unsupported behavior
> rather than silently dropping entries that affect tokenization.
> - Reuse the JSON scanner from OPENNLP-1931. Report malformed or unsupported
> resource files as InvalidFormatException at the file-loading boundary.
> - Keep existing vocab.txt and flat vocab.json behavior unchanged. Document
> the supported tokenizer.json subset in the manual.
> Validation
> - Add fixtures for supported WordPiece configurations and rejection cases for
> BPE, Unigram, missing model types, invalid IDs, incompatible settings, and
> unsupported added_tokens behavior.
> - Check exact token pieces and IDs against reference outputs for
> representative
> cased and uncased WordPiece models. Cover continuation pieces, unknown
> tokens,
> Unicode normalization, and special tokens; vocabulary-map equality alone is
> insufficient.
> - Verify errors through the public DL file-loading boundary and retain
> coverage
> for existing vocabulary formats.
> - Run the affected DL tests and DocumentCategorizerDLEval with a supported
> model
> before merge.
> Out of scope: implementing BPE or Unigram tokenizers, or claiming support for
> arbitrary Hugging Face tokenizer pipelines.
> References
> OPENNLP-1931: https://issues.apache.org/jira/browse/OPENNLP-1931
> PR #1277: https://github.com/apache/opennlp/pull/1277
> Review requesting this split:
> https://github.com/apache/opennlp/pull/1277#issuecomment-5743853571
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
