Kristian Rickert created OPENNLP-1838:
-----------------------------------------
Summary: opennlp-dl: use BertTokenizer so uncased ONNX models
tokenize correctly
Key: OPENNLP-1838
URL: https://issues.apache.org/jira/browse/OPENNLP-1838
Project: OpenNLP
Issue Type: Bug
Components: dl, Tokenizer
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Follow-up to OPENNLP-1837 / PR #1073.
OPENNLP-1837 adds {{opennlp.tools.tokenize.BertTokenizer}} (full BERT basic
tokenization + wordpiece). {{AbstractDL.createTokenizer()}} still builds a bare
{{WordpieceTokenizer}}, so all {{opennlp-dl}} components feed unnormalized
text to the vocabulary lookup:
* {{SentenceVectorsDL}}
* {{DocumentCategorizerDL}}
* {{NameFinderDL}}
h3. Impact
For *uncased* models (both models recommended by the opennlp-dl README are
uncased — {{sentence-transformers/all-MiniLM-L6-v2}} and
{{nlptown/bert-base-multilingual-uncased-sentiment}}), every capitalized or
accented word becomes {{[UNK]}} on the default path.
Measured embedding fidelity vs. the HuggingFace reference for all-MiniLM-L6-v2
was cosine 0.09–0.57 with bare {{WordpieceTokenizer}}; with {{BertTokenizer}}
it exceeds 0.999999 (see OPENNLP-1837 validation).
h3. Proposal
* Switch {{AbstractDL.createTokenizer()}} (or each {{*DL}} component) to use
{{BertTokenizer}} with an explicit uncased/cased configuration flag.
* Default to uncased ({{lowerCase=true}}) to match the recommended models.
* Separately evaluate mean pooling for {{SentenceVectorsDL}} (currently
returns raw CLS hidden state; sentence-transformers uses masked mean + L2
normalization) — may be a separate ticket if scope is too large.
h3. Links
* OPENNLP-1837: https://issues.apache.org/jira/browse/OPENNLP-1837
* PR #1073: https://github.com/apache/opennlp/pull/1073
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
