[
https://issues.apache.org/jira/browse/OPENNLP-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert resolved OPENNLP-1838.
---------------------------------------
Resolution: Implemented
> opennlp-dl: use BertTokenizer so uncased ONNX models tokenize correctly
> -----------------------------------------------------------------------
>
> Key: OPENNLP-1838
> URL: https://issues.apache.org/jira/browse/OPENNLP-1838
> Project: OpenNLP
> Issue Type: Bug
> Components: dl, Tokenizer
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Follow-up to OPENNLP-1837 / PR #1073.
> OPENNLP-1837 adds {{opennlp.tools.tokenize.BertTokenizer}} (full BERT basic
> tokenization + wordpiece). {{AbstractDL.createTokenizer()}} still builds a
> bare
> {{WordpieceTokenizer}}, so all {{opennlp-dl}} components feed unnormalized
> text to the vocabulary lookup:
> * {{SentenceVectorsDL}}
> * {{DocumentCategorizerDL}}
> * {{NameFinderDL}}
> h3. Impact
> For *uncased* models (both models recommended by the opennlp-dl README are
> uncased — {{sentence-transformers/all-MiniLM-L6-v2}} and
> {{nlptown/bert-base-multilingual-uncased-sentiment}}), every capitalized or
> accented word becomes {{[UNK]}} on the default path.
> Measured embedding fidelity vs. the HuggingFace reference for all-MiniLM-L6-v2
> was cosine 0.09–0.57 with bare {{WordpieceTokenizer}}; with {{BertTokenizer}}
> it exceeds 0.999999 (see OPENNLP-1837 validation).
> h3. Proposal
> * Switch {{AbstractDL.createTokenizer()}} (or each {{*DL}} component) to use
> {{BertTokenizer}} with an explicit uncased/cased configuration flag.
> * Default to uncased ({{lowerCase=true}}) to match the recommended models.
> * Separately evaluate mean pooling for {{SentenceVectorsDL}} (currently
> returns raw CLS hidden state; sentence-transformers uses masked mean + L2
> normalization) — may be a separate ticket if scope is too large.
> h3. Links
> * OPENNLP-1837: https://issues.apache.org/jira/browse/OPENNLP-1837
> * PR #1073: https://github.com/apache/opennlp/pull/1073
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
