Kristian Rickert created OPENNLP-1837:
-----------------------------------------
Summary: opennlp-dl tokenization lacks BERT basic tokenization;
embeddings from uncased models are degraded and cannot be used
Key: OPENNLP-1837
URL: https://issues.apache.org/jira/browse/OPENNLP-1837
Project: OpenNLP
Issue Type: Bug
Components: dl, Tokenizer
Affects Versions: 3.0.0-M3
Reporter: Kristian Rickert
Assignee: Kristian Rickert
h2. Summary
{{WordpieceTokenizer}} implements only the wordpiece (subword) stage of BERT
tokenization. The basic tokenization stage that every reference BERT pipeline
runs first -- lower casing and accent stripping for uncased models, control
character cleanup, per-character punctuation splitting, CJK isolation -- is
missing entirely, and nothing in {{opennlp-dl}} compensates for it.
h2. Impact
With uncased models, every capitalized word tokenizes to {{[UNK]}} -- including
sentence-initial "The":
{code}
Input: "The quick brown fox jumps over the lazy dog."
opennlp: [CLS] [UNK] quick brown fox jumps over the lazy dog . [SEP]
reference: [CLS] the quick brown fox jumps over the lazy dog . [SEP]
Input: "OpenNLP now serves embeddings over gRPC."
opennlp: [CLS] [UNK] now serves em ##bed ##ding ##s over g [UNK] . [SEP]
reference: [CLS] open ##nl ##p now serves em ##bed ##ding ##s over gr ##pc .
[SEP]
{code}
The {{opennlp-dl}} README recommends {{sentence-transformers/all-MiniLM-L6-v2}}
and {{nlptown/bert-base-multilingual-uncased-sentiment}} -- both *uncased* --
so every documented path through {{SentenceVectorsDL}},
{{DocumentCategorizerDL}} and {{NameFinderDL}} is affected, silently.
Measured embedding fidelity of {{SentenceVectorsDL}}-style output vs. the
Python reference for {{all-MiniLM-L6-v2}} is cosine *0.09 - 0.57* on a diverse
sentence set; with corrected tokenization (and mean pooling) it is *>
0.9999997*. Inference itself is not at fault: feeding identical token ids to
the same ONNX model through ONNX Runtime in Java and Python produces cosine
1.000000.
h2. Additional defects found in WordpieceTokenizer
# {{tokenizePos()}} returns {{null}} (the released code contains {{// TODO:
Implement this.}}).
# The punctuation regex {{\\p\{Punct\}+}} treats punctuation runs as one token
({{"Wait..."}} produces one {{[UNK]}} instead of three {{.}} tokens) and misses
non-ASCII punctuation.
# Words that only partially match vocabulary pieces emit the matched prefix
pieces followed by {{[UNK]}} (e.g. {{"brownfox"}} -> {{brown}}, {{[UNK]}}),
where the reference implementation replaces the entire word with a single
{{[UNK]}}.
h2. Proposal
* Add a new {{opennlp.tools.tokenize.BertTokenizer}} implementing the full BERT
pipeline: basic tokenization (control character removal, whitespace
normalization, CJK ideograph isolation, optional lower casing with NFD accent
stripping, per-character punctuation isolation) followed by wordpiece
tokenization. Normalization is on by default; cased models opt out via
constructor flag.
* Fix the three {{WordpieceTokenizer}} defects directly: throw
{{UnsupportedOperationException}} from {{tokenizePos()}} instead of returning
{{null}}, switch to per-character Unicode-aware punctuation splitting, and
replace partially matched words with a single unknown token.
* Recommend that {{opennlp-dl}} components ({{SentenceVectorsDL}},
{{DocumentCategorizerDL}}, {{NameFinderDL}}) adopt {{BertTokenizer}} as their
default tokenization so uncased models work correctly out of the box.
h2. Validation
The patch includes unit tests whose expected token sequences were generated
with the HuggingFace {{tokenizers}} reference implementation using the same
vocabularies. Additionally, {{BertTokenizer}} was verified byte-identical to
the reference on the real {{bert-base-uncased}} vocabulary (30k entries) across
a corpus covering capitalization, diacritics, punctuation runs, CJK text, URLs,
numbers and mixed whitespace (12/12 sentences identical).
h2. Related
OPENNLP-1836 fixed the attention-mask encoding in {{SentenceVectorsDL}}; this
issue addresses the tokenization layer in front of it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
