[
https://issues.apache.org/jira/browse/OPENNLP-1837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Wiesner updated OPENNLP-1837:
------------------------------------
Issue Type: Improvement (was: Bug)
> Add BertTokenizer with BERT basic tokenization
> ----------------------------------------------
>
> Key: OPENNLP-1837
> URL: https://issues.apache.org/jira/browse/OPENNLP-1837
> Project: OpenNLP
> Issue Type: Improvement
> Components: dl, Tokenizer
> Affects Versions: 3.0.0-M3
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Critical
> Fix For: 3.0.0-M4, 3.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> h2. Summary
> {{WordpieceTokenizer}} implements only the wordpiece (subword) stage of BERT
> tokenization. The basic tokenization stage that every reference BERT pipeline
> runs first -- lower casing and accent stripping for uncased models, control
> character cleanup, per-character punctuation splitting, CJK isolation -- is
> missing entirely, and nothing in {{opennlp-dl}} compensates for it.
> h2. Impact
> With uncased models, every capitalized word tokenizes to {{[UNK]}} --
> including sentence-initial "The":
> {code}
> Input: "The quick brown fox jumps over the lazy dog."
> opennlp: [CLS] [UNK] quick brown fox jumps over the lazy dog . [SEP]
> reference: [CLS] the quick brown fox jumps over the lazy dog . [SEP]
> Input: "OpenNLP now serves embeddings over gRPC."
> opennlp: [CLS] [UNK] now serves em ##bed ##ding ##s over g [UNK] . [SEP]
> reference: [CLS] open ##nl ##p now serves em ##bed ##ding ##s over gr ##pc .
> [SEP]
> {code}
> The {{opennlp-dl}} README recommends
> {{sentence-transformers/all-MiniLM-L6-v2}} and
> {{nlptown/bert-base-multilingual-uncased-sentiment}} -- both *uncased* -- so
> every documented path through {{SentenceVectorsDL}},
> {{DocumentCategorizerDL}} and {{NameFinderDL}} is affected, silently.
> Measured embedding fidelity of {{SentenceVectorsDL}}-style output vs. the
> Python reference for {{all-MiniLM-L6-v2}} is cosine *0.09 - 0.57* on a
> diverse sentence set; with corrected tokenization (and mean pooling) it is *>
> 0.9999997*. Inference itself is not at fault: feeding identical token ids to
> the same ONNX model through ONNX Runtime in Java and Python produces cosine
> 1.000000.
> h2. Additional defects found in WordpieceTokenizer
> # {{tokenizePos()}} returns {{null}} (the released code contains {{// TODO:
> Implement this.}}).
> # The punctuation regex {{\\p\{Punct\}+}} treats punctuation runs as one
> token ({{"Wait..."}} produces one {{[UNK]}} instead of three {{.}} tokens)
> and misses non-ASCII punctuation.
> # Words that only partially match vocabulary pieces emit the matched prefix
> pieces followed by {{[UNK]}} (e.g. {{"brownfox"}} -> {{brown}}, {{[UNK]}}),
> where the reference implementation replaces the entire word with a single
> {{[UNK]}}.
> h2. Proposal
> * Add a new {{opennlp.tools.tokenize.BertTokenizer}} implementing the full
> BERT pipeline: basic tokenization (control character removal, whitespace
> normalization, CJK ideograph isolation, optional lower casing with NFD accent
> stripping, per-character punctuation isolation) followed by wordpiece
> tokenization. Normalization is on by default; cased models opt out via
> constructor flag.
> * Fix the three {{WordpieceTokenizer}} defects directly: throw
> {{UnsupportedOperationException}} from {{tokenizePos()}} instead of returning
> {{null}}, switch to per-character Unicode-aware punctuation splitting, and
> replace partially matched words with a single unknown token.
> * Recommend that {{opennlp-dl}} components ({{SentenceVectorsDL}},
> {{DocumentCategorizerDL}}, {{NameFinderDL}}) adopt {{BertTokenizer}} as their
> default tokenization so uncased models work correctly out of the box.
> h2. Validation
> The patch includes unit tests whose expected token sequences were generated
> with the HuggingFace {{tokenizers}} reference implementation using the same
> vocabularies. Additionally, {{BertTokenizer}} was verified byte-identical to
> the reference on the real {{bert-base-uncased}} vocabulary (30k entries)
> across a corpus covering capitalization, diacritics, punctuation runs, CJK
> text, URLs, numbers and mixed whitespace (12/12 sentences identical).
> h2. Related
> OPENNLP-1836 fixed the attention-mask encoding in {{SentenceVectorsDL}}; this
> issue addresses the tokenization layer in front of it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
