krickert commented on PR #1165: URL: https://github.com/apache/opennlp/pull/1165#issuecomment-5115440515
All three are in. 1. `BertTokenizer` is back in `opennlp-api` as a shim over `WordpieceEncoder`: `@Deprecated(since = "3.0.0", forRemoval = true)`, the original three constructors, `tokenize()` delegating to `encodeToPieces()`, `tokenizePos()` throwing with the original message. Ids are synthesized from the set order, with the comment, since the `tokenize()` path never reads them. `EncoderTokenizer` is gone. 2. `ReferenceBertPipeline` is gone; the curated and randomized differential tests now run against the shim, and a new `BertTokenizerTest` pins the constructors, the default special token chain, and the exact `tokenizePos` message. The independent expected sequences stay in `WordpieceEncoderReferenceSequencesTest`. 3. `AbstractDL.createTokenizer` returns `BertTokenizer` again, so the old override descriptor holds, and `createPipelineTokenizer` hands back the shim. The compatibility check before deleting the old pipeline surfaced a real divergence: the encoder kept U+2028 and U+2029 inside words while the old `WhitespaceTokenizer` split on them, so a word carrying a line or paragraph separator collapsed to `[UNK]`. Fixed in `cleanAndIsolateCjk` (Zl and Zp map to a space now, matching reference BERT's `str.split()`), with a span-asserting regression test. The old fuzz pool never contained those characters, which is how it survived the differential tests. One deliberate drift to follow convention, documented in the `@throws` clauses: the shim rejects nulls with `IllegalArgumentException` rather than the old `Objects.requireNonNull` NPE, and it fails at construction when a special token is missing from the vocabulary instead of tokenizing toward unmappable pieces. Both follow the null-contract convention this branch was reviewed to. Say so if you want the old NPE behavior kept instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
