This is an automated email from the ASF dual-hosted git repository.
krickert pushed a change to branch sentencepiece
in repository https://gitbox.apache.org/repos/asf/opennlp.git
from 1c0af89a0 OPENNLP-1885: Address review: validation message style,
shared test fixtures, thread safety wording
add 241b70718 OPENNLP-1885: Deprecate BertTokenizer as a shim over
WordpieceEncoder per review
No new revisions were added by this update.
Summary of changes:
.../java/opennlp/tools/tokenize/BertTokenizer.java | 125 +++++++++++++++++++++
.../opennlp/tools/tokenize/WordpieceEncoder.java | 16 ++-
.../src/main/java/opennlp/dl/AbstractDL.java | 37 +++---
.../src/main/java/opennlp/dl/EncoderTokenizer.java | 59 ----------
.../opennlp/tools/tokenize/BertTokenizerTest.java | 102 +++++++++++++++++
.../tools/tokenize/ReferenceBertPipeline.java | 94 ----------------
.../tools/tokenize/WordpieceEncoderTest.java | 35 ++++--
opennlp-docs/src/docbkx/tokenizer.xml | 6 +-
8 files changed, 289 insertions(+), 185 deletions(-)
create mode 100644
opennlp-api/src/main/java/opennlp/tools/tokenize/BertTokenizer.java
delete mode 100644
opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/EncoderTokenizer.java
create mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/BertTokenizerTest.java
delete mode 100644
opennlp-core/opennlp-runtime/src/test/java/opennlp/tools/tokenize/ReferenceBertPipeline.java
