krickert opened a new pull request, #1152: URL: https://github.com/apache/opennlp/pull/1152
> **Stacked on #1165** (`sentencepiece`). This PR is based on the `sentencepiece` branch so the SentencePiece models load; merge #1165 first, then retarget this PR to `main`. Adds a new `opennlp-extensions/opennlp-embeddings` module: a pure-JVM engine for modern static embedding tables (Model2Vec-family distillations, the 2024/2025 successors to word2vec/GloVe: same flat per-token table shape, sentence-transformer semantics, inference is pure lookup). **Positioning:** this is the speed tier of text embedding, not a replacement for anything. The ONNX `SentenceVectorsDL` in opennlp-dl runs the actual transformer and produces contextual vectors, so this isn't made to be a replacement as that's the OOTB accuracy tier. This module trades that context for a static table: no native runtime, no GPU, and throughput in the hundreds of thousands of texts per second per core. Different pipeline stages want different points on that curve. Matches the philosophy of this style of embedding. **What's in it:** - `TextEmbedder` (new interface in `opennlp-api`, package `opennlp.tools.embeddings`): the text-level embedding contract. `embed(CharSequence)`, `embedAll(List)` (default implementation loops; runtimes that batch efficiently should override), and `dimension()`. It is the text-level counterpart of the existing word-level `WordVectorTable`, and the javadoc states that layer difference explicitly. `StaticEmbeddingModel` is the first implementation. `SentenceVectorsDL` is the natural second one: adopting the interface is purely additive (its existing constructors and `getVectors` are untouched), and its ONNX runtime is exactly what the overridable `embedAll` batch method exists for. That adoption is a separate discussion, not part of this PR. - `SafetensorsFile`: reads the safetensors format with a purpose-built cursor parser for the JSON header (no third-party JSON dependency, F32 decode only). safetensors carries no executable content, unlike pickle-based checkpoints, so loading is safe by construction. The embedding matrix is auto-detected as the single 2-D F32 tensor, failing loud and listing candidates on ambiguity rather than guessing a key-name convention. - `WordpieceVocabulary`: BERT-style `vocab.txt`, line number = embedding row id. (Named to match the existing `WordpieceTokenizer` casing.) - `StaticEmbeddingModel`: embeds through the existing `BertTokenizer`/`WordpieceTokenizer`, reused unchanged. The pooling formula is verified against the Moust reference implementations:`[CLS]`/`[SEP]` never pooled, unknown tokens dropped from sum and denominator, optional per-row `weights` tensor, token-count denominator, epsilon-floored normalization. - Convenience surface: `similarity`, `mostSimilar` (bounded top-K over precomputed row norms), `analogy` (input terms excluded by folding through the mode **Posture:** code only, bring your own tab fetched at build or run time, no new dependencies. **Thread safety:** immutable, `@ThreadSafe`, with an 8-thread concurrency test comparing every result against the single-threaded reference. **Measured** (JMH, opt-in `jmh` profile matching opennlp-runtime's pattern; fixture at real published-table scale, 29,528 x 256): `embed` ~766k short sentences/s on one core (1.04M ops/s of 5 sentences at 32 threads); full-vocabulary top-10 scan 649/s per core, ~9.2k/s at 32 threads. Verification: opennlp-embeddings 43/0 plus the new interface contract test, `mvn verify` green including checkstyle and forbiddenapis. Follow-ups (deliberately out of this PR): `TextEmbedder` with a real batched`embedAll` (pending discussion with its author), the gRPC backend in opennlp-sandbox, a concurrent-load comparison against a Python baseline, an ANN index for `mostSimilar`, and bundled-default-model license diligence. https://issues.apache.org/jira/browse/OPEN-1877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
