krickert opened a new pull request, #1152: URL: https://github.com/apache/opennlp/pull/1152
Adds a new `opennlp-extensions/opennlp-embeddings` module: a pure-JVM engine for modern static embedding tables (Model2Vec-family distillations, the 2024/2025 successors to word2vec/GloVe: same flat per-token table shape, sentence-transformer semantics, inference is pure lookup). **What's in it:** - `SafetensorsFile`: reads the safetensors format with a purpose-built cursor parser for the JSON header (no third-party JSON dependency, F32 decode only). safetensors carries no executable content, unlike pickle-based checkpoints, so loading is safe by construction. The embedding matrix is auto-detected as the single 2-D F32 tensor, failing loud and listing candidates on ambiguity rather than guessing a key-name convention. - `WordPieceVocabulary`: BERT-style `vocab.txt`, line number = embedding row id. - `StaticEmbeddingModel`: embeds through the existing `BertTokenizer`/`WordpieceTokenizer`, reused unchanged. The pooling formula is verified against the Model2Vec Python and official Rust reference implementations: `[CLS]`/`[SEP]` never pooled, unknown tokens dropped from sum and denominator, optional per-row `weights` tensor, token-count denominator, epsilon-floored normalization. - Convenience surface: `similarity`, `mostSimilar` (bounded top-K over precomputed row norms), `analogy` (input terms excluded by folding through the model's own tokenizer). **Posture:** code only, bring your own table. No model bundled, nothing fetched at build or run time, no new dependencies. **Thread safety:** immutable, `@ThreadSafe`, with an 8-thread concurrency test comparing every result against the single-threaded reference. **Measured** (JMH, opt-in `jmh` profile matching opennlp-runtime's pattern; fixture at real published-table scale, 29,528 x 256): `embed` ~766k short sentences/s on one core (1.04M ops/s of 5 sentences at 32 threads); full-vocabulary top-10 scan 649/s per core, ~9.2k/s at 32 threads. Verification: opennlp-embeddings 43/0, `mvn verify` green including checkstyle and forbiddenapis. Follow-ups (deliberately out of this PR): the gRPC `EmbeddingProvider` backend in opennlp-sandbox, a concurrent-load comparison against a Python baseline, an ANN index for `mostSimilar`, and bundled-default-model license diligence. https://issues.apache.org/jira/browse/OPENNLP-1877 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
