krickert commented on PR #1165: URL: https://github.com/apache/opennlp/pull/1165#issuecomment-4940870944
Single-thread throughput measurement after 3fb8b6bf, for the record. Machine: AMD Ryzen 9 9950X3D (16 cores, one thread used), Linux, OpenJDK 25.0.3. Workload: 100,100 short texts (77 distinct lines cycled), t5-small unigram model, 32k vocabulary, 1.17M pieces total, 3 warmup passes over the corpus, then 5 timed passes. - opennlp-subword: 6.47M pieces/s (554k texts/s), producing piece, id, and original-text span for every token - Reference implementation, sentencepiece 0.2.1 via its Python binding, one encode call per text, ids only: 4.57M pieces/s (391k texts/s) - opennlp-subword before the optimization commit: 2.83M pieces/s That is 1.42x the reference on the same corpus and model, measured call for call from a host language, so the binding's per-call overhead is included in the reference number; the raw C++ core inside a batch loop is faster than that number. The Java side also does more work per token, since it maps every piece back to a UTF-16 span of the original input, which the reference does not produce against the original string. Output is unchanged: the bundled parity fixtures and the T5-small and ALBERT real-model fixtures assert identical pieces, ids, spans, and normalized forms against the reference before and after the optimization commit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
