[
https://issues.apache.org/jira/browse/OPENNLP-1982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18117189#comment-18117189
]
Kristian Rickert commented on OPENNLP-1982:
-------------------------------------------
Since this works fine now, we should wait until after 3.0 since this means a
*ton* of testing that is GPU/CPU/OS/JDK dependent - you know, the opposite of
everything java is marketed as with code once, run everywhere.
So everything after this is more roadmap-y stuff:
The code fix itself should be small - the hard part is testing. Testing is
insane for this sorta stuff (multiple architectures, OS, memory setups,
fallback questions, thread pools, JDK compat., etc - it's hard testing work).
I'd suggest looking at DJL for inspiration. They use huggingface tokenizers
though, so they may edge us out due to that (it's wrtten in rust and already
regex free).
I've integrated many libraries that do similar CUDA integration, this should
speed up our API by anywhere from 15-50x :) I did some initial tests and it
suggests the same.
It won't be a huge code change; I'd love to collaborate with anyone on this one
but a long evening with a capable frontier model you can probably get most of
the testing done. There's a lot of ways to do this, and it's fun to profile
(if that's the sort of coding you like to do). I'd at least make it a
requirement to have a ton of tests if someone takes this on and limit scope to
only NVidia/Cuda with CPU fallback as a first run.
I'd further recommend apple silicon's API, AMD, intel battleimage, intel NPU as
a 3.x release. It's too bad it's not in JDK21, but JDK25 has bindings that can
make this much faster too. This area is only going to grow!
I've done some of this on the grpc server awhile back
> SentenceVectorsDL cannot run on a GPU
> -------------------------------------
>
> Key: OPENNLP-1982
> URL: https://issues.apache.org/jira/browse/OPENNLP-1982
> Project: OpenNLP
> Issue Type: Improvement
> Reporter: Richard Zowalla
> Priority: Major
> Fix For: 3.0.0
>
>
> {{SentenceVectorsDL}} always creates default {{OrtSession.SessionOptions}},
> so unlike {{NameFinderDL}} and {{DocumentCategorizerDL}} it cannot use the
> CUDA execution provider through {{InferenceOptions}}. Add a constructor that
> takes {{InferenceOptions}} and, once OPENNLP\-1937 is in, expose the device
> \(e.g. {{cpu}} / {{cuda:N}}\), a maximum batch size and the intra\-op thread
> count as options of the ONNX text embedder provider.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
