krickert commented on code in PR #1462:
URL: https://github.com/apache/opennlp/pull/1462#discussion_r4282326514
##########
opennlp-docs/src/docbkx/machine-learning.xml:
##########
@@ -183,5 +183,26 @@ try (TextEmbedder embedder =
providers.select(spec).create(spec)) {
has no provider until another one is installed.
Existing <code>SentenceVectorsDL</code>
constructors remain available.
</para>
+ <para>
+ To compute the vectors on a GPU, depend on
<code>opennlp-dl-gpu</code> in place of
+ <code>opennlp-dl</code> and pass
<code>InferenceOptions</code> to
+ <code>SentenceVectorsDL</code>.
<code>opennlp-dl-gpu</code> supports linux-x64 and
Review Comment:
This is an area I love to code in - so I'll give ya a quick low-down.
MacOS won't work on an Nvidia GPU - if it does we should state explicitly
that it's unsupported. So there *is* a (slow) way to get Mac to work on ONNX,
but Macs should just use the mac steel API that comes with every mac OS and
phone. Mac w/Nvidia is largely unsupported. Given the marriage of NVidia and
windows, I suspect that gap is only going to widen.
Mac's do not do containers, so there's no real way to test CUDA against it
unless you have a 3rd party library that can tie it together (maybe there's an
OSS way? Again, I wouldn't support that). But it only really exists in
CUDA-land.
I plan to make an add-on that uses JNI to use Mac steel. It's by far the
fastest way to make embeddings for the mac and I've heavily tested it (I've
integrated on a couple OSS projects already). In the future, I'd recommend
most devs just use model2vec for default day-to-day development for fast
embeddings that won't pin your CPU, which will be another add-on and capable of
producing 100k vectors/sec that are usable (but at times weak) embeddings.
I also have an NVidia setup with ARM - and that will not work at this time
either. I *can* get it to work but that would likely involve more
dependencies, so it should be an add-on as well. However, this setup is
probably the most common in a cloud environment because gravitron GPUs on AWS
(the most popular setup) are ARM/CUDA and a consumer cloud cheapest option in
AWS (@jzonthemtn correct me if I'm wrong there - I suspect you got more AWS
access than I do). Also, the new Windoze laptops are also ARM/NVidia setups -
so that will certainly be a popular developer option in the near future.
So for now:
ANY architecture = CPU works. NVidia ignored (like today)
x86 + CUDA win+Linux = only thing that should work (or else it's CPU). If
acceleration works on anything else, I'd be shocked :) However, expect a
10-40x+ speed increase when it does, especially w/batching (it's sill sorta
slow without batching) :+1:
If you try any other setup at startup and say you want CUDA, it errors out
(my ARM setups and OSX setups do this right) - you have to specify acceleration
for above only for now.
Future support w/addons:
| Hardware | Backend | JDK | Notes |
|--------|--------|--------|--------|
| OSX | Apple Metal API via JNI | JDK 21-25 | Likely at the time of release,
only most recent version of Mac |
| OSX | ONNX | N/A | Will not be supported, it's slow unless someone wants
to maintain a slower than metal lane|
| OSX | Apple Metal API via FFM | JDK25+ | Should be about 2x faster than
JNI. If it's not, unsupported. |
| Windows/Linux intel | ONNX | JDK 21-25 | Default support and only setup in
mainline due to dependency bloat |
| Windows/Linux Intel | CUDA direct | JDK 25 / FFM | Will be about 2.5-5x
faster than ONNX but only JDK 25 with CUDA |
| Apple on OSX with NVidia chip | N/A | N/A | Unsupported, extremely rare.
Apple won't support anything with NVidia's name on it |
| Windows/ARM NVidia | ONNX | JDK 21-25 | Support via addon or *maybe* some
logic on main. Uncommon now but by next year this will be extrememly popular |
| Linux/ARM NVidia | CUDA | JDK 25 / FFM | Will be supported via an add-on,
similar to x86 and will become extremely common imho |
| Raspberry Pi hailo 10H HAT | Hailio w/ Pi5 8GB acceleration | JDK21-25 |
Supported via JNI, won't do an FFM binding - too rare of a setup |
I've successfully tested most of these setups locally.
It's been nearly a month since I tested this on Windoze - I'll boot into it
(after 2 hours of updates I'm sure - thank you CVEs). But I'm sure it'll work
there, but I'll test and get back to you.
It will also work on Windows with Docker GPU via Linux acceleration - but we
can make an entire blog post to teach a developer how to do that. It'll be a
popular post, a lot of windows people like to do this for some reason.
But all setups 100% work on CPU, but x86 only for acceleration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
