This is an automated email from the ASF dual-hosted git repository.
rzo1 pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/opennlp.git
The following commit(s) were added to refs/heads/main by this push:
new 2cad1f29f GH-1389, GH-1390: Let SentenceVectorsDL run on a GPU and fix
the first CUDA request in a fresh JVM (#1462)
2cad1f29f is described below
commit 2cad1f29ff8d980c0faaacec7e901af919531e22
Author: Martin Wiesner <[email protected]>
AuthorDate: Sun Oct 18 14:32:09 2026 +0200
GH-1389, GH-1390: Let SentenceVectorsDL run on a GPU and fix the first CUDA
request in a fresh JVM (#1462)
SentenceVectorsDL always created default session options, so it ran on
the CPU even with opennlp-dl-gpu. A new constructor takes
InferenceOptions, like NameFinderDL and DocumentCategorizerDL, and uses
the GPU settings and the lower case setting from them; a lower case
setting in the options takes precedence over the lowerCase argument. The
existing constructors keep their behavior.
AbstractDL.sessionOptions added the CUDA provider before any
OrtEnvironment existed. Loading the provider needs ONNX Runtime's default
logger, which only exists once an environment does, so the first GPU
request in a fresh JVM failed with "Attempt to use DefaultLogger but none
has been registered". The environment is now created first. This affects
NameFinderDL and DocumentCategorizerDL as well. When the provider cannot
be added, the session options created for the request are closed before
the exception is thrown.
A test makes the first CUDA request of a child JVM and requires that the
provider is added or rejected for a reason other than the missing
default logger; it runs in opennlp-dl-gpu and fails with the AbstractDL
of main. Other tests request a device id no machine has and check that ONNX
Runtime rejects it for SentenceVectorsDL, NameFinderDL and
DocumentCategorizerDL, and compare vectors on CUDA device 0 with the CPU
session (skipped without a CUDA provider or device). The tiny model test
fixtures are shared in TinyVectorModels.
The opennlp-dl-gpu test classpath no longer has the CPU onnxruntime next
to onnxruntime_gpu. Since onnxruntime_gpu has native libraries for
linux-x64 and win-x64 only, the opennlp-dl-gpu tests are skipped on
macOS and aarch64. The machine learning chapter documents the GPU
setup and the supported platforms.
Fixes #1389
Fixes #1390
Signed-off-by: Kristian Rickert <[email protected]>
Co-authored-by: Kristian Rickert <[email protected]>
Co-authored-by: Richard Zowalla <[email protected]>
Co-authored-by: Claude Code 2.1.296 (Opus 5.5) <[email protected]>
---
opennlp-core/opennlp-ml/opennlp-dl-gpu/pom.xml | 34 ++++
.../src/main/java/opennlp/dl/AbstractDL.java | 22 ++-
.../opennlp/dl/doccat/DocumentCategorizerDL.java | 10 +-
.../java/opennlp/dl/namefinder/NameFinderDL.java | 5 +-
.../java/opennlp/dl/vectors/SentenceVectorsDL.java | 47 ++++-
.../opennlp/dl/AbstractDLColdStartCudaTest.java | 139 +++++++++++++++
.../src/test/java/opennlp/dl/GpuRequestTest.java | 116 +++++++++++++
.../src/test/java/opennlp/dl/LoadVocabTest.java | 10 +-
.../src/test/java/opennlp/dl/OnnxTestSupport.java | 83 +++++++++
.../dl/vectors/SentenceVectorsDLEmbedderTest.java | 59 ++-----
.../SentenceVectorsDLInferenceOptionsTest.java | 189 +++++++++++++++++++++
.../java/opennlp/dl/vectors/TinyVectorModels.java | 112 ++++++++++++
opennlp-docs/src/docbkx/machine-learning.xml | 21 +++
13 files changed, 784 insertions(+), 63 deletions(-)
diff --git a/opennlp-core/opennlp-ml/opennlp-dl-gpu/pom.xml
b/opennlp-core/opennlp-ml/opennlp-dl-gpu/pom.xml
index cf98bb012..0eb2753a1 100644
--- a/opennlp-core/opennlp-ml/opennlp-dl-gpu/pom.xml
+++ b/opennlp-core/opennlp-ml/opennlp-dl-gpu/pom.xml
@@ -79,12 +79,19 @@
<artifactId>slf4j-simple</artifactId>
<scope>test</scope>
</dependency>
+ <!-- Keep the CPU onnxruntime off the test classpath; this module uses
onnxruntime_gpu. -->
<dependency>
<groupId>org.apache.opennlp</groupId>
<artifactId>opennlp-dl</artifactId>
<version>${project.version}</version>
<type>test-jar</type>
<scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>com.microsoft.onnxruntime</groupId>
+ <artifactId>onnxruntime</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
</dependencies>
@@ -102,5 +109,32 @@
</plugins>
</build>
+ <!-- onnxruntime_gpu ships native libraries for linux-x64 and win-x64 only,
unlike the
+ CPU onnxruntime jar, which also covers macOS and Linux aarch64. The
tests cannot run there. -->
+ <profiles>
+ <profile>
+ <id>no-onnxruntime-gpu-on-mac</id>
+ <activation>
+ <os>
+ <family>mac</family>
+ </os>
+ </activation>
+ <properties>
+ <skipTests>true</skipTests>
+ </properties>
+ </profile>
+ <profile>
+ <id>no-onnxruntime-gpu-on-aarch64</id>
+ <activation>
+ <os>
+ <arch>aarch64</arch>
+ </os>
+ </activation>
+ <properties>
+ <skipTests>true</skipTests>
+ </properties>
+ </profile>
+ </profiles>
+
</project>
\ No newline at end of file
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
index 9afe7b372..9ef3e35e2 100644
---
a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/AbstractDL.java
@@ -216,17 +216,31 @@ public abstract class AbstractDL implements AutoCloseable
{
* @param inferenceOptions The inference options to read the GPU
configuration from.
* @return The configured session options.
*
- * @throws OrtException Thrown if the CUDA execution provider cannot be
added.
- * @throws IllegalArgumentException Thrown if {@code inferenceOptions} is
{@code null} or its
- * split sizes are invalid, see {@link #validateSplitOptions(int, int)}.
+ * @throws OrtException Thrown if the CUDA execution provider cannot be
added: the ONNX Runtime
+ * on the classpath has no CUDA support, its CUDA libraries cannot be
loaded, or no device
+ * answers to the configured device id. ONNX Runtime does not fall back
to the CPU. The
+ * session options created for the request are closed before the
exception is thrown.
+ * @throws IllegalArgumentException Thrown if {@code inferenceOptions} is
{@code null}, its
+ * split sizes are invalid, see {@link #validateSplitOptions(int, int)},
or it requests a GPU
+ * with a negative device id.
*/
protected static OrtSession.SessionOptions sessionOptions(final
InferenceOptions inferenceOptions)
throws OrtException {
ParamChecks.requireNonNullArg(inferenceOptions, "inferenceOptions");
validateSplitOptions(inferenceOptions);
+ if (inferenceOptions.isGpu()) {
+ ParamChecks.requireNonNegative(inferenceOptions.getGpuDeviceId(),
"gpuDeviceId");
+ }
final OrtSession.SessionOptions sessionOptions = new
OrtSession.SessionOptions();
if (inferenceOptions.isGpu()) {
- sessionOptions.addCUDA(inferenceOptions.getGpuDeviceId());
+ try {
+ // addCUDA needs ONNX Runtime's default logger, which exists once an
environment does.
+ OrtEnvironment.getEnvironment();
+ sessionOptions.addCUDA(inferenceOptions.getGpuDeviceId());
+ } catch (OrtException | RuntimeException e) {
+ sessionOptions.close();
+ throw e;
+ }
}
return sessionOptions;
}
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
index d2c1c2570..5010cce60 100644
---
a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/doccat/DocumentCategorizerDL.java
@@ -109,8 +109,9 @@ public class DocumentCategorizerDL extends AbstractDL
implements DocumentCategor
* unsupported layout, or sets a lower casing that disagrees with {@code
inferenceOptions}.
* @throws IllegalArgumentException Thrown if {@code model}, {@code
vocabulary},
* {@code categories}, {@code classificationScoringStrategy} or {@code
inferenceOptions} is
- * {@code null}, if the split sizes of {@code inferenceOptions} are
invalid, or if the
- * vocabulary lacks the classification, separator or unknown token.
+ * {@code null}, if the split sizes of {@code inferenceOptions} are
invalid, if it
+ * requests a GPU with a negative device id, or if the vocabulary lacks
the
+ * classification, separator or unknown token.
*/
public DocumentCategorizerDL(File model, File vocabulary, Map<Integer,
String> categories,
ClassificationScoringStrategy
classificationScoringStrategy,
@@ -148,8 +149,9 @@ public class DocumentCategorizerDL extends AbstractDL
implements DocumentCategor
* keys to strings, or a key is not an integer.
* @throws IllegalArgumentException Thrown if {@code model}, {@code
vocabulary}, {@code config},
* {@code classificationScoringStrategy} or {@code inferenceOptions} is
{@code null}, if the
- * split sizes of {@code inferenceOptions} are invalid, or if the
vocabulary lacks the
- * classification, separator or unknown token.
+ * split sizes of {@code inferenceOptions} are invalid, if it requests a
GPU with a
+ * negative device id, or if the vocabulary lacks the classification,
separator or unknown
+ * token.
*/
public DocumentCategorizerDL(File model, File vocabulary, File config,
ClassificationScoringStrategy
classificationScoringStrategy,
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
index 99358f549..afafe2e40 100644
---
a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/namefinder/NameFinderDL.java
@@ -169,8 +169,9 @@ public class NameFinderDL extends AbstractDL implements
OffsetMappingNameFinder
* the lower casing the component is configured with.
* @throws IllegalArgumentException Thrown if {@code model}, {@code
vocabulary},
* {@code ids2Labels}, {@code inferenceOptions} or {@code
sentenceDetector} is
- * {@code null}, if the split sizes of {@code inferenceOptions} are
invalid, or if the
- * vocabulary lacks the classification, separator or unknown token.
+ * {@code null}, if the split sizes of {@code inferenceOptions} are
invalid, if it
+ * requests a GPU with a negative device id, or if the vocabulary lacks
the
+ * classification, separator or unknown token.
*/
public NameFinderDL(File model, File vocabulary, Map<Integer, String>
ids2Labels,
InferenceOptions inferenceOptions,
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
index 8f6c080ff..08e114d83 100644
---
a/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/main/java/opennlp/dl/vectors/SentenceVectorsDL.java
@@ -35,6 +35,7 @@ import ai.onnxruntime.OrtSession;
import ai.onnxruntime.TensorInfo;
import opennlp.dl.AbstractDL;
+import opennlp.dl.InferenceOptions;
import opennlp.dl.Tokens;
import opennlp.tools.commons.ThreadSafe;
import opennlp.tools.embeddings.EmbeddingException;
@@ -156,12 +157,56 @@ public class SentenceVectorsDL extends AbstractDL
implements TextEmbedder {
* {@code [batch, hidden]} or {@code [batch, tokens, hidden]}.
* @throws OrtException Thrown if the {@code model} cannot be loaded.
* @throws IOException Thrown if errors occurred loading the {@code model}
or {@code vocabulary}.
+ * @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code
vocabulary}
+ * is malformed, has an unsupported layout, or sets a lower casing that
disagrees with
+ * the lower casing the component is configured with.
*/
public SentenceVectorsDL(final File model, final File vocabulary, final
boolean lowerCase,
final Pooling pooling, final boolean normalize, final int maxLength)
throws OrtException, IOException {
- super(model, vocabulary, new OrtSession.SessionOptions(), lowerCase);
+ this(model, vocabulary, lowerCase, pooling, normalize, maxLength, new
InferenceOptions());
+
+ }
+
+ /**
+ * Instantiates a {@link SentenceVectorsDL sentence vector generator} using
ONNX models and the
+ * given {@link InferenceOptions}, so the session can run on a GPU through
+ * {@code opennlp-dl-gpu}.
+ *
+ * @param model The file name of a sentence vectors ONNX model.
+ * @param vocabulary The file name of the vocabulary file for the model.
+ * @param lowerCase {@code true} for uncased models (lower casing and accent
+ * stripping during tokenization), {@code false} for cased models. A
lower case setting in
+ * {@code inferenceOptions} takes precedence.
+ * @param pooling How token vectors are pooled. Not used for a model with a
+ * {@code sentence_embedding} output. Must not be {@code null}.
+ * @param normalize {@code true} to scale every vector to unit length.
+ * @param maxLength The maximum number of tokens per input, {@code [CLS]}
and {@code [SEP]}
+ * included; longer input is truncated. Must be at least {@code 2}.
+ * @param inferenceOptions The {@link InferenceOptions}, of which the GPU
settings and the lower
+ * case setting are used. Must not be {@code null}.
+ *
+ * @throws IllegalArgumentException Thrown if {@code pooling} or {@code
inferenceOptions} is
+ * {@code null}, if {@code inferenceOptions} holds invalid split options
or requests a GPU
+ * with a negative device id, if {@code maxLength} is less than {@code
2}, or if the model
+ * has no output of shape {@code [batch, hidden]} or {@code [batch,
tokens, hidden]}.
+ * @throws OrtException Thrown if the {@code model} cannot be loaded, or if
a GPU was requested
+ * and ONNX Runtime cannot provide it. ONNX Runtime does not fall back
to the CPU.
+ * @throws IOException Thrown if errors occurred loading the {@code model}
or {@code vocabulary}.
+ * @throws opennlp.tools.util.InvalidFormatException Thrown if a JSON {@code
vocabulary}
+ * is malformed, has an unsupported layout, or sets a lower casing that
disagrees with
+ * the lower casing the component is configured with.
+ *
+ * @since 3.0.0
+ */
+ public SentenceVectorsDL(final File model, final File vocabulary, final
boolean lowerCase,
+ final Pooling pooling, final boolean normalize, final int maxLength,
+ final InferenceOptions inferenceOptions)
+ throws OrtException, IOException {
+
+ super(model, vocabulary, sessionOptions(inferenceOptions),
+ resolveLowerCase(inferenceOptions, lowerCase));
try {
ParamChecks.requireNonNullArg(pooling, "pooling");
if (maxLength < MIN_LENGTH) {
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/AbstractDLColdStartCudaTest.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/AbstractDLColdStartCudaTest.java
new file mode 100644
index 000000000..6a23c2d83
--- /dev/null
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/AbstractDLColdStartCudaTest.java
@@ -0,0 +1,139 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import ai.onnxruntime.OrtException;
+import ai.onnxruntime.OrtSession;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import static opennlp.dl.OnnxTestSupport.gpu;
+import static opennlp.dl.OnnxTestSupport.hasCudaProvider;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.fail;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Tests that {@link AbstractDL#sessionOptions(InferenceOptions)} can request
the CUDA execution
+ * provider as the first ONNX Runtime call of a JVM. Without a usable CUDA
installation the
+ * request may be rejected, but never because ONNX Runtime's default logger is
missing.
+ *
+ * <p>Whether ONNX Runtime has been used is process-wide state that the other
tests of the module
+ * change, so the request runs in a child JVM through {@link
FirstCudaRequest}. The test runs
+ * where ONNX Runtime has the CUDA provider, which is {@code opennlp-dl-gpu};
a CUDA device is not
+ * needed.</p>
+ */
+class AbstractDLColdStartCudaTest {
+
+ private static final String ADDED = "ADDED";
+
+ private static final String REJECTED = "REJECTED ";
+
+ private static final String DEFAULT_LOGGER_FAILURE = "DefaultLogger";
+
+ private static final long TIMEOUT_MINUTES = 2;
+
+ /**
+ * Runs {@link FirstCudaRequest} in a child JVM and checks that its outcome
line reports the
+ * provider as added, or a rejection that is not about the default logger.
+ */
+ @Test
+ void testFirstCudaRequestOfAJvm(@TempDir final Path dir)
+ throws IOException, InterruptedException {
+ assumeTrue(hasCudaProvider(), "the ONNX Runtime on the classpath has no
CUDA execution provider");
+ final Path out = dir.resolve("stdout.log");
+ final Path err = dir.resolve("stderr.log");
+ final Path java = Path.of(System.getProperty("java.home"), "bin", "java");
+ final Process process = new ProcessBuilder(List.of(java.toString(), "-cp",
+ System.getProperty("java.class.path"),
FirstCudaRequest.class.getName()))
+ .redirectOutput(out.toFile())
+ .redirectError(err.toFile())
+ .start();
+ if (!process.waitFor(TIMEOUT_MINUTES, TimeUnit.MINUTES)) {
+ process.destroyForcibly();
+ fail("the child JVM did not finish within " + TIMEOUT_MINUTES + "
minutes");
+ }
+ // The outcome line is read from stdout only: ONNX Runtime logs to stderr,
on Windows in UTF-16.
+ final String stdout = read(out);
+ final String report = "stdout:\n" + stdout + "\nstderr:\n" + read(err);
+ assertEquals(0, process.exitValue(), report);
+ final String outcome = outcomeLine(stdout, report);
+ assertTrue(outcome.equals(ADDED) ||
!outcome.contains(DEFAULT_LOGGER_FAILURE), report);
+ }
+
+ /**
+ * Reads a file of child JVM output, replacing bytes that are not UTF-8.
+ *
+ * @param file The file to read.
+ * @return The content of the file.
+ * @throws IOException Thrown if the file cannot be read.
+ */
+ private static String read(final Path file) throws IOException {
+ return new String(Files.readAllBytes(file), StandardCharsets.UTF_8);
+ }
+
+ /**
+ * Finds the outcome line that {@link FirstCudaRequest} printed.
+ *
+ * @param stdout The standard output of the child JVM.
+ * @param report The complete child JVM output, for the failure message.
+ * @return The line that is {@link #ADDED} or starts with {@link #REJECTED}.
+ */
+ private static String outcomeLine(final String stdout, final String report) {
+ return stdout.lines()
+ .filter(line -> line.equals(ADDED) || line.startsWith(REJECTED))
+ .findFirst()
+ .orElseGet(() -> fail("no outcome line in the child JVM output:\n" +
report));
+ }
+
+ /**
+ * The child JVM entry point: requests the CUDA execution provider for
device {@code 0} as the
+ * first ONNX Runtime call, then prints {@link #ADDED} or {@link #REJECTED}
followed by the
+ * ONNX Runtime error on one line.
+ */
+ public static final class FirstCudaRequest {
+
+ private FirstCudaRequest() {
+ }
+
+ /**
+ * Entry point of the child JVM that the cold start test starts, because
the CUDA request has
+ * to be the first ONNX Runtime call of a fresh JVM.
+ *
+ * @param args Ignored; the test starts the child JVM without arguments.
+ */
+ public static void main(final String[] args) {
+ try {
+ final OrtSession.SessionOptions sessionOptions =
AbstractDL.sessionOptions(gpu(0));
+ sessionOptions.close();
+ System.out.println(ADDED);
+ } catch (OrtException e) {
+ System.out.println(REJECTED + e.getCode() + ": "
+ + e.getMessage().replace('\n', ' ').replace('\r', ' '));
+ }
+ }
+ }
+}
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/GpuRequestTest.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/GpuRequestTest.java
new file mode 100644
index 000000000..833b6b592
--- /dev/null
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/GpuRequestTest.java
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Map;
+
+import ai.onnxruntime.OrtException;
+import ai.onnxruntime.OrtSession;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.dl.doccat.DocumentCategorizerDL;
+import opennlp.dl.doccat.scoring.AverageClassificationScoringStrategy;
+import opennlp.dl.namefinder.NameFinderDL;
+import opennlp.tools.sentdetect.NewlineSentenceDetector;
+
+import static opennlp.dl.OnnxTestSupport.UNUSABLE_DEVICE_ID;
+import static opennlp.dl.OnnxTestSupport.copy;
+import static opennlp.dl.OnnxTestSupport.gpu;
+import static opennlp.dl.vectors.TinyVectorModels.TINY_VECTORS;
+import static opennlp.dl.vectors.TinyVectorModels.model;
+import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Tests how a GPU request in {@link InferenceOptions} reaches ONNX Runtime: a
request for a
+ * device that is not usable is rejected with an {@link OrtException}, never
served by the CPU,
+ * and a CPU request does not ask for any device.
+ *
+ * <p>In {@code opennlp-dl}, whose ONNX Runtime has no CUDA support, every GPU
request is
+ * rejected, so the tests there check that a request never falls back to the
CPU. The rejection
+ * of the device id itself is checked when {@code opennlp-dl-gpu} runs these
tests.</p>
+ */
+class GpuRequestTest {
+
+ private static final String PLAIN_VOCAB = "/opennlp/dl/vocab-plain.txt";
+ private static final String VOCAB_FILE = "vocab.txt";
+
+ /** The device id is not read while {@code gpu} is off, so no provider is
requested. */
+ @ParameterizedTest
+ @ValueSource(ints = {-1, 0, UNUSABLE_DEVICE_ID})
+ void testSessionOptionsWithoutGpuIgnoreTheDeviceId(final int deviceId) {
+ final InferenceOptions options = new InferenceOptions();
+ options.setGpuDeviceId(deviceId);
+ assertDoesNotThrow(() -> {
+ try (OrtSession.SessionOptions sessionOptions =
AbstractDL.sessionOptions(options)) {
+ assertNotNull(sessionOptions);
+ }
+ });
+ }
+
+ /** A name finder on a device id that no machine has fails instead of
running on the CPU. */
+ @Test
+ void testNameFinderRejectsAnUnusableDevice(@TempDir final Path dir) throws
IOException {
+ final File model = model(dir, TINY_VECTORS);
+ final File vocab = copy(dir, PLAIN_VOCAB, VOCAB_FILE);
+ final InferenceOptions options = gpu(UNUSABLE_DEVICE_ID);
+ final Map<Integer, String> labels = Map.of(0, "O", 1, "B-PER");
+ assertThrows(OrtException.class, () -> new NameFinderDL(model, vocab,
labels, options,
+ new NewlineSentenceDetector()).close());
+ }
+
+ /** A categorizer on a device id that no machine has fails instead of
running on the CPU. */
+ @Test
+ void testDocumentCategorizerRejectsAnUnusableDevice(@TempDir final Path dir)
+ throws IOException {
+ final File model = model(dir, TINY_VECTORS);
+ final File vocab = copy(dir, PLAIN_VOCAB, VOCAB_FILE);
+ final InferenceOptions options = gpu(UNUSABLE_DEVICE_ID);
+ final Map<Integer, String> categories = Map.of(0, "a", 1, "b");
+ assertThrows(OrtException.class, () -> new DocumentCategorizerDL(model,
vocab, categories,
+ new AverageClassificationScoringStrategy(), options).close());
+ }
+
+ /** Null options are rejected before a device is requested. */
+ @Test
+ void testSessionOptionsRejectNullOptions() {
+ assertThrows(IllegalArgumentException.class, () ->
AbstractDL.sessionOptions(null));
+ }
+
+ /**
+ * Invalid split sizes are rejected with an {@link IllegalArgumentException}
before a device is
+ * requested, so the unusable device id never reaches ONNX Runtime.
+ */
+ @ParameterizedTest
+ @CsvSource({"0,0", "-1,0", "10,-1", "10,10", "10,11"})
+ void testSessionOptionsRejectInvalidSplitBeforeAnyDevice(final int splitSize,
+ final int overlap) {
+ final InferenceOptions options = gpu(UNUSABLE_DEVICE_ID);
+ options.setDocumentSplitSize(splitSize);
+ options.setSplitOverlapSize(overlap);
+ assertThrows(IllegalArgumentException.class, () ->
AbstractDL.sessionOptions(options));
+ }
+}
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
index 4428469ae..8ee1c9711 100644
---
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/LoadVocabTest.java
@@ -19,13 +19,10 @@ package opennlp.dl;
import java.io.File;
import java.io.IOException;
-import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
-import java.nio.file.StandardCopyOption;
import java.util.List;
import java.util.Map;
-import java.util.Objects;
import java.util.stream.Stream;
import org.junit.jupiter.api.Test;
@@ -83,12 +80,7 @@ public class LoadVocabTest {
private Path tempDir;
private File getResource(String name) throws IOException {
- try (InputStream is = Objects.requireNonNull(
- getClass().getResourceAsStream("/opennlp/dl/" + name))) {
- final Path file = tempDir.resolve(name);
- Files.copy(is, file, StandardCopyOption.REPLACE_EXISTING);
- return file.toFile();
- }
+ return OnnxTestSupport.copy(tempDir, "/opennlp/dl/" + name, name);
}
/**
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/OnnxTestSupport.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/OnnxTestSupport.java
new file mode 100644
index 000000000..ebdcbb80a
--- /dev/null
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/OnnxTestSupport.java
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.dl;
+
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import java.util.Objects;
+
+import ai.onnxruntime.OrtEnvironment;
+import ai.onnxruntime.OrtProvider;
+
+/**
+ * Test helpers for the ONNX components of this module: test resources as
files, and the GPU
+ * settings of {@link InferenceOptions}.
+ */
+public final class OnnxTestSupport {
+
+ /** A CUDA device id that no machine has. */
+ public static final int UNUSABLE_DEVICE_ID = 99;
+
+ private OnnxTestSupport() {
+ }
+
+ /**
+ * Copies a classpath resource into a directory. The resource is copied
rather than resolved in
+ * place, because inside the {@code opennlp-dl} test-jar, as used by {@code
opennlp-dl-gpu}, it
+ * is not a file.
+ *
+ * @param dir The directory to copy into.
+ * @param resource The absolute classpath name of the resource.
+ * @param name The file name of the copy.
+ * @return The copied file.
+ * @throws IOException Thrown if the resource cannot be copied.
+ */
+ public static File copy(final Path dir, final String resource, final String
name)
+ throws IOException {
+ final Path file = dir.resolve(name);
+ try (InputStream is = Objects.requireNonNull(
+ OnnxTestSupport.class.getResourceAsStream(resource), resource)) {
+ Files.copy(is, file, StandardCopyOption.REPLACE_EXISTING);
+ }
+ return file.toFile();
+ }
+
+ /**
+ * {@return inference options that request the GPU with the given device id}
+ *
+ * @param deviceId The CUDA device id.
+ */
+ public static InferenceOptions gpu(final int deviceId) {
+ final InferenceOptions options = new InferenceOptions();
+ options.setGpu(true);
+ options.setGpuDeviceId(deviceId);
+ return options;
+ }
+
+ /**
+ * {@return whether the ONNX Runtime on the classpath is built with the CUDA
execution provider}
+ * This says nothing about a usable CUDA device.
+ */
+ public static boolean hasCudaProvider() {
+ return OrtEnvironment.getAvailableProviders().contains(OrtProvider.CUDA);
+ }
+}
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java
index 8351f8930..01d2fe470 100644
---
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLEmbedderTest.java
@@ -19,14 +19,11 @@ package opennlp.dl.vectors;
import java.io.File;
import java.io.IOException;
-import java.io.InputStream;
import java.nio.file.Files;
import java.nio.file.Path;
-import java.nio.file.StandardCopyOption;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
-import java.util.Objects;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
@@ -42,6 +39,12 @@ import static
opennlp.dl.vectors.OnnxTextEmbedderProvider.MAX_LENGTH_OPTION;
import static opennlp.dl.vectors.OnnxTextEmbedderProvider.NORMALIZE_OPTION;
import static opennlp.dl.vectors.OnnxTextEmbedderProvider.POOLING_OPTION;
import static opennlp.dl.vectors.OnnxTextEmbedderProvider.VOCABULARY_OPTION;
+import static opennlp.dl.vectors.TinyVectorModels.DELTA;
+import static opennlp.dl.vectors.TinyVectorModels.TINY_POOLED;
+import static opennlp.dl.vectors.TinyVectorModels.TINY_VECTORS;
+import static opennlp.dl.vectors.TinyVectorModels.model;
+import static opennlp.dl.vectors.TinyVectorModels.scale;
+import static opennlp.dl.vectors.TinyVectorModels.vocab;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotSame;
@@ -60,8 +63,6 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
*/
class SentenceVectorsDLEmbedderTest {
- private static final float DELTA = 1e-5f;
-
// W / |W|, the unit-length vector of every input of the tiny models.
private static final float[] UNIT_VECTOR = scale(1 / (float)
Math.sqrt(5.25));
@@ -74,34 +75,6 @@ class SentenceVectorsDLEmbedderTest {
private static final float[] CLS_VECTOR = scale(7);
- private static float[] scale(final float factor) {
- return new float[] {0.5f * factor, -1f * factor, 2f * factor};
- }
-
- // Copy the model out of the classpath rather than resolving it in place:
when this test runs
- // from the opennlp-dl test-jar (as it does in opennlp-dl-gpu) the resource
URI is inside a jar
- // and is not hierarchical, so new File(uri) would fail.
- private static File model(final Path dir, final String name) throws
IOException {
- final Path file = dir.resolve(name);
- try (InputStream is =
Objects.requireNonNull(SentenceVectorsDLEmbedderTest.class
- .getResourceAsStream("/opennlp/dl/vectors/" + name))) {
- Files.copy(is, file, StandardCopyOption.REPLACE_EXISTING);
- }
- return file.toFile();
- }
-
- private static File model(final Path dir) throws IOException {
- return model(dir, "tiny-vectors.onnx");
- }
-
- private static File vocab(final Path dir) throws IOException {
- final Path file = dir.resolve("vocab.txt");
- // Line number = id: [UNK]=2, [SEP]=3, hello=4, world=5, [CLS]=7.
- Files.write(file, List.of("[PAD]", "unused1", "[UNK]", "[SEP]", "hello",
"world",
- "unused2", "[CLS]"));
- return file.toFile();
- }
-
// The vocabulary above as a tokenizer.json of an uncased WordPiece model.
private static File tokenizerJson(final Path dir, final boolean lowercase)
throws IOException {
final Path file = dir.resolve("tokenizer.json");
@@ -114,12 +87,12 @@ class SentenceVectorsDLEmbedderTest {
private static SentenceVectorsDL meanVectors(final Path dir, final int
maxLength)
throws Exception {
- return new SentenceVectorsDL(model(dir), vocab(dir), true, Pooling.MEAN,
false, maxLength);
+ return new SentenceVectorsDL(model(dir, TINY_VECTORS), vocab(dir), true,
Pooling.MEAN, false, maxLength);
}
@Test
void testSelectedProviderPreservesInference(@TempDir final Path dir) throws
Exception {
- final Path graph = model(dir).toPath();
+ final Path graph = model(dir, TINY_VECTORS).toPath();
vocab(dir);
final Map<String, String> options = Map.of(VOCABULARY_OPTION, "vocab.txt");
final ProviderSpec spec = ProviderSpec.of(graph, options);
@@ -157,7 +130,7 @@ class SentenceVectorsDLEmbedderTest {
@Test
void testEmbedderContractOverARealSession(@TempDir final Path dir) throws
Exception {
- try (SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir),
vocab(dir))) {
+ try (SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir,
TINY_VECTORS), vocab(dir))) {
// The primary entry point, against which the adapter below is compared.
assertArrayEquals(UNIT_VECTOR, vectors.getVectors("hello world"), DELTA);
@@ -192,7 +165,7 @@ class SentenceVectorsDLEmbedderTest {
@Test
void testPooling(@TempDir final Path dir) throws Exception {
try (SentenceVectorsDL mean = meanVectors(dir,
SentenceVectorsDL.DEFAULT_MAX_LENGTH);
- SentenceVectorsDL cls = new SentenceVectorsDL(model(dir), vocab(dir),
true,
+ SentenceVectorsDL cls = new SentenceVectorsDL(model(dir,
TINY_VECTORS), vocab(dir), true,
Pooling.CLS, false, SentenceVectorsDL.DEFAULT_MAX_LENGTH)) {
assertArrayEquals(HELLO_WORLD_MEAN, mean.embed("hello world"), DELTA);
assertArrayEquals(HELLO_MEAN, mean.embed("hello"), DELTA);
@@ -213,26 +186,26 @@ class SentenceVectorsDLEmbedderTest {
assertArrayEquals(three.embed("hello"), three.embedAll(List.of("hello
world"))[0]);
}
assertThrows(IllegalArgumentException.class, () -> meanVectors(dir, 1));
- assertThrows(IllegalArgumentException.class, () -> new
SentenceVectorsDL(model(dir),
+ assertThrows(IllegalArgumentException.class, () -> new
SentenceVectorsDL(model(dir, TINY_VECTORS),
vocab(dir), true, null, false, SentenceVectorsDL.DEFAULT_MAX_LENGTH));
}
@Test
void testTokenizerJsonVocabularyMustAgreeWithLowerCase(@TempDir final Path
dir)
throws Exception {
- try (SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir),
tokenizerJson(dir, true),
+ try (SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir,
TINY_VECTORS), tokenizerJson(dir, true),
true, Pooling.MEAN, false, SentenceVectorsDL.DEFAULT_MAX_LENGTH))
{
assertArrayEquals(HELLO_WORLD_MEAN, vectors.embed("Hello World"), DELTA);
}
final InvalidFormatException e = assertThrows(InvalidFormatException.class,
- () -> new SentenceVectorsDL(model(dir), tokenizerJson(dir, false),
true));
+ () -> new SentenceVectorsDL(model(dir, TINY_VECTORS),
tokenizerJson(dir, false), true));
assertTrue(e.getMessage().contains("tokenizer.json"), e.getMessage());
assertTrue(e.getMessage().contains("normalizer.lowercase"),
e.getMessage());
}
@Test
void testPooledOutputIsSelectedByName(@TempDir final Path dir) throws
Exception {
- final File pooledModel = model(dir, "tiny-pooled.onnx");
+ final File pooledModel = model(dir, TINY_POOLED);
try (SentenceVectorsDL raw = new SentenceVectorsDL(pooledModel,
vocab(dir), true,
Pooling.CLS, false, SentenceVectorsDL.DEFAULT_MAX_LENGTH);
SentenceVectorsDL unit = new SentenceVectorsDL(pooledModel,
vocab(dir))) {
@@ -247,7 +220,7 @@ class SentenceVectorsDLEmbedderTest {
@Test
void testEmbedAfterCloseThrows(@TempDir final Path dir) throws Exception {
- final SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir),
vocab(dir));
+ final SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir,
TINY_VECTORS), vocab(dir));
vectors.close();
vectors.close();
assertThrows(IllegalStateException.class, () -> vectors.embed("hello"));
@@ -280,7 +253,7 @@ class SentenceVectorsDLEmbedderTest {
*/
@Test
void testEmbedAllEdges(@TempDir final Path dir) throws Exception {
- try (SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir),
vocab(dir))) {
+ try (SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir,
TINY_VECTORS), vocab(dir))) {
assertEquals(0, vectors.embedAll(List.of()).length);
assertEquals("texts[1] must not be null",
assertThrows(IllegalArgumentException.class,
() -> vectors.embedAll(Arrays.asList("hello", null))).getMessage());
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLInferenceOptionsTest.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLInferenceOptionsTest.java
new file mode 100644
index 000000000..da8cb3a70
--- /dev/null
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/SentenceVectorsDLInferenceOptionsTest.java
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.dl.vectors;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.stream.Stream;
+
+import ai.onnxruntime.OrtException;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
+import org.junit.jupiter.params.provider.ValueSource;
+
+import opennlp.dl.InferenceOptions;
+
+import static opennlp.dl.OnnxTestSupport.UNUSABLE_DEVICE_ID;
+import static opennlp.dl.OnnxTestSupport.gpu;
+import static opennlp.dl.vectors.TinyVectorModels.DELTA;
+import static opennlp.dl.vectors.TinyVectorModels.TINY_POOLED;
+import static opennlp.dl.vectors.TinyVectorModels.TINY_VECTORS;
+import static opennlp.dl.vectors.TinyVectorModels.gpuOrAbort;
+import static opennlp.dl.vectors.TinyVectorModels.model;
+import static opennlp.dl.vectors.TinyVectorModels.scale;
+import static opennlp.dl.vectors.TinyVectorModels.vocab;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+/**
+ * Tests the {@link SentenceVectorsDL} constructor that takes {@link
InferenceOptions}.
+ */
+class SentenceVectorsDLInferenceOptionsTest {
+
+ private static final String HELLO = "hello";
+
+ private static final String TEXT = "hello world";
+
+ private static final String LONGER_TEXT = "world hello world";
+
+ /**
+ * {@return the model, pooling and unit length combinations, one per path
after the inference}
+ */
+ static Stream<Arguments> gpuParityCases() {
+ return Stream.of(
+ Arguments.of(TINY_VECTORS, Pooling.MEAN, false),
+ Arguments.of(TINY_VECTORS, Pooling.MEAN, true),
+ Arguments.of(TINY_VECTORS, Pooling.CLS, false),
+ Arguments.of(TINY_VECTORS, Pooling.CLS, true),
+ Arguments.of(TINY_POOLED, Pooling.CLS, false),
+ Arguments.of(TINY_POOLED, Pooling.MEAN, true));
+ }
+
+ /**
+ * Checks that every path gives the same vectors on CUDA device {@code 0} as
on the CPU, where
+ * a CUDA device exists; skipped elsewhere.
+ */
+ @ParameterizedTest
+ @MethodSource("gpuParityCases")
+ void testGpuVectorsMatchTheCpu(final String modelName, final Pooling pooling,
+ final boolean normalize, @TempDir final Path dir) throws Exception {
+ final File model = model(dir, modelName);
+ final File vocab = vocab(dir);
+ try (SentenceVectorsDL gpu = gpuOrAbort(model, vocab, pooling, normalize);
+ SentenceVectorsDL cpu = new SentenceVectorsDL(model, vocab, true,
pooling, normalize,
+ SentenceVectorsDL.DEFAULT_MAX_LENGTH, new InferenceOptions())) {
+ assertEquals(cpu.dimension(), gpu.dimension());
+ assertArrayEquals(cpu.embed(TEXT), gpu.embed(TEXT), DELTA);
+ final List<String> texts = List.of(TEXT, LONGER_TEXT, "");
+ final float[][] cpuBatch = cpu.embedAll(texts);
+ final float[][] gpuBatch = gpu.embedAll(texts);
+ for (int i = 0; i < cpuBatch.length; i++) {
+ assertArrayEquals(cpuBatch[i], gpuBatch[i], DELTA, "row " + i);
+ }
+ }
+ }
+
+ /**
+ * Checks that a GPU request on a negative device id is rejected before ONNX
Runtime is asked
+ * for a device.
+ */
+ @ParameterizedTest
+ @ValueSource(ints = {-1, Integer.MIN_VALUE})
+ void testNegativeDeviceIdsAreRejected(final int deviceId, @TempDir final
Path dir)
+ throws IOException {
+ final File model = model(dir, TINY_VECTORS);
+ final File vocab = vocab(dir);
+ final InferenceOptions options = gpu(deviceId);
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> new SentenceVectorsDL(model, vocab, true, Pooling.MEAN, false,
+ SentenceVectorsDL.DEFAULT_MAX_LENGTH, options).close());
+ assertEquals("gpuDeviceId must not be negative", e.getMessage());
+ }
+
+ /**
+ * Checks that a GPU request on a device id that no machine has is rejected
by ONNX Runtime
+ * instead of running on the CPU.
+ */
+ @ParameterizedTest
+ @ValueSource(ints = {UNUSABLE_DEVICE_ID, Integer.MAX_VALUE})
+ void testUnusableDeviceIdsAreRejected(final int deviceId, @TempDir final
Path dir)
+ throws IOException {
+ final File model = model(dir, TINY_VECTORS);
+ final File vocab = vocab(dir);
+ final InferenceOptions options = gpu(deviceId);
+ assertThrows(OrtException.class, () -> new SentenceVectorsDL(model, vocab,
true, Pooling.MEAN,
+ false, SentenceVectorsDL.DEFAULT_MAX_LENGTH, options).close());
+ }
+
+ /**
+ * {@return the lower case in the options ({@code null} for unset), the
argument, and the
+ * expected lower casing}
+ */
+ static Stream<Arguments> lowerCaseCases() {
+ return Stream.of(
+ Arguments.of(Boolean.TRUE, true, true),
+ Arguments.of(Boolean.TRUE, false, true),
+ Arguments.of(Boolean.FALSE, true, false),
+ Arguments.of(Boolean.FALSE, false, false),
+ Arguments.of(null, true, true),
+ Arguments.of(null, false, false));
+ }
+
+ /**
+ * Checks that the lower case setting is {@code options.getLowerCase()} when
it is set and the
+ * {@code lowerCase} argument when it is not. "HELLO" encodes as {@code
[UNK]} (id 2) unless
+ * lower cased to {@code hello} (id 4), so the mean of the ids is {@code 4}
or {@code 14 / 3}.
+ */
+ @ParameterizedTest
+ @MethodSource("lowerCaseCases")
+ void testLowerCasePrecedence(final Boolean inOptions, final boolean argument,
+ final boolean expectedLowerCase, @TempDir final Path dir) throws
Exception {
+ final InferenceOptions options = new InferenceOptions();
+ if (inOptions != null) {
+ options.setLowerCase(inOptions);
+ }
+ try (SentenceVectorsDL vectors = new SentenceVectorsDL(model(dir,
TINY_VECTORS), vocab(dir),
+ argument, Pooling.MEAN, false, SentenceVectorsDL.DEFAULT_MAX_LENGTH,
options)) {
+ assertArrayEquals(scale(expectedLowerCase ? 14 / 3f : 4f),
vectors.embed("HELLO"), DELTA);
+ }
+ }
+
+ /**
+ * Checks that default {@link InferenceOptions} give the same vectors as the
constructor
+ * without options.
+ */
+ @Test
+ void testDefaultOptionsMatchTheDefaultSession(@TempDir final Path dir)
throws Exception {
+ final File model = model(dir, TINY_VECTORS);
+ final File vocab = vocab(dir);
+ try (SentenceVectorsDL plain = new SentenceVectorsDL(model, vocab, true,
Pooling.MEAN, false,
+ SentenceVectorsDL.DEFAULT_MAX_LENGTH);
+ SentenceVectorsDL withOptions = new SentenceVectorsDL(model, vocab,
true, Pooling.MEAN,
+ false, SentenceVectorsDL.DEFAULT_MAX_LENGTH, new
InferenceOptions())) {
+ assertArrayEquals(plain.embed(TEXT), withOptions.embed(TEXT), DELTA);
+ }
+ }
+
+ /**
+ * Checks that {@code null} options are rejected with the parameter name.
+ */
+ @Test
+ void testNullOptionsAreRejected(@TempDir final Path dir) throws IOException {
+ final File model = model(dir, TINY_VECTORS);
+ final File vocab = vocab(dir);
+ final IllegalArgumentException e =
assertThrows(IllegalArgumentException.class,
+ () -> new SentenceVectorsDL(model, vocab, true, Pooling.MEAN, false,
+ SentenceVectorsDL.DEFAULT_MAX_LENGTH, null).close());
+ assertEquals("inferenceOptions must not be null", e.getMessage());
+ }
+}
diff --git
a/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/TinyVectorModels.java
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/TinyVectorModels.java
new file mode 100644
index 000000000..d48d4ae16
--- /dev/null
+++
b/opennlp-core/opennlp-ml/opennlp-dl/src/test/java/opennlp/dl/vectors/TinyVectorModels.java
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.dl.vectors;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+
+import ai.onnxruntime.OrtException;
+
+import static opennlp.dl.OnnxTestSupport.copy;
+import static opennlp.dl.OnnxTestSupport.gpu;
+import static opennlp.dl.OnnxTestSupport.hasCudaProvider;
+import static org.junit.jupiter.api.Assumptions.abort;
+import static org.junit.jupiter.api.Assumptions.assumeTrue;
+
+/**
+ * Test fixtures for the bundled tiny ONNX models. {@code tiny-vectors.onnx}
(see
+ * {@code gen_tiny_vectors_model.py}) computes {@code output[b][t] =
float(input_ids[b][t]) * W}
+ * with {@code W = [0.5, -1, 2]}; {@code tiny-pooled.onnx} (see {@code
gen_tiny_pooled_model.py})
+ * adds a {@code sentence_embedding} output holding the sum of the ids times
{@code W}.
+ */
+public final class TinyVectorModels {
+
+ /** The model with a token output only. */
+ public static final String TINY_VECTORS = "tiny-vectors.onnx";
+
+ /** The model with an additional {@code sentence_embedding} output. */
+ public static final String TINY_POOLED = "tiny-pooled.onnx";
+
+ /** The tolerance for comparing vectors of the tiny models. */
+ public static final float DELTA = 1e-5f;
+
+ private static final String VECTORS_RESOURCES = "/opennlp/dl/vectors/";
+
+ private TinyVectorModels() {
+ }
+
+ /**
+ * Copies a tiny model into a directory.
+ *
+ * @param dir The directory to copy into.
+ * @param name {@link #TINY_VECTORS} or {@link #TINY_POOLED}.
+ * @return The model file.
+ * @throws IOException Thrown if the model cannot be copied.
+ */
+ public static File model(final Path dir, final String name) throws
IOException {
+ return copy(dir, VECTORS_RESOURCES + name, name);
+ }
+
+ /**
+ * Writes the vocabulary of the tiny models, where the line number is the id:
+ * {@code [UNK]=2, [SEP]=3, hello=4, world=5, [CLS]=7}.
+ *
+ * @param dir The directory to write into.
+ * @return The vocabulary file.
+ * @throws IOException Thrown if the vocabulary cannot be written.
+ */
+ public static File vocab(final Path dir) throws IOException {
+ final Path file = dir.resolve("vocab.txt");
+ Files.write(file, List.of("[PAD]", "unused1", "[UNK]", "[SEP]", "hello",
"world",
+ "unused2", "[CLS]"));
+ return file.toFile();
+ }
+
+ /**
+ * {@return {@code W} times {@code factor}, the expected vector for a mean
or summed id}
+ *
+ * @param factor The mean or summed token id.
+ */
+ public static float[] scale(final float factor) {
+ return new float[] {0.5f * factor, -1f * factor, 2f * factor};
+ }
+
+ /**
+ * Opens a session on CUDA device {@code 0}, or aborts the calling test
where the ONNX Runtime
+ * on the classpath has no CUDA execution provider or the device is not
usable.
+ *
+ * @param model The model file.
+ * @param vocab The vocabulary file.
+ * @param pooling The pooling.
+ * @param normalize {@code true} to scale every vector to unit length.
+ * @return The session on the GPU.
+ * @throws IOException Thrown if the model or vocabulary cannot be read.
+ */
+ public static SentenceVectorsDL gpuOrAbort(final File model, final File
vocab,
+ final Pooling pooling, final boolean normalize) throws IOException {
+ assumeTrue(hasCudaProvider(), "the ONNX Runtime on the classpath has no
CUDA execution provider");
+ try {
+ return new SentenceVectorsDL(model, vocab, true, pooling, normalize,
+ SentenceVectorsDL.DEFAULT_MAX_LENGTH, gpu(0));
+ } catch (OrtException e) {
+ return abort("no usable CUDA device 0: " + e.getMessage());
+ }
+ }
+}
diff --git a/opennlp-docs/src/docbkx/machine-learning.xml
b/opennlp-docs/src/docbkx/machine-learning.xml
index a1d5a8660..69c3f72d3 100644
--- a/opennlp-docs/src/docbkx/machine-learning.xml
+++ b/opennlp-docs/src/docbkx/machine-learning.xml
@@ -183,5 +183,26 @@ try (TextEmbedder embedder =
providers.select(spec).create(spec)) {
has no provider until another one is installed.
Existing <code>SentenceVectorsDL</code>
constructors remain available.
</para>
+ <para>
+ To compute the vectors on a GPU, depend on
<code>opennlp-dl-gpu</code> in place of
+ <code>opennlp-dl</code> and pass
<code>InferenceOptions</code> to
+ <code>SentenceVectorsDL</code>.
<code>opennlp-dl-gpu</code> supports NVIDIA GPUs on
+ linux-x64 and win-x64 only. On macOS, use
<code>opennlp-dl</code>. The machine needs
+ the CUDA and cuDNN versions that the ONNX Runtime
release requires, see the
+ <ulink
url="https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements";>CUDA
+ execution provider requirements</ulink>. The
<code>onnx</code> provider spec has no device
+ option, so an embedder created through it runs on the
CPU. If CUDA or the configured
+ device is not available, the constructor throws an
<code>OrtException</code>; ONNX Runtime
+ does not fall back to the CPU. A negative device id is
rejected with an
+ <code>IllegalArgumentException</code>. A lower case
setting in the options takes
+ precedence over the <code>lowerCase</code> argument.
+ </para>
+ <programlisting language="java"><![CDATA[InferenceOptions
options = new InferenceOptions();
+options.setGpu(true);
+options.setGpuDeviceId(0);
+try (SentenceVectorsDL vectors = new SentenceVectorsDL(model, vocab, true,
Pooling.MEAN, true,
+ SentenceVectorsDL.DEFAULT_MAX_LENGTH, options)) {
+ float[] vector = vectors.embed("first sentence");
+}]]></programlisting>
</section>
</chapter>
\ No newline at end of file
