This is an automated email from the ASF dual-hosted git repository.
hansva pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new 3119f36462 Embed text: Unicode spaces are sent to the model, an
unresolved batch… (#9371)
3119f36462 is described below
commit 3119f36462bf39784340501758061c131aee3aea
Author: Bart Maertens <[email protected]>
AuthorDate: Thu Nov 12 10:49:39 2026 +0100
Embed text: Unicode spaces are sent to the model, an unresolved batch…
(#9371)
* Embed text: Unicode spaces are sent to the model, an unresolved batch
size falls back to 16, and a provider failure says only
java.net.ConnectException, fixes #9287
* Embed text: address review comments, mask secrets in failures, retry only
what may pass, #9287
---
.../ROOT/pages/pipeline/transforms/embedtext.adoc | 6 +-
integration-tests/ai/0004-embed-blank-text.hpl | 17 +-
integration-tests/ai/README.md | 2 +-
.../apache/hop/ai/engine/AiEmbeddingFactory.java | 78 ++++++-
.../hop/ai/transforms/embedtext/EmbedText.java | 222 ++++++++++++++++++--
.../hop/ai/transforms/embedtext/EmbedTextData.java | 10 +
.../hop/ai/transforms/embedtext/EmbedTextMeta.java | 55 ++++-
.../embedtext/messages/messages_en_US.properties | 5 +-
.../hop/ai/engine/AiEmbeddingFactoryTest.java | 52 +++++
.../ai/transforms/embedtext/EmbedTextMetaTest.java | 40 ++++
.../hop/ai/transforms/embedtext/EmbedTextTest.java | 233 ++++++++++++++++++++-
11 files changed, 673 insertions(+), 47 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/embedtext.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/embedtext.adoc
index 0244d2af66..a07639ca2b 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/embedtext.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/embedtext.adoc
@@ -55,10 +55,10 @@ Vector is the better choice when the Vector value type
plugin is installed. When
|Transform name|Name of the transform, unique within the pipeline.
|AI provider|The AI provider serving the embedding model.
|Model|Optional. Overrides the provider's EMBEDDING model for this transform
only. Accepts variables.
-|Input field|Field holding the text to embed. A row whose text is empty or
only whitespace is passed on with empty output fields rather than being
dropped, and keeps its place in the stream.
+|Input field|Field holding the text to embed. A row whose text is empty or
only whitespace, including no-break and zero-width spaces, is passed on with
empty output fields rather than being dropped, and keeps its place in the
stream.
|Output field|Field the embedding is written to.
|Output format|`STRING` for a JSON array, `VECTOR` for a Vector field.
-|Batch size|Rows buffered before the provider is called. Accepts variables.
Rows with empty text are buffered so they keep their place in the stream, but
are not sent, so a batch can send fewer texts than its size.
+|Batch size|Rows buffered before the provider is called, 16 when empty.
Accepts variables; a value that is not a whole number, for example an unset
variable, stops the transform when the pipeline starts. Rows with empty text
are buffered so they keep their place in the stream, but are not sent, so a
batch can send fewer texts than its size.
|Include model metadata|Adds the model name and the vector width to each row.
|Model field|Output field for the embedding model name.
|Dimensions field|Output field for the number of dimensions in the vector.
@@ -72,4 +72,6 @@ Rows leave this transform in the order they arrived,
including the ones with no
That also shapes what happens on failure. The provider answers per batch and
cannot say which text it choked on, so when an error hop is attached every row
that was sent is diverted rather than one row. Rows with empty text were never
sent and are passed on normally. A batch size of 1 gives per-row precision at
the cost of throughput.
+A call that fails for a reason that may pass, such as no connection, a
timeout, a rate limit or a server error, is retried twice after a short pause.
Anything else, for example a bad API key or an unknown model, fails at once.
The error names the model, the AI provider, its endpoint and the underlying
cause, with anything that looks like a key or password masked. Stopping the
pipeline while it waits to retry drops the waiting batch without sending its
rows to the error hop.
+
**Include model metadata** is worth leaving on for anything that will be
re-indexed later. Knowing which model produced a stored vector is what lets you
re-embed only what a model change actually affects, and mixing vectors from two
models in one index produces silently poor results.
diff --git a/integration-tests/ai/0004-embed-blank-text.hpl
b/integration-tests/ai/0004-embed-blank-text.hpl
index faa5c9c899..e76c5e46f5 100644
--- a/integration-tests/ai/0004-embed-blank-text.hpl
+++ b/integration-tests/ai/0004-embed-blank-text.hpl
@@ -91,6 +91,21 @@ limitations under the License.
<item/>
<item>N</item>
</line>
+ <line>
+ <item>  </item>
+ <item/>
+ <item>N</item>
+ </line>
+ <line>
+ <item> </item>
+ <item/>
+ <item>N</item>
+ </line>
+ <line>
+ <item>​</item>
+ <item/>
+ <item>N</item>
+ </line>
<line>
<item>Embeddings turn text into vectors for retrieval.</item>
<item/>
@@ -354,7 +369,7 @@ limitations under the License.
<value>
<name>constant</name>
<type>Integer</type>
- <text>5</text>
+ <text>8</text>
<length>-1</length>
<precision>-1</precision>
<isnull>N</isnull>
diff --git a/integration-tests/ai/README.md b/integration-tests/ai/README.md
index eb6a0950e9..c00f32a007 100644
--- a/integration-tests/ai/README.md
+++ b/integration-tests/ai/README.md
@@ -42,6 +42,6 @@ standard suite. See `disabled.txt`.
| Test | Covers |
|---|---|
| `0001-embed-text` | The Embed text transform against `nomic-embed-text`: the
vector width, the model name on the row, and that an embedding comes back. The
vector values themselves vary per call, so they are not compared. |
-| `0004-embed-blank-text` | Embed text with empty and whitespace-only values
in the same batch as real text: those rows pass through without an embedding,
the others get one, and no row is lost. |
+| `0004-embed-blank-text` | Embed text with empty and whitespace-only values
(spaces, tab and newline, no-break spaces, a narrow no-break space, a
zero-width space) in the same batch as real text: those rows pass through
without an embedding, the others get one, and no row is lost. |
| `0005-thinking` | Language Model Chat against the thinking model
`qwen3:0.6b` with Thinking Off and On. Ollama counts reasoning tokens as
output, so Off must answer in under 64 output tokens and On must use more than
Off; if `think` did not reach Ollama both counts would be the same at
temperature 0. The answer with Off must not contain `<think>`. |
| `0006-structured-extract-thinking-off` | Structured extract with
`qwen3:0.6b` through the `ollama-thinking-off` AI provider: a thinking model
held to a JSON schema with Thinking Off must still return both required fields.
It cannot show that `think` was sent, since Structured extract reports no token
counts and the JSON comes back either way; `StructuredExtractThinkingTest`
checks the flag on this request. |
diff --git
a/plugins/tech/ai/src/main/java/org/apache/hop/ai/engine/AiEmbeddingFactory.java
b/plugins/tech/ai/src/main/java/org/apache/hop/ai/engine/AiEmbeddingFactory.java
index 31a41569c7..f7426fa9d2 100644
---
a/plugins/tech/ai/src/main/java/org/apache/hop/ai/engine/AiEmbeddingFactory.java
+++
b/plugins/tech/ai/src/main/java/org/apache/hop/ai/engine/AiEmbeddingFactory.java
@@ -58,22 +58,74 @@ public final class AiEmbeddingFactory {
IVariables variables,
IHopMetadataProvider metadataProvider)
throws HopException {
+ return resolveEmbeddingModel(providerName, modelName, variables,
metadataProvider, null);
+ }
+
+ /**
+ * As {@link #resolveEmbeddingModel(String, String, IVariables,
IHopMetadataProvider)}, with the
+ * number of times the model retries a failed call itself.
+ *
+ * <p>langchain4j logs a full stack trace at WARN for every retry, outside
of Hop's log level. A
+ * caller that retries on its own, so it can say what failed in one line,
passes 0.
+ *
+ * @param maxRetries the model's own retries, or null for langchain4j's
default
+ */
+ public static ResolvedEmbeddingModel resolveEmbeddingModel(
+ String providerName,
+ String modelName,
+ IVariables variables,
+ IHopMetadataProvider metadataProvider,
+ Integer maxRetries)
+ throws HopException {
AiProvider provider = AiProviderLoader.load(providerName,
metadataProvider);
String resolvedName =
Utils.isEmpty(modelName)
?
variables.resolve(provider.resolveModelName(AiModelRole.EMBEDDING))
: variables.resolve(modelName);
+ // of() rejects a null or half configured provider, so there is one guard,
not two.
+ AiProviderSettings settings = AiProviderSettings.of(provider, variables);
return new ResolvedEmbeddingModel(
- createEmbeddingModel(provider, resolvedName, variables), resolvedName);
+ createEmbeddingModel(provider, settings, resolvedName, variables,
maxRetries),
+ resolvedName,
+ provider.getName(),
+ effectiveBaseUrl(settings));
}
- /** An embedding model and the name of the model it talks to. */
- public record ResolvedEmbeddingModel(EmbeddingModel model, String modelName)
{}
+ /**
+ * An embedding model, the name of the model it talks to, and where it
lives: the provider name
+ * and the resolved base URL, which a failed call needs to say which
endpoint did not answer.
+ */
+ public record ResolvedEmbeddingModel(
+ EmbeddingModel model, String modelName, String providerName, String
baseUrl) {}
+
+ /**
+ * The endpoint langchain4j uses when an OpenAI compatible provider has no
base URL, as its
+ * OpenAiEmbeddingModel falls back to it.
+ */
+ static final String OPEN_AI_DEFAULT_BASE_URL = "https://api.openai.com/v1";
+
+ /** The URL the model actually talks to, so a failed call can name it even
when none is set. */
+ static String effectiveBaseUrl(AiProviderSettings settings) {
+ if (Utils.isEmpty(settings.baseUrl()) &&
"OPEN_AI".equals(settings.type())) {
+ return OPEN_AI_DEFAULT_BASE_URL;
+ }
+ return settings.baseUrl();
+ }
+ /** Builds the provider's embedding model, with langchain4j's default
retries. */
public static EmbeddingModel createEmbeddingModel(
AiProvider provider, String modelName, IVariables variables) throws
HopException {
- // of() rejects a null or half configured provider, so there is one guard,
not two.
- AiProviderSettings settings = AiProviderSettings.of(provider, variables);
+ return createEmbeddingModel(
+ provider, AiProviderSettings.of(provider, variables), modelName,
variables, null);
+ }
+
+ private static EmbeddingModel createEmbeddingModel(
+ AiProvider provider,
+ AiProviderSettings settings,
+ String modelName,
+ IVariables variables,
+ Integer maxRetries)
+ throws HopException {
// Resolution order: the transform's override, then the provider's
EMBEDDING row. The
// provider's plain modelName is the chat model and is deliberately not a
fallback here.
@@ -90,8 +142,8 @@ public final class AiEmbeddingFactory {
}
return switch (settings.type()) {
- case "OLLAMA" -> ollamaModel(settings, model);
- case "OPEN_AI" -> openAiModel(settings, model);
+ case "OLLAMA" -> ollamaModel(settings, model, maxRetries);
+ case "OPEN_AI" -> openAiModel(settings, model, maxRetries);
default ->
throw new HopException(
"Provider type '"
@@ -101,18 +153,26 @@ public final class AiEmbeddingFactory {
};
}
- private static EmbeddingModel ollamaModel(AiProviderSettings settings,
String model) {
+ private static EmbeddingModel ollamaModel(
+ AiProviderSettings settings, String model, Integer maxRetries) {
OllamaEmbeddingModel.OllamaEmbeddingModelBuilder builder =
OllamaEmbeddingModel.builder().baseUrl(settings.baseUrl()).modelName(model);
+ if (maxRetries != null) {
+ builder.maxRetries(maxRetries);
+ }
if (settings.timeout() != null) {
builder.timeout(settings.timeout());
}
return builder.build();
}
- private static EmbeddingModel openAiModel(AiProviderSettings settings,
String model) {
+ private static EmbeddingModel openAiModel(
+ AiProviderSettings settings, String model, Integer maxRetries) {
OpenAiEmbeddingModel.OpenAiEmbeddingModelBuilder builder =
OpenAiEmbeddingModel.builder().modelName(model);
+ if (maxRetries != null) {
+ builder.maxRetries(maxRetries);
+ }
if (!Utils.isEmpty(settings.baseUrl())) {
builder.baseUrl(settings.baseUrl());
}
diff --git
a/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedText.java
b/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedText.java
index ef7c3e1fc6..22dc0a3db0 100644
---
a/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedText.java
+++
b/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedText.java
@@ -18,8 +18,13 @@ package org.apache.hop.ai.transforms.embedtext;
import dev.langchain4j.data.embedding.Embedding;
import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.exception.NonRetriableException;
+import dev.langchain4j.exception.RetriableException;
+import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
+import java.util.concurrent.TimeoutException;
+import java.util.regex.Pattern;
import org.apache.hop.ai.engine.AiEmbeddingFactory;
import org.apache.hop.core.Const;
import org.apache.hop.core.exception.HopException;
@@ -36,6 +41,12 @@ public class EmbedText extends BaseTransform<EmbedTextMeta,
EmbedTextData> {
private static final Class<?> PKG = EmbedTextMeta.class;
+ /**
+ * Retries of a failed call, after the first attempt. The same as
langchain4j's default, which
+ * this transform switches off for its model so a retry is one log line
rather than a stack trace.
+ */
+ static final int MAX_RETRIES = 2;
+
public EmbedText(
TransformMeta transformMeta,
EmbedTextMeta meta,
@@ -52,11 +63,13 @@ public class EmbedText extends BaseTransform<EmbedTextMeta,
EmbedTextData> {
logError(BaseMessages.getString(PKG,
"EmbedText.Validation.ProviderRequired"));
return false;
}
- data.batchSize = Const.toInt(resolve(meta.getBatchSize()), 16);
- if (data.batchSize < 1) {
- logError(BaseMessages.getString(PKG,
"EmbedText.Validation.BatchSizePositive"));
+ // An unset variable stays as ${...}. Running with some other batch size
would hide that.
+ EmbedTextMeta.BatchSize batchSize =
EmbedTextMeta.parseBatchSize(resolve(meta.getBatchSize()));
+ if (batchSize.error() != null) {
+ logError(batchSize.error());
return false;
}
+ data.batchSize = batchSize.value();
data.emitVector = meta.getOutputFormat() == EmbedTextOutputFormat.VECTOR;
return super.init();
}
@@ -117,11 +130,15 @@ public class EmbedText extends
BaseTransform<EmbedTextMeta, EmbedTextData> {
List<Embedding> embeddings = List.of();
if (!segments.isEmpty()) {
try {
- embeddings = data.model.embedAll(segments).content();
+ embeddings = embedWithRetries(segments);
} catch (Exception e) {
+ if (isStopped()) {
+ // The user stopped the run, possibly while waiting to retry. These
rows did not fail.
+ return;
+ }
+ String failure = describeFailure(e);
if (!getTransformMeta().isDoingErrorHandling()) {
- throw new HopException(
- BaseMessages.getString(PKG, "EmbedText.Error.Embedding",
data.modelName), e);
+ throw new HopException(failure, e);
}
// Only the rows that were actually in the request failed. The others
carry no text, were
// never sent, and belong in the output stream.
@@ -130,12 +147,7 @@ public class EmbedText extends
BaseTransform<EmbedTextMeta, EmbedTextData> {
putRow(data.outputRowMeta, resize(rows.get(i)));
} else {
putError(
- data.inputRowMeta,
- rows.get(i),
- 1,
- e.getMessage(),
- meta.getInputField(),
- "EMBEDTEXT001");
+ data.inputRowMeta, rows.get(i), 1, failure,
meta.getInputField(), "EMBEDTEXT001");
}
}
return;
@@ -161,12 +173,186 @@ public class EmbedText extends
BaseTransform<EmbedTextMeta, EmbedTextData> {
}
/**
- * True for text that has no content to embed. langchain4j refuses a text
segment that trims to
- * nothing, which takes control characters as well as spaces, and would stop
the transform before
- * the provider is called. Unicode whitespace has nothing to embed either.
+ * Calls the provider, retrying a failure that may pass (no connection, a
timeout, a rate limit, a
+ * server error). Anything else, such as a bad API key, an unknown model or
an answer that cannot
+ * be read, fails at once: it would fail the same way again.
+ */
+ private List<Embedding> embedWithRetries(List<TextSegment> segments) {
+ long delay = data.retryDelayMillis;
+ for (int attempt = 1; ; attempt++) {
+ try {
+ return data.model.embedAll(segments).content();
+ } catch (RuntimeException e) {
+ if (attempt > MAX_RETRIES || !isRetriable(e) || isStopped()) {
+ throw e;
+ }
+ logBasic(
+ BaseMessages.getString(
+ PKG,
+ "EmbedText.Log.Retrying",
+ describeFailure(e),
+ String.valueOf(attempt),
+ String.valueOf(MAX_RETRIES)));
+ if (isDebug()) {
+ logDebug(Const.getStackTracker(e));
+ }
+ if (!pause(delay)) {
+ throw e;
+ }
+ delay = delay * 3 / 2;
+ }
+ }
+ }
+
+ /**
+ * True for a failure that may pass on another attempt: langchain4j's own
retriable ones (a
+ * timeout, a rate limit, a server error), and a connection or I/O failure,
which langchain4j
+ * passes on wrapped in a plain exception.
+ */
+ static boolean isRetriable(Throwable e) {
+ if (e instanceof NonRetriableException) {
+ return false;
+ }
+ for (Throwable t = e; t != null; t = t.getCause() == t ? null :
t.getCause()) {
+ if (t instanceof RetriableException
+ || t instanceof IOException
+ || t instanceof TimeoutException) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** Waits, but gives up as soon as the pipeline is stopped. False when the
wait was cut short. */
+ private boolean pause(long millis) {
+ long until = System.currentTimeMillis() + millis;
+ try {
+ while (!isStopped()) {
+ long left = until - System.currentTimeMillis();
+ if (left <= 0) {
+ return true;
+ }
+ Thread.sleep(Math.min(left, 100L));
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ }
+ return false;
+ }
+
+ /**
+ * One line saying which model, provider and endpoint failed, and why. The
exception of a failed
+ * connection carries no message of its own, so on its own it says nothing
about where it went.
+ */
+ String describeFailure(Throwable e) {
+ String endpoint = endpointForMessages(data.baseUrl);
+ return BaseMessages.getString(
+ PKG,
+ "EmbedText.Error.Embedding",
+ data.modelName,
+ data.providerName,
+ Utils.isEmpty(endpoint) ? "?" : endpoint,
+ rootCause(e));
+ }
+
+ /**
+ * The deepest cause, as its class name and its message when it has one. The
message is the
+ * provider's or the HTTP client's, so it can quote a key or a URL with one
in it, or be a whole
+ * HTML error page: it is masked and cut short, as it ends up in the log and
in error rows.
+ */
+ static String rootCause(Throwable e) {
+ Throwable root = e;
+ while (root.getCause() != null && root.getCause() != root) {
+ root = root.getCause();
+ }
+ String message = maskSecrets(root.getMessage());
+ String name = root.getClass().getSimpleName();
+ return Utils.isEmpty(message) ? name : name + ": " + message;
+ }
+
+ /** Longest cause message passed on, so it fits an error description column.
*/
+ static final int MAX_CAUSE_LENGTH = 300;
+
+ private static final Pattern URL_USER_INFO =
+ Pattern.compile("(?i)\\b([a-z][a-z0-9+.-]*://)[^\\s/?#]*@");
+ private static final Pattern SECRET_PARAMETER =
+ Pattern.compile(
+
"(?i)\\b((?:api[_-]?key|key|access[_-]?token|token|secret|password|pwd)\\s*[=:]\\s*)[^\\s&,;\"']+");
+ private static final Pattern BEARER =
Pattern.compile("(?i)\\b(bearer\\s+)[^\\s\"',;]+");
+ private static final Pattern API_KEY =
Pattern.compile("\\b(sk|pk|gsk|xai)-[A-Za-z0-9_*.-]{6,}");
+
+ /**
+ * Masks what looks like a credential: user info in a URL, a key, token or
password parameter, a
+ * bearer token or an OpenAI style {@code sk-...} key. Collapses white
space, so an HTML error
+ * page stays on one line, and cuts the result at {@link #MAX_CAUSE_LENGTH}.
+ */
+ static String maskSecrets(String message) {
+ if (Utils.isEmpty(message)) {
+ return message;
+ }
+ String masked = URL_USER_INFO.matcher(message).replaceAll("$1***@");
+ masked = SECRET_PARAMETER.matcher(masked).replaceAll("$1***");
+ masked = BEARER.matcher(masked).replaceAll("$1***");
+ masked = API_KEY.matcher(masked).replaceAll("$1-***");
+ masked = masked.replaceAll("\\s+", " ").trim();
+ if (masked.length() > MAX_CAUSE_LENGTH) {
+ masked = masked.substring(0, MAX_CAUSE_LENGTH) + "...";
+ }
+ return masked;
+ }
+
+ /**
+ * The base URL without user info, query or fragment, which is where a key
would sit if someone
+ * put one in the URL. Error rows can end up in a table, so they must not
carry it.
+ *
+ * <p>This is plain text work rather than {@link java.net.URI}: a host name
with an underscore,
+ * common for a Docker service, or a URL without a scheme parses without a
host, and then the
+ * parts that hold a key are not split out either.
+ */
+ static String endpointForMessages(String baseUrl) {
+ if (Utils.isEmpty(baseUrl)) {
+ return "";
+ }
+ String url = baseUrl.trim();
+ int end = indexOfAny(url, '?', '#');
+ if (end >= 0) {
+ url = url.substring(0, end);
+ }
+ int at = url.lastIndexOf('@');
+ if (at >= 0) {
+ int scheme = url.indexOf("://");
+ String prefix = scheme >= 0 && scheme < at ? url.substring(0, scheme +
3) : "";
+ url = prefix + url.substring(at + 1);
+ }
+ return url;
+ }
+
+ private static int indexOfAny(String text, char... characters) {
+ for (int i = 0; i < text.length(); i++) {
+ for (char c : characters) {
+ if (text.charAt(i) == c) {
+ return i;
+ }
+ }
+ }
+ return -1;
+ }
+
+ /**
+ * True for text that has no content to embed: only whitespace, Unicode
spaces (no-break, narrow
+ * no-break, ideographic, ...), control characters and invisible format
characters such as the
+ * zero-width space. langchain4j refuses a segment that trims to nothing,
which would stop the
+ * transform, and the others would be sent and come back as an embedding
that means nothing.
*/
static boolean hasNothingToEmbed(String text) {
- return text == null || text.trim().isEmpty() || text.isBlank();
+ return text == null || text.codePoints().allMatch(EmbedText::isInvisible);
+ }
+
+ private static boolean isInvisible(int codePoint) {
+ return Character.isWhitespace(codePoint)
+ || Character.isSpaceChar(codePoint)
+ || Character.isISOControl(codePoint)
+ || Character.getType(codePoint) == Character.FORMAT;
}
private void withEmbedding(Object[] output, float[] vector) {
@@ -232,9 +418,11 @@ public class EmbedText extends
BaseTransform<EmbedTextMeta, EmbedTextData> {
private void openModel() throws HopException {
AiEmbeddingFactory.ResolvedEmbeddingModel resolved =
AiEmbeddingFactory.resolveEmbeddingModel(
- resolve(meta.getAiProvider()), meta.getModelName(), this,
getMetadataProvider());
+ resolve(meta.getAiProvider()), meta.getModelName(), this,
getMetadataProvider(), 0);
data.model = resolved.model();
data.modelName = resolved.modelName();
+ data.providerName = resolved.providerName();
+ data.baseUrl = resolved.baseUrl();
}
@Override
diff --git
a/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedTextData.java
b/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedTextData.java
index bbb8da0534..092cee14c5 100644
---
a/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedTextData.java
+++
b/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedTextData.java
@@ -33,6 +33,16 @@ public class EmbedTextData extends BaseTransformData
implements ITransformData {
/** Model name actually in use, for the optional output field and for error
messages. */
public String modelName;
+ /** The AI provider and its resolved base URL, so a failed call can say
which endpoint it was. */
+ public String providerName;
+
+ public String baseUrl;
+
+ /**
+ * Pause before the first retry of a failed call. Each following retry waits
half as long again.
+ */
+ public long retryDelayMillis = 500L;
+
/** Batch size resolved once in init, so a variable is not re-resolved per
row. */
public int batchSize;
diff --git
a/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedTextMeta.java
b/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedTextMeta.java
index ee9f05c28e..4bd4a6c06e 100644
---
a/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedTextMeta.java
+++
b/plugins/tech/ai/src/main/java/org/apache/hop/ai/transforms/embedtext/EmbedTextMeta.java
@@ -17,12 +17,10 @@
package org.apache.hop.ai.transforms.embedtext;
import java.util.List;
-import java.util.function.IntPredicate;
import lombok.Getter;
import lombok.Setter;
import org.apache.hop.ai.metadata.AiProvider;
import org.apache.hop.core.CheckResult;
-import org.apache.hop.core.Const;
import org.apache.hop.core.ICheckResult;
import org.apache.hop.core.annotations.Transform;
import org.apache.hop.core.exception.HopTransformException;
@@ -71,6 +69,9 @@ public class EmbedTextMeta extends
BaseTransformMeta<EmbedText, EmbedTextData> {
private static final Class<?> PKG = EmbedTextMeta.class;
+ /** Used when the batch size is left empty. */
+ static final int DEFAULT_BATCH_SIZE = 16;
+
@GuiWidgetElement(
order = "0100",
type = GuiElementType.METADATA,
@@ -302,22 +303,54 @@ public class EmbedTextMeta extends
BaseTransformMeta<EmbedText, EmbedTextData> {
error(remarks, transformMeta,
"EmbedText.Validation.OutputFieldRequired");
}
// Batch size accepts variables, which a design-time check cannot resolve.
Only a value that is
- // genuinely fixed can be judged here.
- if (isResolvedNumberBad(variables, batchSize, size -> size < 1)) {
- error(remarks, transformMeta, "EmbedText.Validation.BatchSizePositive");
+ // genuinely fixed can be judged here, read the same way as in
EmbedText.init().
+ String resolvedBatchSize = variables.resolve(batchSize);
+ if (!StringUtil.containsVariableToken(resolvedBatchSize)) {
+ BatchSize size = parseBatchSize(resolvedBatchSize);
+ if (size.error() != null) {
+ remarks.add(new CheckResult(ICheckResult.TYPE_RESULT_ERROR,
size.error(), transformMeta));
+ }
}
if (includeModelMetadata && Utils.isEmpty(modelField) &&
Utils.isEmpty(dimensionsField)) {
warning(remarks, transformMeta, "EmbedText.Validation.NoMetadataFields");
}
}
- private static boolean isResolvedNumberBad(
- IVariables variables, String value, IntPredicate invalid) {
- String resolved = variables.resolve(value);
- if (StringUtil.containsVariableToken(resolved)) {
- return false;
+ /**
+ * A resolved batch size as the transform runs it, or why it cannot run with
it.
+ *
+ * @param value the batch size, meaningful only when there is no error
+ * @param error the reason the value cannot be used, or null
+ */
+ record BatchSize(int value, String error) {}
+
+ /**
+ * Reads a resolved batch size: trimmed, and {@link #DEFAULT_BATCH_SIZE}
when empty. Shared by
+ * {@link #check} and {@link EmbedText#init()} so Verify and a run never
disagree.
+ */
+ static BatchSize parseBatchSize(String resolved) {
+ if (resolved == null || resolved.isBlank()) {
+ return new BatchSize(DEFAULT_BATCH_SIZE, null);
+ }
+ String text = resolved.trim();
+ int value;
+ try {
+ value = Integer.parseInt(text);
+ } catch (NumberFormatException e) {
+ String key =
+ text.matches("\\+?\\d+")
+ ? "EmbedText.Validation.BatchSizeTooLarge"
+ : text.matches("-\\d+")
+ ? "EmbedText.Validation.BatchSizePositive"
+ : "EmbedText.Validation.BatchSizeNotANumber";
+ return new BatchSize(
+ 0, BaseMessages.getString(PKG, key, resolved,
String.valueOf(Integer.MAX_VALUE)));
+ }
+ if (value < 1) {
+ return new BatchSize(
+ 0, BaseMessages.getString(PKG,
"EmbedText.Validation.BatchSizePositive"));
}
- return invalid.test(Const.toInt(resolved, -1));
+ return new BatchSize(value, null);
}
private static void error(
diff --git
a/plugins/tech/ai/src/main/resources/org/apache/hop/ai/transforms/embedtext/messages/messages_en_US.properties
b/plugins/tech/ai/src/main/resources/org/apache/hop/ai/transforms/embedtext/messages/messages_en_US.properties
index 1afead0a30..98a55f6bca 100644
---
a/plugins/tech/ai/src/main/resources/org/apache/hop/ai/transforms/embedtext/messages/messages_en_US.properties
+++
b/plugins/tech/ai/src/main/resources/org/apache/hop/ai/transforms/embedtext/messages/messages_en_US.properties
@@ -43,9 +43,12 @@ EmbedText.Validation.InputFieldRequired=An input field is
required
EmbedText.Validation.InputFieldNotFound=Input field ''{0}'' not found in the
input stream
EmbedText.Validation.OutputFieldRequired=An output field is required
EmbedText.Validation.BatchSizePositive=Batch size must be at least 1
+EmbedText.Validation.BatchSizeNotANumber=Batch size ''{0}'' is not a whole
number. Check that every variable in it is set.
+EmbedText.Validation.BatchSizeTooLarge=Batch size ''{0}'' is too large. The
largest batch size is {1}.
EmbedText.Validation.NoMetadataFields=Include model metadata is on, but
neither the model field nor the dimensions field has a name, so nothing is added
-EmbedText.Error.Embedding=Error calling embedding model {0}
+EmbedText.Error.Embedding=Error calling embedding model {0} of AI provider
''{1}'' at {2}: {3}
+EmbedText.Log.Retrying={0}. Retry {1} of {2}.
EmbedText.Error.BatchSizeMismatch=The provider returned {0} embeddings for the
{1} text(s) it was sent
EmbedTextMeta.Injection.AI_PROVIDER=AI provider serving the embedding model
diff --git
a/plugins/tech/ai/src/test/java/org/apache/hop/ai/engine/AiEmbeddingFactoryTest.java
b/plugins/tech/ai/src/test/java/org/apache/hop/ai/engine/AiEmbeddingFactoryTest.java
new file mode 100644
index 0000000000..0a778c0593
--- /dev/null
+++
b/plugins/tech/ai/src/test/java/org/apache/hop/ai/engine/AiEmbeddingFactoryTest.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hop.ai.engine;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+import org.apache.hop.ai.provider.IAiProvider;
+import org.apache.hop.ai.providers.CustomOpenAiProvider;
+import org.apache.hop.ai.providers.OllamaProvider;
+import org.junit.jupiter.api.Test;
+
+class AiEmbeddingFactoryTest {
+
+ @Test
+ void namesLangchain4jsEndpointForAnOpenAiCompatibleProviderWithoutABaseUrl()
{
+ // OpenAiEmbeddingModel talks to api.openai.com when it gets no base URL,
so a failure there
+ // should say so rather than "at ?".
+ assertEquals(
+ AiEmbeddingFactory.OPEN_AI_DEFAULT_BASE_URL,
+ AiEmbeddingFactory.effectiveBaseUrl(settings("", new
CustomOpenAiProvider())));
+ }
+
+ @Test
+ void keepsAConfiguredBaseUrl() {
+ assertEquals(
+ "http://llm:8000/v1",
+ AiEmbeddingFactory.effectiveBaseUrl(
+ settings("http://llm:8000/v1", new CustomOpenAiProvider())));
+ assertEquals(
+ "http://localhost:11434",
+ AiEmbeddingFactory.effectiveBaseUrl(
+ settings("http://localhost:11434", new OllamaProvider())));
+ }
+
+ private static AiProviderSettings settings(String baseUrl, IAiProvider
backend) {
+ return new AiProviderSettings(baseUrl, "", null, null, null, null, null,
backend);
+ }
+}
diff --git
a/plugins/tech/ai/src/test/java/org/apache/hop/ai/transforms/embedtext/EmbedTextMetaTest.java
b/plugins/tech/ai/src/test/java/org/apache/hop/ai/transforms/embedtext/EmbedTextMetaTest.java
index 06a3041d47..1d1ef0005a 100644
---
a/plugins/tech/ai/src/test/java/org/apache/hop/ai/transforms/embedtext/EmbedTextMetaTest.java
+++
b/plugins/tech/ai/src/test/java/org/apache/hop/ai/transforms/embedtext/EmbedTextMetaTest.java
@@ -117,6 +117,46 @@ class EmbedTextMetaTest {
assertTrue(errorText(check(meta)).contains("Batch size"),
errorText(check(meta)));
}
+ /** check() reads the batch size the same way init() does: trimmed, empty
means the default. */
+ @Test
+ void acceptsTheBatchSizesThatRun() {
+ for (String batchSize : new String[] {" 32", "32 ", "", " "}) {
+ EmbedTextMeta meta = completeMeta();
+ meta.setBatchSize(batchSize);
+
+ assertTrue(
+ errorText(check(meta)).isEmpty(), "'" + batchSize + "': " +
errorText(check(meta)));
+ }
+ }
+
+ @Test
+ void rejectsAFixedBatchSizeThatIsNotANumber() {
+ EmbedTextMeta meta = completeMeta();
+ meta.setBatchSize("ten");
+
+ assertTrue(
+ errorText(check(meta)).contains("'ten' is not a whole number"),
errorText(check(meta)));
+ }
+
+ @Test
+ void saysAWholeNumberThatIsTooLargeIsTooLarge() {
+ EmbedTextMeta meta = completeMeta();
+ meta.setBatchSize("99999999999");
+
+ String errors = errorText(check(meta));
+ assertTrue(errors.contains("'99999999999' is too large"), errors);
+ }
+
+ @Test
+ void acceptsTheLowestIntAsANumberAndRejectsItAsTooSmall() {
+ // The old check used Integer.MIN_VALUE as its 'not a number' marker.
+ EmbedTextMeta meta = completeMeta();
+ meta.setBatchSize(String.valueOf(Integer.MIN_VALUE));
+
+ String errors = errorText(check(meta));
+ assertTrue(errors.contains("at least 1"), errors);
+ }
+
@Test
void reportsAMissingProviderAndInputField() {
EmbedTextMeta meta = new EmbedTextMeta();
diff --git
a/plugins/tech/ai/src/test/java/org/apache/hop/ai/transforms/embedtext/EmbedTextTest.java
b/plugins/tech/ai/src/test/java/org/apache/hop/ai/transforms/embedtext/EmbedTextTest.java
index 146e09eadb..3662fc3f32 100644
---
a/plugins/tech/ai/src/test/java/org/apache/hop/ai/transforms/embedtext/EmbedTextTest.java
+++
b/plugins/tech/ai/src/test/java/org/apache/hop/ai/transforms/embedtext/EmbedTextTest.java
@@ -30,8 +30,14 @@ import static org.mockito.Mockito.when;
import dev.langchain4j.data.embedding.Embedding;
import dev.langchain4j.data.segment.TextSegment;
+import dev.langchain4j.exception.AuthenticationException;
+import dev.langchain4j.exception.InternalServerException;
+import dev.langchain4j.exception.LangChain4jException;
+import dev.langchain4j.exception.RateLimitException;
import dev.langchain4j.model.embedding.EmbeddingModel;
import dev.langchain4j.model.output.Response;
+import java.net.ConnectException;
+import java.net.SocketTimeoutException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Iterator;
@@ -52,7 +58,11 @@ class EmbedTextTest {
private TransformMockHelper<EmbedTextMeta, EmbedTextData> helper;
private List<Object[]> passed;
private List<Object[]> diverted;
- private boolean failTheProvider;
+ private List<String> errorDescriptions;
+ private RuntimeException failure;
+ private int failuresLeft;
+ private int calls;
+ private boolean stopAfterFirstCall;
@BeforeAll
static void setUpClass() throws Exception {
@@ -66,7 +76,11 @@ class EmbedTextTest {
when(helper.pipeline.isRunning()).thenReturn(true);
passed = new ArrayList<>();
diverted = new ArrayList<>();
- failTheProvider = false;
+ errorDescriptions = new ArrayList<>();
+ failure = null;
+ failuresLeft = 0;
+ calls = 0;
+ stopAfterFirstCall = false;
}
@AfterEach
@@ -127,11 +141,190 @@ class EmbedTextTest {
assertFalse(EmbedText.hasNothingToEmbed(" a "));
}
+ @Test
+ void findsNothingToEmbedInNoBreakAndZeroWidthSpaces() {
+ // isBlank() keeps all of these, so they used to be sent and came back as
a meaningless vector.
+ assertTrue(EmbedText.hasNothingToEmbed("\u00A0\u00A0"), "no-break spaces");
+ assertTrue(EmbedText.hasNothingToEmbed("\u202F"), "narrow no-break space");
+ assertTrue(EmbedText.hasNothingToEmbed("\u2007"), "figure space");
+ assertTrue(EmbedText.hasNothingToEmbed("\u200B"), "zero-width space");
+ assertTrue(EmbedText.hasNothingToEmbed("\uFEFF \u2060"), "byte order mark
and word joiner");
+ assertFalse(EmbedText.hasNothingToEmbed("\u200Bword\u00A0"));
+ assertFalse(EmbedText.hasNothingToEmbed("\u4E2D"), "a CJK character is
text");
+ }
+
+ @Test
+ void doesNotSendTextOfOnlyUnicodeSpaces() throws Exception {
+ run(Arrays.asList(new Object[] {"first"}, new Object[]
{"\u00A0\u202F\u200B"}));
+
+ assertEquals(2, passed.size());
+ assertEquals("[1.0,2.0]", passed.get(0)[1]);
+ assertNull(passed.get(1)[1], "a row of Unicode spaces gets no embedding");
+ }
+
+ @Test
+ void refusesABatchSizeThatIsNotANumber() {
+ // An unset variable stays as ${...}; it used to run with 16 without a
word.
+ assertFalse(initWithBatchSize("${EMBED_BATCH_NOT_SET}"));
+ assertFalse(initWithBatchSize("ten"));
+ }
+
+ @Test
+ void usesTheDefaultBatchSizeWhenItIsEmptyAndTheValueWhenItIsSet() {
+ EmbedTextData data = new EmbedTextData();
+ assertTrue(initWithBatchSize("", data));
+ assertEquals(EmbedTextMeta.DEFAULT_BATCH_SIZE, data.batchSize);
+
+ data = new EmbedTextData();
+ assertTrue(initWithBatchSize(" 8 ", data));
+ assertEquals(8, data.batchSize);
+
+ data = new EmbedTextData();
+ assertTrue(initWithBatchSize(" ", data));
+ assertEquals(EmbedTextMeta.DEFAULT_BATCH_SIZE, data.batchSize);
+ }
+
+ @Test
+ void namesTheProviderAndEndpointWhenTheProviderCannotBeReached() throws
Exception {
+ when(helper.transformMeta.isDoingErrorHandling()).thenReturn(true);
+ failWith(new RuntimeException(new ConnectException()), Integer.MAX_VALUE);
+
+ run(Arrays.asList(new Object[] {"first"}, new Object[] {"second"}));
+
+ assertEquals(2, diverted.size());
+ String description = errorDescriptions.get(0);
+ assertTrue(description.contains("test-model"), description);
+ assertTrue(description.contains("'local-ollama'"), description);
+ assertTrue(description.contains("http://localhost:11499"), description);
+ assertTrue(description.contains("ConnectException"), description);
+ }
+
+ @Test
+ void stopsWithTheProviderAndEndpointWhenThereIsNoErrorHop() {
+ failWith(new RuntimeException(new ConnectException()), Integer.MAX_VALUE);
+
+ HopException e =
+ assertThrows(HopException.class, () -> run(List.<Object[]>of(new
Object[] {"first"})));
+ assertTrue(e.getMessage().contains("'local-ollama' at
http://localhost:11499"), e.getMessage());
+ assertEquals(1 + EmbedText.MAX_RETRIES, calls, "the first attempt and
every retry");
+ }
+
+ @Test
+ void retriesAFailedCallAndKeepsTheEmbeddings() throws Exception {
+ failWith(new RateLimitException("provider is busy"),
EmbedText.MAX_RETRIES);
+
+ run(Arrays.asList(new Object[] {"first"}, new Object[] {"second"}));
+
+ assertEquals(1 + EmbedText.MAX_RETRIES, calls);
+ assertEquals(2, passed.size());
+ assertEquals("[1.0,2.0]", passed.get(1)[1]);
+ }
+
+ @Test
+ void doesNotRetryARefusalThatWillNotChange() {
+ failWith(new AuthenticationException("invalid api key"),
Integer.MAX_VALUE);
+
+ assertThrows(HopException.class, () -> run(List.<Object[]>of(new Object[]
{"first"})));
+ assertEquals(1, calls);
+ }
+
+ @Test
+ void doesNotRetryAFailureThatWouldHappenAgain() {
+ // An answer that cannot be read is not going to read better on the next
attempt.
+ failWith(new IllegalStateException("cannot map the response"),
Integer.MAX_VALUE);
+
+ assertThrows(HopException.class, () -> run(List.<Object[]>of(new Object[]
{"first"})));
+ assertEquals(1, calls);
+ }
+
+ @Test
+ void retriesOnlyFailuresThatMayPass() {
+ assertTrue(EmbedText.isRetriable(new RateLimitException("slow down")));
+ assertTrue(EmbedText.isRetriable(new InternalServerException("502")));
+ assertTrue(EmbedText.isRetriable(new LangChain4jException(new
ConnectException())));
+ assertTrue(EmbedText.isRetriable(new RuntimeException(new
SocketTimeoutException())));
+ assertFalse(EmbedText.isRetriable(new AuthenticationException("invalid api
key")));
+ assertFalse(EmbedText.isRetriable(new IllegalStateException("cannot map
the response")));
+ assertFalse(EmbedText.isRetriable(new RuntimeException("input must not be
blank")));
+ }
+
+ @Test
+ void abandonsTheBatchWhenThePipelineIsStoppedWhileRetrying() throws
Exception {
+ // Pressing Stop during an outage is not a failure of the rows that were
waiting.
+ when(helper.transformMeta.isDoingErrorHandling()).thenReturn(true);
+ failWith(new LangChain4jException(new ConnectException()),
Integer.MAX_VALUE);
+ stopAfterFirstCall = true;
+
+ run(Arrays.asList(new Object[] {"first"}, new Object[] {"second"}));
+
+ assertEquals(1, calls, "no retry after the stop");
+ assertTrue(diverted.isEmpty(), "no error rows for a stopped run");
+ assertTrue(passed.isEmpty());
+ }
+
+ @Test
+ void leavesCredentialsOutOfTheEndpointInMessages() {
+ assertEquals(
+ "https://host:8443/v1",
+
EmbedText.endpointForMessages("https://user:secret@host:8443/v1?api_key=abc#x"));
+ assertEquals("http://localhost:11434",
EmbedText.endpointForMessages("http://localhost:11434"));
+ assertEquals("", EmbedText.endpointForMessages(""));
+ }
+
+ @Test
+ void leavesCredentialsOutOfAnEndpointThatJavaNetUriCannotSplit() {
+ // An underscore makes the authority registry based, so URI finds no host
and no user info.
+ assertEquals(
+ "http://ollama_host:11434",
+ EmbedText.endpointForMessages("http://user:secret@ollama_host:11434"));
+ // No scheme, so URI reads it as opaque and finds no query.
+ assertEquals("myhost:11434/v1",
EmbedText.endpointForMessages("myhost:11434/v1?api_key=abc"));
+ assertEquals("ollama:11434",
EmbedText.endpointForMessages("user:secret@ollama:11434"));
+ }
+
+ @Test
+ void masksSecretsInTheCauseMessage() {
+ String masked =
+ EmbedText.maskSecrets(
+ "Incorrect API key provided: sk-proj-abcdef123456. Called"
+ + " https://user:pw@host/v1/embeddings?api_key=abc123&x=1 with
Bearer xyz789");
+
+ assertFalse(masked.contains("abcdef123456"), masked);
+ assertFalse(masked.contains("pw@"), masked);
+ assertFalse(masked.contains("abc123"), masked);
+ assertFalse(masked.contains("xyz789"), masked);
+
assertTrue(masked.contains("https://***@host/v1/embeddings?api_key=***&x=1"),
masked);
+ }
+
+ @Test
+ void cutsALongCauseMessageShortAndKeepsItOnOneLine() {
+ String page = "<html>\n<body>\n" + "Bad gateway ".repeat(200) +
"</body>\n</html>";
+
+ String masked = EmbedText.maskSecrets(page);
+
+ assertEquals(EmbedText.MAX_CAUSE_LENGTH + 3, masked.length());
+ assertFalse(masked.contains("\n"), masked);
+ }
+
+ @Test
+ void masksTheCauseInTheErrorRows() throws Exception {
+ when(helper.transformMeta.isDoingErrorHandling()).thenReturn(true);
+ failWith(
+ new AuthenticationException("Incorrect API key provided:
sk-live-0123456789abcdef"),
+ Integer.MAX_VALUE);
+
+ run(List.<Object[]>of(new Object[] {"first"}));
+
+ String description = errorDescriptions.get(0);
+ assertFalse(description.contains("0123456789abcdef"), description);
+ assertTrue(description.contains("AuthenticationException"), description);
+ }
+
@Test
void divertsOnlyTheRowsThatWereActuallySent() throws Exception {
// A row with no text is not in the request, so a provider failure is not
its failure.
when(helper.transformMeta.isDoingErrorHandling()).thenReturn(true);
- failTheProvider = true;
+ failWith(new RuntimeException("provider is down"), Integer.MAX_VALUE);
run(Arrays.asList(new Object[] {"first"}, new Object[] {""}, new Object[]
{"third"}));
@@ -163,6 +356,26 @@ class EmbedTextTest {
assertTrue(e.getMessage().contains("output field"), e.getMessage());
}
+ private void failWith(RuntimeException exception, int times) {
+ failure = exception;
+ failuresLeft = times;
+ }
+
+ private boolean initWithBatchSize(String batchSize) {
+ return initWithBatchSize(batchSize, new EmbedTextData());
+ }
+
+ private boolean initWithBatchSize(String batchSize, EmbedTextData data) {
+ EmbedTextMeta meta = new EmbedTextMeta();
+ meta.setDefault();
+ meta.setAiProvider("ollama");
+ meta.setInputField("chunk_text");
+ meta.setBatchSize(batchSize);
+ EmbedText transform =
+ new EmbedText(helper.transformMeta, meta, data, 0,
helper.pipelineMeta, helper.pipeline);
+ return transform.init();
+ }
+
private void run(List<Object[]> rows) throws Exception {
EmbedTextMeta meta = new EmbedTextMeta();
meta.setDefault();
@@ -180,14 +393,19 @@ class EmbedTextTest {
data.inputFieldIndex = 0;
data.batchSize = 16;
data.modelName = "test-model";
+ data.providerName = "local-ollama";
+ data.baseUrl = "http://localhost:11499";
+ data.retryDelayMillis = 0L;
data.outputFieldName = "embedding";
EmbeddingModel model = mock(EmbeddingModel.class);
when(model.embedAll(any()))
.thenAnswer(
invocation -> {
- if (failTheProvider) {
- throw new RuntimeException("provider is down");
+ calls++;
+ if (failuresLeft > 0) {
+ failuresLeft--;
+ throw failure;
}
List<TextSegment> segments = invocation.getArgument(0);
// Like the providers, reject blank input for the whole request.
@@ -225,11 +443,16 @@ class EmbedTextTest {
doAnswer(
invocation -> {
diverted.add(invocation.getArgument(1));
+ errorDescriptions.add(invocation.getArgument(3));
return null;
})
.when(transform)
.putError(any(), any(), anyLong(), any(), any(), any());
+ if (stopAfterFirstCall) {
+ doAnswer(invocation -> calls > 0).when(transform).isStopped();
+ }
+
while (transform.processRow()) {
// drain
}