This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new 61589bed69 TIKA-4896: requestParameters and imageInput on
openai-embedding-engin… (#3183)
61589bed69 is described below
commit 61589bed69f8f816f149d82f886e76c54160cece
Author: Tim Allison <[email protected]>
AuthorDate: Wed Sep 16 13:32:27 2026 -0400
TIKA-4896: requestParameters and imageInput on openai-embedding-engin…
(#3183)
---
CHANGES.txt | 3 +
.../ROOT/pages/configuration/inference.adoc | 89 +++++++++++++++++++++-
.../apache/tika/cli/TikaCLITextVectorsTest.java | 6 +-
.../tika/inference/OpenAIEmbeddingEngine.java | 64 +++++++++++++++-
.../tika/inference/OpenAIEmbeddingEngineTest.java | 52 +++++++++++++
5 files changed, 210 insertions(+), 4 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index ff34ddd949..4a41f7d0a1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,8 @@
Release 4.1.0 - unreleased
+ * openai-embedding-engine gains "requestParameters" to enable custom
+ parameters per vendor (TIKA-4896).
+
* New catalog presets, inert until the config names them. (TIKA-4856).
* The inference dispatcher no longer copies the top-level document's own
diff --git a/docs/modules/ROOT/pages/configuration/inference.adoc
b/docs/modules/ROOT/pages/configuration/inference.adoc
index f559595fe7..8a958813a6 100644
--- a/docs/modules/ROOT/pages/configuration/inference.adoc
+++ b/docs/modules/ROOT/pages/configuration/inference.adoc
@@ -49,7 +49,94 @@ different models. Secrets and paths live here and only here.
`openai-embedding-engine` posts to `/v1/embeddings` in the OpenAI shape and
sends up to
`maxBatchSize` images per request. Options: `baseUrl`, `model`, `apiKey`,
`timeoutMillis`
-(120000), `maxBatchSize` (32), `embeddingsPath`, `apiKeyHeaderName`,
`apiKeyPrefix`.
+(120000), `maxBatchSize` (32), `embeddingsPath`, `apiKeyHeaderName`,
`apiKeyPrefix`,
+`requestParameters`, `imageInput`.
+
+=== Vendor keys: `requestParameters`
+
+Most hosted and self-hosted embedding services speak the OpenAI shape and add
their own keys
+to it. `requestParameters` is a map copied verbatim into every request body,
beside the
+engine's `model` and `input`; the engine neither knows nor checks the keys, so
use the names
+your vendor documents. Three keys recur under different names: a Matryoshka
dimension
+(`dimensions`, `output_dimension`), a query-versus-document hint (`task`,
`input_type`,
+`task_type`; use the document/passage value, since Tika indexes), and
normalize/truncate
+flags. Two rules: `model` and `input` belong to the engine and are refused; so
is any key that
+changes the response from float vectors by index (`encoding_format`,
`embedding_type`,
+`output_type`, `output_dtype`, `return_multivector`), since that is what the
engine reads back.
+
+Jina `jina-embeddings-v5-omni` embeds text, images and rendered pages in one
space; `task`
+selects the indexing adapter and `dimensions` the vector size. `PAGES` sends
each page's
+rendering as a PNG, never the PDF itself, and the PDF parser renders only when
its
+`inference.input` says so:
+
+[source,json]
+----
+{
+ "engines": {
+ "jina": { "openai-embedding-engine": {
+ "baseUrl": "https://api.jina.ai";, "apiKey": "...",
+ "model": "jina-embeddings-v5-omni-small", "maxBatchSize": 16,
+ "requestParameters": { "task": "retrieval.passage", "dimensions": 1024 }
} }
+ },
+ "inference": [
+ { "id": "pictures", "engine": "jina", "input": "IMAGES", "tasks":
["embed"], "maxBytes": 5000000 },
+ { "id": "pages", "engine": "jina", "input": "PAGES", "tasks":
["embed"], "maxBytes": 5000000 },
+ { "id": "text", "engine": "jina", "input": "TEXT", "tasks":
["embed"],
+ "chunker": { "markdown-chunker": { "maxChunkChars": 1500,
"overlapChars": 200 } } }
+ ],
+ "parsers": [
+ { "default-parser": {} },
+ { "pdf-parser": { "inference": { "input": ["TEXT", "PAGES"] },
+ "rendering": { "dpi": 96, "imageType": "RGB" } } }
+ ]
+}
+----
+
+Azure OpenAI puts the key in an `api-key` header with no prefix and the
deployment and API
+version in the path; `dimensions` truncates `text-embedding-3-large`. Plain
OpenAI is the
+same engine with the defaults and `"baseUrl": "https://api.openai.com"`:
+
+[source,json]
+----
+{
+ "engines": {
+ "azure": { "openai-embedding-engine": {
+ "baseUrl": "https://my-resource.openai.azure.com";,
+ "embeddingsPath":
"/openai/deployments/text-embedding-3-large/embeddings?api-version=2024-10-21",
+ "apiKeyHeaderName": "api-key", "apiKeyPrefix": "", "apiKey": "...",
+ "requestParameters": { "dimensions": 1024 } } }
+ }
+}
+----
+
+The OpenAI shape has no image input, so services differ on how an image goes
into `input`:
+`imageInput` is `object` (the default: `{"image": "data:..."}`, Jina's form)
or `data-uri` (a
+bare `data:...` string, the form LiteLLM takes).
+
+For a service whose request shape is not OpenAI's (Google's `embedContent`,
Amazon Bedrock's
+`InvokeModel`, Cohere's `/v2/embed`), put a gateway that speaks the OpenAI
shape in front of it
+and point this engine at the gateway. LiteLLM routes `gemini-embedding-2` this
way, text and
+images alike, on the Gemini API path (its Vertex path fuses every input into
one vector,
+which this engine refuses because it expects one per input), and Amazon Nova 2
on Bedrock
+for text and images, one input per request; the gateway holds the cloud
credentials, and
+provider keys ride `requestParameters`. Audio and video on those services go
through
+asynchronous, object-store-backed jobs that an embeddings call cannot use.
+
+[source,json]
+----
+{
+ "engines": {
+ "gemini": { "openai-embedding-engine": {
+ "baseUrl": "http://litellm:4000";, "apiKey": "...",
+ "model": "gemini-embedding-2-preview", "imageInput": "data-uri",
"maxBatchSize": 6 } },
+ "nova": { "openai-embedding-engine": {
+ "baseUrl": "http://litellm:4000";, "apiKey": "...",
+ "model": "bedrock/amazon.nova-2-multimodal-embeddings-v1:0",
+ "imageInput": "data-uri", "maxBatchSize": 1,
+ "requestParameters": { "dimensions": 1024, "embedding_purpose":
"GENERIC_INDEX" } } }
+ }
+}
+----
A text recognizer is an engine too: `"tesseract": { "tesseract-ocr-parser": {
"language": "eng" } }`
here, and `{ "engine": "tesseract" }` in
diff --git
a/tika-app/src/test/java/org/apache/tika/cli/TikaCLITextVectorsTest.java
b/tika-app/src/test/java/org/apache/tika/cli/TikaCLITextVectorsTest.java
index 899e7ac1c4..1676bbc16d 100644
--- a/tika-app/src/test/java/org/apache/tika/cli/TikaCLITextVectorsTest.java
+++ b/tika-app/src/test/java/org/apache/tika/cli/TikaCLITextVectorsTest.java
@@ -67,7 +67,8 @@ public class TikaCLITextVectorsTest {
Files.writeString(config, """
{
"engines": {
- "embedder": { "openai-embedding-engine": { "baseUrl":
"BASE_URL", "model": "bge" } }
+ "embedder": { "openai-embedding-engine": { "baseUrl":
"BASE_URL", "model": "bge",
+ "requestParameters": { "task": "retrieval.passage",
"dimensions": 2, "normalized": true } } }
},
"inference": [
{ "id": "text-vectors", "engine": "embedder", "input":
"TEXT", "tasks": ["embed"],
@@ -84,6 +85,9 @@ public class TikaCLITextVectorsTest {
JsonNode request = new
ObjectMapper().readTree(server.takeRequest().body());
// the zip's own markdown content is two headings, so two chunks;
one per note
assertEquals(4, request.get("input").size(), "chunks of all three
documents in one request");
+ assertEquals("retrieval.passage", request.get("task").asText(),
"requestParameters ride the body");
+ assertEquals(2, request.get("dimensions").asInt());
+ assertTrue(request.get("normalized").asBoolean());
assertEquals("# alpha.txt", request.get("input").get(0).asText());
assertEquals("alpha is the first note",
request.get("input").get(2).asText());
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIEmbeddingEngine.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIEmbeddingEngine.java
index a6d431d3f9..d64485e4e2 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIEmbeddingEngine.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/OpenAIEmbeddingEngine.java
@@ -20,8 +20,10 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.Base64;
import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
@@ -41,6 +43,12 @@ import org.apache.tika.utils.StringUtils;
* An OpenAI-compatible embeddings endpoint ({@code POST /v1/embeddings}),
named in
* {@code "engines"}. Sends up to {@code maxBatchSize} images per request as
* {@code {"image": "data:<mime>;base64,..."}} inputs and reads the vectors
back by index.
+ * {@code requestParameters} ride every request body verbatim, for the keys a
vendor adds to
+ * the shape (Jina's {@code task}, a Matryoshka {@code dimensions}); the
engine reads float
+ * vectors back, so a key that changes the response shape is refused at load.
The OpenAI
+ * shape has no image input, so vendors differ: {@code imageInput} is {@code
"object"} for
+ * the {@code {"image": ...}} form (Jina) or {@code "data-uri"} for a bare
data URI in
+ * {@code input} (LiteLLM and other gateways).
*/
@TikaComponent(name = "openai-embedding-engine", spi = false)
public class OpenAIEmbeddingEngine implements EmbeddingEngine, Initializable {
@@ -55,6 +63,16 @@ public class OpenAIEmbeddingEngine implements
EmbeddingEngine, Initializable {
private String embeddingsPath = "/v1/embeddings";
private String apiKeyHeaderName = "Authorization";
private String apiKeyPrefix = "Bearer ";
+ private Map<String, Object> requestParameters = new LinkedHashMap<>();
+ private String imageInput = IMAGE_INPUT_OBJECT;
+
+ public static final String IMAGE_INPUT_OBJECT = "object";
+ public static final String IMAGE_INPUT_DATA_URI = "data-uri";
+
+ private static final Set<String> ENGINE_OWNED = Set.of("model", "input");
+ /** Keys that would make the response something other than float vectors
by index. */
+ private static final Set<String> RESPONSE_SHAPING =
Set.of("encoding_format",
+ "embedding_type", "output_type", "output_dtype",
"return_multivector");
private TikaHttpClient httpClient;
@@ -66,6 +84,21 @@ public class OpenAIEmbeddingEngine implements
EmbeddingEngine, Initializable {
if (maxBatchSize < 1) {
throw new TikaConfigException("maxBatchSize must be at least 1");
}
+ if (!IMAGE_INPUT_OBJECT.equals(imageInput) &&
!IMAGE_INPUT_DATA_URI.equals(imageInput)) {
+ throw new TikaConfigException("imageInput must be \"" +
IMAGE_INPUT_OBJECT + "\" or \""
+ + IMAGE_INPUT_DATA_URI + "\", not \"" + imageInput + "\"");
+ }
+ for (String key : requestParameters.keySet()) {
+ if (ENGINE_OWNED.contains(key)) {
+ throw new TikaConfigException("requestParameters may not set
\"" + key
+ + "\"; the engine writes it");
+ }
+ if (RESPONSE_SHAPING.contains(key)) {
+ throw new TikaConfigException("requestParameters may not set
\"" + key
+ + "\": the engine reads float vectors back, so the
response shape "
+ + "must stay the default");
+ }
+ }
httpClient = TikaHttpClient.build(30);
}
@@ -75,8 +108,13 @@ public class OpenAIEmbeddingEngine implements
EmbeddingEngine, Initializable {
ObjectNode root = request();
ArrayNode input = root.putArray("input");
for (int i = 0; i < images.size(); i++) {
- input.addObject().put("image", "data:" + mimeTypes.get(i) +
";base64,"
- + Base64.getEncoder().encodeToString(images.get(i)));
+ String dataUri = "data:" + mimeTypes.get(i) + ";base64,"
+ + Base64.getEncoder().encodeToString(images.get(i));
+ if (IMAGE_INPUT_DATA_URI.equals(imageInput)) {
+ input.add(dataUri);
+ } else {
+ input.addObject().put("image", dataUri);
+ }
}
return post(root, images.size(), context);
}
@@ -94,6 +132,9 @@ public class OpenAIEmbeddingEngine implements
EmbeddingEngine, Initializable {
private ObjectNode request() {
ObjectNode root = MAPPER.createObjectNode();
+ for (Map.Entry<String, Object> e : requestParameters.entrySet()) {
+ root.set(e.getKey(), MAPPER.valueToTree(e.getValue()));
+ }
if (!StringUtils.isBlank(model)) {
root.put("model", model);
}
@@ -220,6 +261,25 @@ public class OpenAIEmbeddingEngine implements
EmbeddingEngine, Initializable {
this.apiKeyHeaderName = apiKeyHeaderName;
}
+ /** How an image goes into {@code input}: {@code "object"} or {@code
"data-uri"}; see the class note. */
+ public String getImageInput() {
+ return imageInput;
+ }
+
+ public void setImageInput(String imageInput) {
+ this.imageInput = imageInput;
+ }
+
+ /** Extra keys for every request body, as the vendor names them; see the
class note. */
+ public Map<String, Object> getRequestParameters() {
+ return requestParameters;
+ }
+
+ public void setRequestParameters(Map<String, Object> requestParameters) {
+ this.requestParameters = requestParameters == null ? new
LinkedHashMap<>()
+ : new LinkedHashMap<>(requestParameters);
+ }
+
public String getApiKeyPrefix() {
return apiKeyPrefix;
}
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/OpenAIEmbeddingEngineTest.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/OpenAIEmbeddingEngineTest.java
index fdb658c5f0..9c1fcc7b0a 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/OpenAIEmbeddingEngineTest.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/OpenAIEmbeddingEngineTest.java
@@ -18,6 +18,7 @@ package org.apache.tika.inference;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import java.nio.charset.StandardCharsets;
import java.util.List;
@@ -123,4 +124,55 @@ public class OpenAIEmbeddingEngineTest {
bad.setMaxBatchSize(0);
assertThrows(org.apache.tika.exception.TikaConfigException.class,
bad::initialize);
}
+
+ /** Vendor keys ride every body verbatim, beside the engine's own model
and input. */
+ @Test
+ public void testRequestParametersRideEveryRequest() throws Exception {
+ engine.setRequestParameters(new
java.util.LinkedHashMap<>(java.util.Map.of(
+ "task", "retrieval.passage", "dimensions", 1024, "normalized",
true)));
+ engine.initialize();
+ ObjectMapper mapper = new ObjectMapper();
+ for (int i = 0; i < 2; i++) {
+ server.enqueue(new TikaTestHttpServer.MockResponse(200,
+ "{\"data\":[{\"index\":0,\"embedding\":[0.5]}]}"));
+ }
+ engine.embedTexts(List.of("alpha"), new ParseContext());
+ engine.embedImages(List.of(bytes("png")), List.of("image/png"), new
ParseContext());
+ for (int i = 0; i < 2; i++) {
+ JsonNode request = mapper.readTree(server.takeRequest().body());
+ assertEquals("retrieval.passage", request.get("task").asText());
+ assertEquals(1024, request.get("dimensions").asInt());
+ assertTrue(request.get("normalized").asBoolean());
+ assertEquals("clip", request.get("model").asText(), "the engine
still writes model");
+ assertEquals(1, request.get("input").size());
+ }
+ }
+
+ /** A gateway that takes bare data URIs in input, as LiteLLM does, gets
strings, not objects. */
+ @Test
+ public void testImageInputAsDataUriString() throws Exception {
+ engine.setImageInput(OpenAIEmbeddingEngine.IMAGE_INPUT_DATA_URI);
+ engine.initialize();
+ server.enqueue(new TikaTestHttpServer.MockResponse(200,
+ "{\"data\":[{\"index\":0,\"embedding\":[0.5]}]}"));
+ engine.embedImages(List.of(bytes("png")), List.of("image/png"), new
ParseContext());
+ JsonNode input = new
ObjectMapper().readTree(server.takeRequest().body()).get("input");
+ assertTrue(input.get(0).isTextual(), "a bare string, not an object");
+ assertTrue(input.get(0).asText().startsWith("data:image/png;base64,"));
+
+ OpenAIEmbeddingEngine bad = new OpenAIEmbeddingEngine();
+ bad.setImageInput("base64");
+ assertThrows(org.apache.tika.exception.TikaConfigException.class,
bad::initialize);
+ }
+
+ /** The engine owns model and input, and reads float vectors: those keys
are refused at load. */
+ @Test
+ public void testRequestParametersReservedKeysAreRefused() {
+ for (String key : List.of("model", "input", "encoding_format",
"embedding_type",
+ "output_type")) {
+ OpenAIEmbeddingEngine bad = new OpenAIEmbeddingEngine();
+ bad.setRequestParameters(java.util.Map.of(key, "x"));
+ assertThrows(org.apache.tika.exception.TikaConfigException.class,
bad::initialize, key);
+ }
+ }
}
