This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new dccc702fca TIKA-4889: PAGES input for inference bindings (#3155)
dccc702fca is described below
commit dccc702fca054c47f43fa3c7ce73cc328ace1ebf
Author: Tim Allison <[email protected]>
AuthorDate: Fri Sep 11 17:27:55 2026 -0400
TIKA-4889: PAGES input for inference bindings (#3155)
---
CHANGES.txt | 8 ++
.../ROOT/pages/configuration/inference.adoc | 58 +++++++-
.../pages/configuration/parsers/pdf-parser.adoc | 9 ++
.../apache/tika/cli/TikaCLIPageVectorsTest.java | 147 +++++++++++++++++++++
.../org/apache/tika/parser/AutoDetectParser.java | 2 +-
.../org/apache/tika/parser/hook/ParseHook.java | 17 +++
.../org/apache/tika/parser/hook/ParseHooks.java | 65 ++++++++-
.../tika/parser/inference/InferenceDispatcher.java | 36 ++++-
.../tika/parser/inference/InferenceUnit.java | 20 ++-
.../apache/tika/parser/hook/ParseHooksTest.java | 109 ++++++++++++++-
.../parser/inference/InferenceDispatcherTest.java | 45 +++++++
.../java/org/apache/tika/inference/EmbedTask.java | 10 +-
.../org/apache/tika/inference/EmbedTaskTest.java | 26 ++++
.../resources/config-examples/pdf-parser-full.json | 3 +
.../apache/tika/parser/pdf/AbstractPDF2XHTML.java | 116 ++++++++++++----
.../apache/tika/parser/pdf/InferenceConfig.java | 41 ++++++
.../java/org/apache/tika/parser/pdf/OCR2XHTML.java | 5 +-
.../apache/tika/parser/pdf/PDFParserConfig.java | 11 ++
.../org/apache/tika/parser/pdf/PDFParserTest.java | 129 ++++++++++++++++++
.../apache/tika/parser/pdf/tika-inline-config.json | 1 +
.../tika/config/loader/InferenceLoaderTest.java | 18 +++
.../server/standard/InferenceBindingsWireTest.java | 66 +++++++++
.../configs/tika-config-inference-bindings.json | 5 +-
23 files changed, 896 insertions(+), 51 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 617757c94a..7d9d547f0d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,13 @@
Release 4.1.0 - unreleased
+ * PAGES input for inference bindings: pdf-parser.inference.input lists what
+ the PDF releases (TEXT by default; PAGES renders every page for a PAGES
+ binding). One render per page feeds OCR and the bindings, and each page's
+ vector lands on the PDF with a page locator. A page render emitted as an
+ embedded document is no longer an IMAGES unit. OCR_ONLY annotated an OCR'd
+ page twice when pages were rendered as embedded documents; once now
+ (TIKA-4889).
+
* New "inference" config section: bind a named engine to an input (IMAGES)
and tasks (embed), and every image document Tika parses is embedded in
one request per document tree; vectors land on the document the image
diff --git a/docs/modules/ROOT/pages/configuration/inference.adoc
b/docs/modules/ROOT/pages/configuration/inference.adoc
index 1757489bb0..5b242d20f8 100644
--- a/docs/modules/ROOT/pages/configuration/inference.adoc
+++ b/docs/modules/ROOT/pages/configuration/inference.adoc
@@ -59,10 +59,14 @@ A list of bindings. Each names an engine, says what it is
fed, and what to ask:
`id`:: Names the binding; defaults to `<engine>-<input>`. Unique within the
list.
`input`:: What the binding is fed. `IMAGES` is every image document Tika
parses: an inline
- picture, an image attachment, a top-level image file. (`PAGES`, `TEXT` and
`MEDIA` follow.)
+ picture, an image attachment, a top-level image file; a page render emitted
as an embedded
+ document is not one. `PAGES` is one image per rendered page of a document
whose parser
+ renders pages for inference: the PDF parser, when its `inference.input` says
so (see
+ <<pages>>). (`TEXT` and `MEDIA` follow.)
`tasks`:: One call per task on the same engine and input. `embed` writes one
vector chunk per
unit into `tk:chunks`, with the page locator when the unit has a page.
-`maxChunks`:: How many units one document tree may send; -1 for no limit.
+`maxChunks`:: How many units one document tree may send; for `PAGES`, how many
pages of one
+ document; -1 for no limit.
`maxBytes`:: Largest unit the binding accepts (20 MB unless set; -1 for no
limit); larger
ones are skipped and counted.
`enabled`:: `false` keeps the binding configured but idle until a request
selects it.
@@ -91,6 +95,56 @@ The units are held as files the dispatcher owns, bounded by
each binding's `maxC
(`/rmeta`, `-J`, pipes); a streaming `/tika` parse keeps only what lands on
the top-level
document.
+[#pages]
+== Pages
+
+A `PAGES` binding is fed by the parsers that render pages. Today that is the
PDF parser, and
+only when its own config releases pages:
+
+[source,json]
+----
+{
+ "engines": {
+ "clip": { "openai-embedding-engine": { "baseUrl": "http://clip:8000/v1";,
"model": "clip" } }
+ },
+ "inference": [
+ { "id": "page-vectors", "engine": "clip", "input": "PAGES", "tasks":
["embed"], "maxChunks": 50 }
+ ],
+ "parsers": [
+ { "default-parser": {} },
+ { "pdf-parser": { "inference": { "input": ["PAGES"] },
+ "ocr": { "strategy": "NO_OCR", "dpi": 100, "imageType":
"RGB" } } }
+ ]
+}
+----
+
+`pdf-parser.inference.input` lists what the PDF releases to inference: `TEXT`,
the default
+(text bindings follow), and `PAGES`. With `PAGES`, and a `PAGES` binding that
runs for the
+request, the PDF renders every page once and hands the same image to the OCR
engine, when
+`ocr.strategy` asks for it, and to the bindings: a scanned page costs one
render whether one
+consumer or two see it. Each page is one unit. Its vector lands on the PDF's
own `tk:chunks`
+with a `paginated` locator naming the page, and `maxChunks` is the number of
pages one document
+sends. Nothing is rendered for inference unless a `PAGES` binding runs, so the
block can stay on
+a config whose bindings are on standby.
+
+The render is the OCR render: `ocr.dpi`, `imageType`, `imageFormat` and
`imageQuality` shape
+it, and the defaults (300 dpi grayscale PNG) suit Tesseract, not an embedding
endpoint. Set
+`dpi` and `imageType` to what the engine wants; when OCR and page vectors run
together the one
+setting serves both. `renderingStrategy` and `maxImagePixels` apply as for
OCR; a page skipped
+for its size is not offered.
+
+Page renders emitted as embedded documents (`imageStrategy` set to
`RENDER_PAGES_AT_PAGE_END`
+or `RENDER_PAGES_BEFORE_PARSE`) are not images to an `IMAGES` binding: pages
reach inference
+through `PAGES` only, so no page is embedded twice.
+
+Per request, like every PDF setting:
+
+[source,json]
+----
+{ "parse-context": { "pdf-parser": { "inference": { "input": ["PAGES"] } },
+ "inference": { "bindings": ["page-vectors"] } } }
+----
+
== Per request
[source,json]
diff --git a/docs/modules/ROOT/pages/configuration/parsers/pdf-parser.adoc
b/docs/modules/ROOT/pages/configuration/parsers/pdf-parser.adoc
index 16d386b406..1216768113 100644
--- a/docs/modules/ROOT/pages/configuration/parsers/pdf-parser.adoc
+++ b/docs/modules/ROOT/pages/configuration/parsers/pdf-parser.adoc
@@ -73,5 +73,14 @@ render is parsed for its metadata but not enriched again, so
no page reaches the
twice, and `NO_OCR` means no page OCR anywhere while the other enrichers still
see every page.
`pdf:ocr-page-count` records how many pages tripped the verdict, whether or
not an engine ran.
+
+== Pages for inference (`inference`)
+
+`input`::
+What the parser releases to the
xref:configuration/inference.adoc#pages[inference bindings]:
+`["TEXT"]` by default. Add `PAGES` to render every page for a `PAGES` binding;
the render is
+the `ocr` render, made once per page for OCR and inference alike, and nothing
is rendered
+unless a `PAGES` binding runs for the request. Per request under `pdf-parser`
in
+`parse-context`, like the `ocr` block.
Every `ocr` option can be overridden per request through `parse-context` under
`pdf-parser`; the
xref:using-tika/server/index.adoc[server documentation] shows the `NO_OCR`
override as a preset.
diff --git
a/tika-app/src/test/java/org/apache/tika/cli/TikaCLIPageVectorsTest.java
b/tika-app/src/test/java/org/apache/tika/cli/TikaCLIPageVectorsTest.java
new file mode 100644
index 0000000000..89bcd2daac
--- /dev/null
+++ b/tika-app/src/test/java/org/apache/tika/cli/TikaCLIPageVectorsTest.java
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.cli;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.PrintStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.pdfbox.pdmodel.PDDocument;
+import org.apache.pdfbox.pdmodel.PDPage;
+import org.apache.pdfbox.pdmodel.PDPageContentStream;
+import org.apache.pdfbox.pdmodel.common.PDRectangle;
+import org.apache.pdfbox.pdmodel.font.PDType1Font;
+import org.apache.pdfbox.pdmodel.font.Standard14Fonts;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import org.apache.tika.http.TikaTestHttpServer;
+import org.apache.tika.inference.Chunk;
+import org.apache.tika.inference.ChunkSerializer;
+
+/**
+ * The consumer's path for pages: a PAGES binding and
pdf-parser.inference.input, and -J shows
+ * one vector per page on the PDF itself, each naming its page, from one
request.
+ */
+public class TikaCLIPageVectorsTest {
+
+ private static final String EMBEDDING_RESPONSE = "{\"data\":["
+ + "{\"index\":1,\"embedding\":[1.0,0.2,0.3]},"
+ + "{\"index\":0,\"embedding\":[0.0,0.2,0.3]}]}";
+
+ @TempDir
+ Path tmp;
+
+ @Test
+ public void testOneVectorPerPageOnThePdf() throws Exception {
+ Path pdf = tmp.resolve("two-pages.pdf");
+ writePdf(pdf, 2);
+ try (TikaTestHttpServer server = new TikaTestHttpServer()) {
+ server.enqueue(new TikaTestHttpServer.MockResponse(200,
EMBEDDING_RESPONSE));
+ Path config = tmp.resolve("tika-config.json");
+ Files.writeString(config, """
+ {
+ "engines": {
+ "clip": { "openai-embedding-engine": { "baseUrl":
"BASE_URL", "model": "clip" } }
+ },
+ "inference": [
+ { "id": "page-vectors", "engine": "clip", "input":
"PAGES",
+ "tasks": ["embed"], "maxChunks": 50 }
+ ],
+ "parsers": [
+ { "default-parser": {} },
+ { "pdf-parser": { "ocr": { "strategy": "NO_OCR",
"dpi": 20 },
+ "inference": { "input": ["PAGES"] }
} }
+ ]
+ }
+ """.replace("BASE_URL", server.url()));
+
+ JsonNode output = new ObjectMapper().readTree(
+ run("--config=" + config, "-J", pdf.toUri().toString()));
+
+ assertEquals(1, server.getRequestCount(), "both pages in one
request");
+ JsonNode request = new
ObjectMapper().readTree(server.takeRequest().body());
+ assertEquals(2, request.get("input").size());
+ assertTrue(request.get("input").get(0).get("image").asText()
+ .startsWith("data:image/png;base64,"));
+
+ assertEquals(1, output.size(), "the pages are the PDF's: no
embedded documents");
+ JsonNode root = output.get(0);
+ assertTrue(text(root, "tk:content").contains("This is page 2"),
+ "text extraction is untouched");
+ List<Chunk> chunks = ChunkSerializer.fromJson(text(root,
"tk:chunks"));
+ assertEquals(2, chunks.size());
+ Set<Integer> pages = new HashSet<>();
+ for (Chunk chunk : chunks) {
+ assertEquals(3, chunk.getVector().length);
+ pages.add(chunk.getLocators().getPaginated().get(0).getPage());
+ assertNull(chunk.getLocators().getEmbedded());
+ }
+ assertEquals(Set.of(1, 2), pages);
+ }
+ }
+
+ private static void writePdf(Path path, int pages) throws IOException {
+ try (PDDocument document = new PDDocument()) {
+ for (int i = 1; i <= pages; i++) {
+ PDPage page = new PDPage(PDRectangle.LETTER);
+ document.addPage(page);
+ try (PDPageContentStream content = new
PDPageContentStream(document, page)) {
+ content.beginText();
+ content.setFont(new
PDType1Font(Standard14Fonts.FontName.HELVETICA), 12);
+ content.newLineAtOffset(72, 700);
+ content.showText("This is page " + i + " of the test
document.");
+ content.endText();
+ }
+ }
+ document.save(path.toFile());
+ }
+ }
+
+ private static String text(JsonNode node, String field) {
+ JsonNode value = node.get(field);
+ assertNotNull(value, field);
+ return value.isArray() ? value.get(0).asText() : value.asText();
+ }
+
+ private static String run(String... args) throws Exception {
+ PrintStream stdout = System.out;
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ try {
+ System.setOut(new PrintStream(out, true, UTF_8.name()));
+ TikaCLI.main(args);
+ } finally {
+ System.setOut(stdout);
+ }
+ String json = out.toString(UTF_8);
+ assertTrue(json.startsWith("["), json);
+ return json;
+ }
+}
diff --git
a/tika-core/src/main/java/org/apache/tika/parser/AutoDetectParser.java
b/tika-core/src/main/java/org/apache/tika/parser/AutoDetectParser.java
index 9fe2a5fcb1..fb49df0194 100644
--- a/tika-core/src/main/java/org/apache/tika/parser/AutoDetectParser.java
+++ b/tika-core/src/main/java/org/apache/tika/parser/AutoDetectParser.java
@@ -209,7 +209,7 @@ public class AutoDetectParser extends CompositeParser {
}
} finally {
if (run != null) {
- run.exit(parent);
+ run.exit();
}
if (topLevel) {
try {
diff --git a/tika-core/src/main/java/org/apache/tika/parser/hook/ParseHook.java
b/tika-core/src/main/java/org/apache/tika/parser/hook/ParseHook.java
index 2165044b19..c6b5180823 100644
--- a/tika-core/src/main/java/org/apache/tika/parser/hook/ParseHook.java
+++ b/tika-core/src/main/java/org/apache/tika/parser/hook/ParseHook.java
@@ -54,6 +54,23 @@ public interface ParseHook {
ParseContext context) throws IOException, TikaException
{
}
+ /**
+ * Whether this hook wants the pages of the document being parsed rendered
as images of
+ * this type. Its parser asks once, before rendering anything for the
hooks.
+ */
+ default boolean wantsPages(MediaType renderType, Metadata document,
ParseContext context)
+ throws TikaException {
+ return false;
+ }
+
+ /**
+ * A page of {@code document} rendered by its own parser; {@code page} is
1-based,
+ * {@code bytes} and {@code parent} as in {@link #offer}.
+ */
+ default void offerPage(MediaType type, Metadata document, Metadata parent,
int page,
+ Path bytes, ParseContext context) throws
IOException, TikaException {
+ }
+
/** End of the top-level parse; {@code failed} when the parse threw. */
default void end(Metadata root, boolean failed, ParseContext context) {
}
diff --git
a/tika-core/src/main/java/org/apache/tika/parser/hook/ParseHooks.java
b/tika-core/src/main/java/org/apache/tika/parser/hook/ParseHooks.java
index 53492addd2..24b1d928c5 100644
--- a/tika-core/src/main/java/org/apache/tika/parser/hook/ParseHooks.java
+++ b/tika-core/src/main/java/org/apache/tika/parser/hook/ParseHooks.java
@@ -18,6 +18,9 @@ package org.apache.tika.parser.hook;
import java.io.IOException;
import java.nio.file.Path;
+import java.util.ArrayDeque;
+import java.util.Deque;
+import java.util.Iterator;
import java.util.List;
import org.slf4j.Logger;
@@ -41,19 +44,34 @@ public final class ParseHooks implements
TransientParseState {
private static final Logger LOG =
LoggerFactory.getLogger(ParseHooks.class);
- /** Marks a top-level parse in progress; tracks the document being parsed.
*/
+ /** Marks a top-level parse in progress; tracks the documents being
parsed, outermost last. */
public static final class Run implements TransientParseState {
- private Metadata current;
+ private final Deque<Metadata> open = new ArrayDeque<>();
/** Enters a document; returns the one whose parser embedded it, null
at the top. */
public Metadata enter(Metadata metadata) {
- Metadata parent = current;
- current = metadata;
+ Metadata parent = open.peek();
+ open.push(metadata);
return parent;
}
- public void exit(Metadata parent) {
- current = parent;
+ public void exit() {
+ open.pop();
+ }
+
+ /** The document whose parser is running; null between documents. */
+ public Metadata current() {
+ return open.peek();
+ }
+
+ /** The document the current one is embedded in; null at the top. */
+ public Metadata parent() {
+ Iterator<Metadata> it = open.iterator();
+ if (!it.hasNext()) {
+ return null;
+ }
+ it.next();
+ return it.hasNext() ? it.next() : null;
}
}
@@ -109,6 +127,41 @@ public final class ParseHooks implements
TransientParseState {
}
}
+ /** Whether any hook wants the current document's pages rendered as images
of this type. */
+ public boolean wantsPages(MediaType renderType, ParseContext context)
throws TikaException {
+ Run run = context.get(Run.class);
+ Metadata document = run == null ? null : run.current();
+ if (document == null) {
+ return false;
+ }
+ for (ParseHook hook : hooks) {
+ if (hook.wantsPages(renderType, document, context)) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /** A rendered page of the current document, from its own parser; never
fails it. */
+ public void offerPage(MediaType type, int page, Path bytes, ParseContext
context) {
+ Run run = context.get(Run.class);
+ Metadata document = run == null ? null : run.current();
+ if (document == null) {
+ return;
+ }
+ Metadata parent = run.parent();
+ for (ParseHook hook : hooks) {
+ try {
+ hook.offerPage(type, document, parent, page, bytes, context);
+ } catch (IOException | TikaException | RuntimeException e) {
+ LOG.warn("parse hook {} failed on page {}",
hook.getClass().getName(), page, e);
+ document.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING,
+ "parse hook " + hook.getClass().getSimpleName() + " on
page " + page
+ + ": " + e.getMessage());
+ }
+ }
+ }
+
public void end(Metadata root, boolean failed, ParseContext context) {
for (ParseHook hook : hooks) {
try {
diff --git
a/tika-core/src/main/java/org/apache/tika/parser/inference/InferenceDispatcher.java
b/tika-core/src/main/java/org/apache/tika/parser/inference/InferenceDispatcher.java
index a2d5147c49..c5a7e5c5a9 100644
---
a/tika-core/src/main/java/org/apache/tika/parser/inference/InferenceDispatcher.java
+++
b/tika-core/src/main/java/org/apache/tika/parser/inference/InferenceDispatcher.java
@@ -86,14 +86,38 @@ public final class InferenceDispatcher implements
ParseHook, TransientParseState
@Override
public boolean wants(MediaType type, Metadata metadata, ParseContext
context)
throws TikaException {
- InputKind kind = InputKind.of(type);
+ InputKind kind = kindOf(type, metadata);
return kind != null && wants(kind, type, context);
}
@Override
public void offer(MediaType type, Metadata metadata, Metadata parent, Path
bytes,
ParseContext context) throws IOException, TikaException {
- offer(InputKind.of(type), type, metadata, parent, bytes, context);
+ InputKind kind = kindOf(type, metadata);
+ if (kind != null) {
+ offer(kind, type, metadata, parent, bytes, context);
+ }
+ }
+
+ @Override
+ public boolean wantsPages(MediaType renderType, Metadata document,
ParseContext context)
+ throws TikaException {
+ return wants(InputKind.PAGES, renderType, context);
+ }
+
+ @Override
+ public void offerPage(MediaType type, Metadata document, Metadata parent,
int page,
+ Path bytes, ParseContext context) throws
IOException, TikaException {
+ offer(InputKind.PAGES, type, document, parent, bytes, page, context);
+ }
+
+ /** A page render emitted as an embedded document is a page, not an image:
PAGES only. */
+ private static InputKind kindOf(MediaType type, Metadata metadata) {
+ if (TikaCoreProperties.EmbeddedResourceType.RENDERING.name()
+
.equals(metadata.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE))) {
+ return null;
+ }
+ return InputKind.of(type);
}
/** Runs the buffered units, or drops them when the parse failed: no
engine call for a document nobody gets. */
@@ -123,6 +147,12 @@ public final class InferenceDispatcher implements
ParseHook, TransientParseState
*/
public void offer(InputKind kind, MediaType type, Metadata target,
Metadata parent,
Path source, ParseContext context) throws IOException,
TikaException {
+ offer(kind, type, target, parent, source, -1, context);
+ }
+
+ private void offer(InputKind kind, MediaType type, Metadata target,
Metadata parent,
+ Path source, int page, ParseContext context)
+ throws IOException, TikaException {
State state = state(context);
long size = Files.size(source);
InferenceUnit unit = null;
@@ -144,7 +174,7 @@ public final class InferenceDispatcher implements
ParseHook, TransientParseState
if (unit == null) {
Path copy = state.tmp.createTempFile();
Files.copy(source, copy, StandardCopyOption.REPLACE_EXISTING);
- unit = new InferenceUnit(kind, type, target, parent, copy);
+ unit = new InferenceUnit(kind, type, target, parent, copy,
page);
}
units.add(unit);
}
diff --git
a/tika-core/src/main/java/org/apache/tika/parser/inference/InferenceUnit.java
b/tika-core/src/main/java/org/apache/tika/parser/inference/InferenceUnit.java
index 96d4b085b5..0ab0a76934 100644
---
a/tika-core/src/main/java/org/apache/tika/parser/inference/InferenceUnit.java
+++
b/tika-core/src/main/java/org/apache/tika/parser/inference/InferenceUnit.java
@@ -41,16 +41,24 @@ public final class InferenceUnit {
private final String parentIdPath;
private final Path path;
private final long size;
+ private final int page;
public InferenceUnit(InputKind kind, MediaType type, Metadata target,
Metadata parent,
Path path) throws IOException {
+ this(kind, type, target, parent, path, -1);
+ }
+
+ /** A unit that is one page of its target: {@code page} is 1-based. */
+ public InferenceUnit(InputKind kind, MediaType type, Metadata target,
Metadata parent,
+ Path path, int page) throws IOException {
this(kind, type, target, parent,
target.get(TikaCoreProperties.EMBEDDED_ID_PATH),
parent == null ? null :
parent.get(TikaCoreProperties.EMBEDDED_ID_PATH), path,
- Files.size(path));
+ Files.size(path), page);
}
private InferenceUnit(InputKind kind, MediaType type, Metadata target,
Metadata parent,
- String targetIdPath, String parentIdPath, Path path,
long size) {
+ String targetIdPath, String parentIdPath, Path path,
long size,
+ int page) {
this.kind = kind;
this.type = type;
this.target = target;
@@ -59,12 +67,18 @@ public final class InferenceUnit {
this.parentIdPath = parentIdPath;
this.path = path;
this.size = size;
+ this.page = page;
}
/** The same unit aimed at the metadata objects that are still read. */
InferenceUnit retargeted(Metadata target, Metadata parent) {
return new InferenceUnit(kind, type, target, parent, targetIdPath,
parentIdPath, path,
- size);
+ size, page);
+ }
+
+ /** The 1-based page this unit renders, for {@link InputKind#PAGES}; -1
otherwise. */
+ public int getPage() {
+ return page;
}
public InputKind getKind() {
diff --git
a/tika-core/src/test/java/org/apache/tika/parser/hook/ParseHooksTest.java
b/tika-core/src/test/java/org/apache/tika/parser/hook/ParseHooksTest.java
index 046e66fd17..52a0d6491d 100644
--- a/tika-core/src/test/java/org/apache/tika/parser/hook/ParseHooksTest.java
+++ b/tika-core/src/test/java/org/apache/tika/parser/hook/ParseHooksTest.java
@@ -58,6 +58,7 @@ public class ParseHooksTest {
private static final MediaType PNG = MediaType.image("png");
private static final MediaType CONTAINER =
MediaType.application("x-test-container");
+ private static final MediaType PAGED =
MediaType.application("x-test-paged");
/** Detection by the declared type: these tests are about the seam, not
the detector. */
private static final Detector DECLARED = (tis, metadata, ctx) ->
@@ -67,7 +68,11 @@ public class ParseHooksTest {
final List<String> events = new ArrayList<>();
final List<Metadata> offeredParents = new ArrayList<>();
final List<byte[]> offeredBytes = new ArrayList<>();
+ final List<Metadata> pageDocuments = new ArrayList<>();
+ final List<Metadata> pageParents = new ArrayList<>();
+ final List<String> pageBytes = new ArrayList<>();
Set<MediaType> wanted = Set.of(PNG);
+ boolean wantsPages;
boolean failOffer;
boolean failStart;
@@ -95,12 +100,58 @@ public class ParseHooksTest {
offeredBytes.add(Files.readAllBytes(bytes));
}
+ @Override
+ public boolean wantsPages(MediaType renderType, Metadata document,
ParseContext context) {
+ events.add("wantsPages " + renderType);
+ return wantsPages;
+ }
+
+ @Override
+ public void offerPage(MediaType type, Metadata document, Metadata
parent, int page,
+ Path bytes, ParseContext context) throws
IOException {
+ if (failOffer) {
+ throw new IOException("no page");
+ }
+ events.add("page " + page);
+ pageDocuments.add(document);
+ pageParents.add(parent);
+ pageBytes.add(Files.readString(bytes));
+ }
+
@Override
public void end(Metadata root, boolean failed, ParseContext context) {
events.add(failed ? "end failed" : "end");
}
}
+ /** A parser that renders two pages for the hooks, the way the PDF parser
does. */
+ static final class PagedParser implements Parser {
+ @Override
+ public Set<MediaType> getSupportedTypes(ParseContext context) {
+ return Collections.singleton(PAGED);
+ }
+
+ @Override
+ public void parse(TikaInputStream tis, ContentHandler handler,
Metadata metadata,
+ ParseContext context) throws IOException,
SAXException, TikaException {
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(handler,
metadata, context);
+ xhtml.startDocument();
+ ParseHooks hooks = context.get(ParseHooks.class);
+ if (hooks != null && hooks.wantsPages(PNG, context)) {
+ for (int page = 1; page <= 2; page++) {
+ Path render = Files.createTempFile("page", ".png");
+ try {
+ Files.writeString(render, "PAGE-" + page);
+ hooks.offerPage(PNG, page, render, context);
+ } finally {
+ Files.delete(render);
+ }
+ }
+ }
+ xhtml.endDocument();
+ }
+ }
+
static final class TypedParser implements Parser {
private final MediaType type;
private final boolean fail;
@@ -129,6 +180,16 @@ public class ParseHooksTest {
/** A container whose one child is a PNG, parsed through the embedded
extractor. */
static final class ContainerParser implements Parser {
+ private final MediaType childType;
+
+ ContainerParser() {
+ this(PNG);
+ }
+
+ ContainerParser(MediaType childType) {
+ this.childType = childType;
+ }
+
@Override
public Set<MediaType> getSupportedTypes(ParseContext context) {
return Collections.singleton(CONTAINER);
@@ -140,7 +201,7 @@ public class ParseHooksTest {
XHTMLContentHandler xhtml = new XHTMLContentHandler(handler,
metadata, context);
xhtml.startDocument();
Metadata child = new Metadata();
- child.set(HttpHeaders.CONTENT_TYPE, PNG.toString());
+ child.set(HttpHeaders.CONTENT_TYPE, childType.toString());
child.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE, "INLINE");
try (TikaInputStream childStream =
TikaInputStream.get("PNG-BYTES".getBytes(UTF_8))) {
EmbeddedDocumentUtil.getEmbeddedDocumentExtractor(context)
@@ -250,6 +311,52 @@ public class ParseHooksTest {
assertNull(context.get(ParseHooks.Run.class));
}
+ @Test
+ public void testPagesAreOfferedWithTheirDocumentAndItsParent() throws
Exception {
+ RecordingHook hook = new RecordingHook();
+ hook.wantsPages = true;
+ hook.wanted = Set.of();
+ Metadata root = typed(CONTAINER);
+ ParseContext context = new ParseContext();
+ try (TikaInputStream tis =
TikaInputStream.get("CONTAINER".getBytes(UTF_8))) {
+ parser(hook, new ContainerParser(PAGED), new PagedParser())
+ .parse(tis, new DefaultHandler(), root, context);
+ }
+ assertEquals(List.of("start", "wantsPages image/png", "page 1", "page
2", "end"),
+ hook.events);
+ assertEquals(List.of("PAGE-1", "PAGE-2"), hook.pageBytes);
+ assertEquals(PAGED.toString(),
hook.pageDocuments.get(0).get(HttpHeaders.CONTENT_TYPE),
+ "the page belongs to the paged child");
+ assertSame(hook.pageDocuments.get(0), hook.pageDocuments.get(1));
+ assertSame(root, hook.pageParents.get(0), "and the child's parent
rides along");
+ }
+
+ @Test
+ public void testNoPagesWhenNoHookWantsThem() throws Exception {
+ RecordingHook hook = new RecordingHook();
+ try (TikaInputStream tis =
TikaInputStream.get("PAGED".getBytes(UTF_8))) {
+ parser(hook, new PagedParser()).parse(tis, new DefaultHandler(),
typed(PAGED),
+ new ParseContext());
+ }
+ assertEquals(List.of("start", "wantsPages image/png", "end"),
hook.events);
+ }
+
+ @Test
+ public void testPageOfferFailureIsAWarningOnTheDocument() throws Exception
{
+ RecordingHook hook = new RecordingHook();
+ hook.wantsPages = true;
+ hook.failOffer = true;
+ Metadata metadata = typed(PAGED);
+ try (TikaInputStream tis =
TikaInputStream.get("PAGED".getBytes(UTF_8))) {
+ parser(hook, new PagedParser()).parse(tis, new DefaultHandler(),
metadata,
+ new ParseContext());
+ }
+ assertEquals(List.of("start", "wantsPages image/png", "end"),
hook.events);
+ String[] warnings =
metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING);
+ assertEquals(2, warnings.length);
+ assertTrue(warnings[0].contains("on page 1: no page"), warnings[0]);
+ }
+
@Test
public void testNoHooksNoTrace() throws Exception {
AutoDetectParser adp = new AutoDetectParser(DECLARED, new
TypedParser(PNG, false));
diff --git
a/tika-core/src/test/java/org/apache/tika/parser/inference/InferenceDispatcherTest.java
b/tika-core/src/test/java/org/apache/tika/parser/inference/InferenceDispatcherTest.java
index 36e68b03b3..179ccb452f 100644
---
a/tika-core/src/test/java/org/apache/tika/parser/inference/InferenceDispatcherTest.java
+++
b/tika-core/src/test/java/org/apache/tika/parser/inference/InferenceDispatcherTest.java
@@ -347,6 +347,51 @@ public class InferenceDispatcherTest {
assertEquals(2, handler.getMetadataList().size());
}
+ @Test
+ public void testPagesComeFromTheRendererOnly() throws Exception {
+ RecordingTask pageTask = new RecordingTask();
+ RecordingTask imageTask = new RecordingTask();
+ RecordingEngine engine = new RecordingEngine();
+ InferenceDispatcher dispatcher = new InferenceDispatcher(List.of(
+ new InferenceDispatcher.Bound(binding("pages",
InputKind.PAGES, null, 1),
+ engine, List.of(pageTask)),
+ new InferenceDispatcher.Bound(binding("images",
InputKind.IMAGES, null, -1),
+ engine, List.of(imageTask))));
+ ParseContext context = new ParseContext();
+ Metadata pdf = new Metadata();
+ assertTrue(dispatcher.wantsPages(PNG, pdf, context));
+ Metadata rendering = new Metadata();
+ rendering.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE, "RENDERING");
+ assertFalse(dispatcher.wants(PNG, rendering, context),
+ "a render emitted as an embedded document is a page, not an
image");
+ Metadata inline = new Metadata();
+ inline.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE, "INLINE");
+ assertTrue(dispatcher.wants(PNG, inline, context));
+
+ dispatcher.offerPage(PNG, pdf, null, 1, file("p1"), context);
+ dispatcher.offerPage(PNG, pdf, null, 2, file("p2"), context);
+ dispatcher.offer(PNG, rendering, pdf, file("r"), context);
+ Metadata root = new Metadata();
+ dispatcher.flush(root, context);
+
+ assertEquals(1, pageTask.runs.size());
+ List<InferenceUnit> units = pageTask.runs.get(0);
+ assertEquals(1, units.size(), "maxChunks 1 is one page");
+ assertEquals(InputKind.PAGES, units.get(0).getKind());
+ assertEquals(1, units.get(0).getPage());
+ assertSame(pdf, units.get(0).getTarget());
+ assertEquals("p1", new String(pageTask.bytes.get(0).get(0), UTF_8));
+ assertTrue(imageTask.runs.isEmpty(), "the rendering child reached no
IMAGES binding");
+ assertTrue(root.get(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING)
+ .contains("pages over maxChunks: skipped 1 units"));
+ }
+
+ @Test
+ public void testNoPagesBindingWantsNoPages() throws Exception {
+ InferenceDispatcher dispatcher = pngDispatcher(new RecordingTask());
+ assertFalse(dispatcher.wantsPages(PNG, new Metadata(), new
ParseContext()));
+ }
+
@Test
public void testFailedParseRunsNothingAndCleansUp() throws Exception {
RecordingTask task = new RecordingTask();
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/EmbedTask.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/EmbedTask.java
index 333dde4a22..9fab518a83 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/EmbedTask.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/main/java/org/apache/tika/inference/EmbedTask.java
@@ -37,7 +37,7 @@ import org.apache.tika.parser.inference.InputKind;
/**
* The {@code embed} task: one vector chunk per unit, written where the unit's
chunks belong
* (its parent for an inline picture or a page render, itself otherwise), with
the page
- * locator when the unit has a page. Units go to the engine in batches of its
size.
+ * locator when the unit is a page or has one. Units go to the engine in
batches of its size.
*/
@TikaComponent(name = "embed", spi = false)
public class EmbedTask implements InferenceTask {
@@ -48,8 +48,8 @@ public class EmbedTask implements InferenceTask {
throw new TikaConfigException("task \"embed\" needs an embedding
engine; \""
+ binding.getEngine() + "\" is not one");
}
- if (binding.getInput() != InputKind.IMAGES) {
- throw new TikaConfigException("task \"embed\" takes IMAGES;
binding \""
+ if (binding.getInput() != InputKind.IMAGES && binding.getInput() !=
InputKind.PAGES) {
+ throw new TikaConfigException("task \"embed\" takes IMAGES or
PAGES; binding \""
+ binding.getId() + "\" is on " + binding.getInput());
}
}
@@ -97,7 +97,9 @@ public class EmbedTask implements InferenceTask {
InferenceUnit unit = batch.get(i);
Locators locators = new Locators();
String page = unit.getTarget().get(TikaPagedText.PAGE_NUMBER);
- if (page != null) {
+ if (unit.getPage() > 0) {
+ locators.addPaginated(new PaginatedLocator(unit.getPage()));
+ } else if (page != null) {
locators.addPaginated(new
PaginatedLocator(Integer.parseInt(page)));
}
Chunk chunk = new Chunk(null, locators);
diff --git
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/EmbedTaskTest.java
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/EmbedTaskTest.java
index 49d3015748..0980bd4edd 100644
---
a/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/EmbedTaskTest.java
+++
b/tika-parsers/tika-parsers-ml/tika-inference/src/test/java/org/apache/tika/inference/EmbedTaskTest.java
@@ -135,6 +135,32 @@ public class EmbedTaskTest {
unit.getTarget().get(TikaCoreProperties.TIKA_CHUNKS)).size());
}
+ /** A page unit lands on its own document with the page as its locator. */
+ @Test
+ public void testPageUnitsCarryThePage() throws Exception {
+ server.enqueue(new TikaTestHttpServer.MockResponse(200, response(0,
1)));
+ Metadata pdf = new Metadata();
+ pdf.set(TikaCoreProperties.EMBEDDED_ID_PATH, "/1");
+ pdf.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE, "ATTACHMENT");
+ Path p1 = Files.createTempFile(tmp, "p1", ".png");
+ Path p2 = Files.createTempFile(tmp, "p2", ".png");
+ Files.write(p1, "one".getBytes(StandardCharsets.UTF_8));
+ Files.write(p2, "two".getBytes(StandardCharsets.UTF_8));
+ InferenceBinding pages = new InferenceBinding("pages", "clip",
InputKind.PAGES,
+ List.of("embed"), null, null, -1, -1, true);
+ EmbedTask task = new EmbedTask();
+ task.validate(pages, engine);
+ task.run(pages, List.of(
+ new InferenceUnit(InputKind.PAGES, MediaType.image("png"),
pdf, new Metadata(), p1, 3),
+ new InferenceUnit(InputKind.PAGES, MediaType.image("png"),
pdf, new Metadata(), p2, 4)),
+ engine, new ParseContext());
+ List<Chunk> chunks =
ChunkSerializer.fromJson(pdf.get(TikaCoreProperties.TIKA_CHUNKS));
+ assertEquals(2, chunks.size());
+ assertEquals(3,
chunks.get(0).getLocators().getPaginated().get(0).getPage());
+ assertEquals(4,
chunks.get(1).getLocators().getPaginated().get(0).getPage());
+ assertNull(chunks.get(0).getLocators().getEmbedded(), "the pdf keeps
its own pages");
+ }
+
@Test
public void testValidateRejectsANonEmbeddingEngine() {
assertThrows(TikaConfigException.class,
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/pdf-parser-full.json
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/pdf-parser-full.json
index e48cd08540..55650f2a40 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/pdf-parser-full.json
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-integration-tests/src/test/resources/config-examples/pdf-parser-full.json
@@ -48,6 +48,9 @@
// How to render page images; NONE renders nothing.
// Options: NONE, RAW_IMAGES, RENDER_PAGES_BEFORE_PARSE,
RENDER_PAGES_AT_PAGE_END
"imageStrategy": "NONE",
+ // What the parser releases to the inference bindings (see the
inference page):
+ // TEXT is the extracted text; PAGES renders every page for a PAGES
binding.
+ "inference": { "input": ["TEXT"] },
// Max incremental updates to parse when parseIncrementalUpdates is
true.
"maxIncrementalUpdates": 10,
// Max memory to load a PDF before buffering to a temp file (default
512MB).
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/AbstractPDF2XHTML.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/AbstractPDF2XHTML.java
index f8e41ba0c2..961f3fc967 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/AbstractPDF2XHTML.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/AbstractPDF2XHTML.java
@@ -121,6 +121,7 @@ import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.Parser;
import org.apache.tika.parser.enricher.CompositeContentEnricher;
import org.apache.tika.parser.enricher.ContentEnrichers;
+import org.apache.tika.parser.hook.ParseHooks;
import org.apache.tika.parser.pdf.updates.IncrementalUpdateRecord;
import org.apache.tika.parser.pdf.updates.IsIncrementalUpdate;
import org.apache.tika.parser.pdf.updates.StartXRefOffset;
@@ -177,6 +178,8 @@ class AbstractPDF2XHTML extends PDFTextStripper {
private final Parser pageAnnotators;
final MediaType ocrImageMediaType;
private PageText pageDecision = PageText.UNDECIDED;
+ /** What the OCR step did with the current page; SKIPPED until it runs. */
+ private PageOcr currentPageOcr = PageOcr.SKIPPED;
// Text held back from an OCR_WANTED page; replayed if OCR does not run.
private List<PageTextBuffer.SaxEvent> pendingText =
Collections.emptyList();
final ParseContext context;
@@ -185,6 +188,12 @@ class AbstractPDF2XHTML extends PDFTextStripper {
final PDFParserConfig config;
final Renderer renderer;
final CompositeContentEnricher contentEnrichers;
+ private final ParseHooks hooks;
+ /** Pages are rendered for inference: the config says PAGES and a hook
wants them. */
+ private final boolean pagesForInference;
+ /** The current page's render, made once for OCR, annotators and inference
alike. */
+ private RenderResult pageRender;
+ private TemporaryResources pageRenderResources;
/**
* Format used for signature dates
* TODO Make this thread-safe
@@ -258,6 +267,16 @@ class AbstractPDF2XHTML extends PDFTextStripper {
this.renderer = renderer;
this.contentEnrichers = contentEnrichers;
embeddedDocumentExtractor =
EmbeddedDocumentUtil.getEmbeddedDocumentExtractor(context);
+ this.hooks = context.get(ParseHooks.class);
+ boolean wantsPages = false;
+ if (hooks != null &&
config.getInference().getInput().contains(InferenceConfig.Input.PAGES)) {
+ try {
+ wantsPages = hooks.wantsPages(ocrImageMediaType, context);
+ } catch (TikaException e) {
+ throw new IOException(e);
+ }
+ }
+ this.pagesForInference = wantsPages;
}
private static void addNonNullAttribute(String name, String value,
AttributesImpl attributes) {
@@ -664,6 +683,13 @@ class AbstractPDF2XHTML extends PDFTextStripper {
PageOcr doOCROnCurrentPage(PDPage pdPage, OcrConfig.Strategy ocrStrategy)
throws IOException, TikaException, SAXException {
+ PageOcr result = dispatchOcr(pdPage, ocrStrategy);
+ currentPageOcr = result;
+ return result;
+ }
+
+ private PageOcr dispatchOcr(PDPage pdPage, OcrConfig.Strategy ocrStrategy)
+ throws IOException, TikaException, SAXException {
if (ocrStrategy.equals(NO_OCR)) {
return PageOcr.SKIPPED;
}
@@ -702,30 +728,29 @@ class AbstractPDF2XHTML extends PDFTextStripper {
}
}
- /** Renders the page and runs the engine on it; true if the engine wrote
text. */
+ /** Runs the engine on the page's render; true if the engine wrote text. */
private boolean enrichCurrentPage(PDPage pdPage, Parser engine)
throws IOException, TikaException, SAXException {
- try (TemporaryResources tmp = new TemporaryResources()) {
- try (RenderResult renderResult = renderCurrentPage(pdPage, tmp)) {
- Metadata renderMetadata = renderResult.getMetadata();
- TextCounter counter = new TextCounter(xhtml);
- try (TikaInputStream tis = renderResult.getInputStream()) {
- renderMetadata.set(HttpHeaders.CONTENT_TYPE,
ocrImageMediaType.toString());
- engine.parse(tis,
- new EmbeddedContentHandler(new
BodyContentHandler(counter)),
- renderMetadata, context);
- }
- // Propagate enrichment metadata added by the OCR parser (e.g.
tk:chunks
- // from image embedding parsers) back to the parent document
so it isn't
- // silently discarded when the renderMetadata goes out of
scope.
- String renderChunks =
renderMetadata.get(TikaCoreProperties.TIKA_CHUNKS);
- if (renderChunks != null) {
- metadata.set(TikaCoreProperties.TIKA_CHUNKS,
-
mergeChunkArrays(metadata.get(TikaCoreProperties.TIKA_CHUNKS),
- renderChunks));
- }
- return counter.sawText();
- }
+ try {
+ RenderResult renderResult = currentPageRender(pdPage);
+ Metadata renderMetadata = renderResult.getMetadata();
+ TextCounter counter = new TextCounter(xhtml);
+ try (TikaInputStream tis = renderResult.getInputStream()) {
+ renderMetadata.set(HttpHeaders.CONTENT_TYPE,
ocrImageMediaType.toString());
+ engine.parse(tis,
+ new EmbeddedContentHandler(new
BodyContentHandler(counter)),
+ renderMetadata, context);
+ }
+ // Propagate enrichment metadata added by the OCR parser (e.g.
tk:chunks
+ // from image embedding parsers) back to the parent document so it
isn't
+ // silently discarded when the renderMetadata goes out of scope.
+ String renderChunks =
renderMetadata.get(TikaCoreProperties.TIKA_CHUNKS);
+ if (renderChunks != null) {
+ metadata.set(TikaCoreProperties.TIKA_CHUNKS,
+
mergeChunkArrays(metadata.get(TikaCoreProperties.TIKA_CHUNKS),
+ renderChunks));
+ }
+ return counter.sawText();
} catch (IOException e) {
handleCatchableIOE(e);
} catch (TikaTimeoutException e) {
@@ -736,6 +761,41 @@ class AbstractPDF2XHTML extends PDFTextStripper {
return false;
}
+ /** The page's render, made on first use and closed at page end. */
+ private RenderResult currentPageRender(PDPage pdPage) throws IOException,
TikaException {
+ if (pageRender == null) {
+ pageRenderResources = new TemporaryResources();
+ pageRender = renderCurrentPage(pdPage, pageRenderResources);
+ }
+ return pageRender;
+ }
+
+ private void closePageRender() {
+ RenderResult render = pageRender;
+ TemporaryResources resources = pageRenderResources;
+ pageRender = null;
+ pageRenderResources = null;
+ try (resources; render) {
+ // closing deletes the render file
+ } catch (IOException e) {
+ EmbeddedDocumentUtil.recordEmbeddedStreamException(e, metadata,
context);
+ }
+ }
+
+ /** Hands the page's render to the hooks when a PAGES binding wants it. */
+ private void offerCurrentPageToInference(PDPage pdPage) throws
IOException, TikaException {
+ if (!pagesForInference) {
+ return;
+ }
+ RenderResult render = currentPageRender(pdPage);
+ if (render.getStatus() != RenderResult.STATUS.SUCCESS) {
+ return;
+ }
+ try (TikaInputStream tis = render.getInputStream()) {
+ hooks.offerPage(ocrImageMediaType, getCurrentPageNo(),
tis.getPath(), context);
+ }
+ }
+
/**
* Appends two serialized tk:chunks JSON arrays without a JSON dependency,
so each
* OCR'd page's chunks accumulate on the parent instead of first-page-wins.
@@ -913,18 +973,18 @@ class AbstractPDF2XHTML extends PDFTextStripper {
for (PDAnnotation annotation : page.getAnnotations()) {
processPageAnnotation(annotation);
}
- PageOcr ocr = PageOcr.SKIPPED;
if (config.getOcr().getStrategy() == OCR_AND_TEXT_EXTRACTION) {
- ocr = doOCROnCurrentPage(page, OCR_AND_TEXT_EXTRACTION);
+ doOCROnCurrentPage(page, OCR_AND_TEXT_EXTRACTION);
} else if (pageDecision == PageText.OCR_WANTED) {
- ocr = doOCROnCurrentPage(page, AUTO);
- if (ocr != PageOcr.TEXT) {
+ if (doOCROnCurrentPage(page, AUTO) != PageOcr.TEXT) {
replay(pendingText);
}
}
- if (ocr == PageOcr.SKIPPED) {
+ // the engine composite already ran the annotators on a page it
OCR'd
+ if (currentPageOcr == PageOcr.SKIPPED) {
annotateCurrentPage(page);
}
+ offerCurrentPageToInference(page);
PDPageAdditionalActions pageActions = page.getActions();
if (pageActions != null) {
@@ -937,6 +997,8 @@ class AbstractPDF2XHTML extends PDFTextStripper {
} catch (IOException e) {
handleCatchableIOE(e);
} finally {
+ closePageRender();
+ currentPageOcr = PageOcr.SKIPPED;
totalCharsPerPage = 0;
unmappedUnicodeCharsPerPage = 0;
pendingText = Collections.emptyList();
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/InferenceConfig.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/InferenceConfig.java
new file mode 100644
index 0000000000..1997615b0c
--- /dev/null
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/InferenceConfig.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.pdf;
+
+import java.io.Serializable;
+import java.util.List;
+
+/** What the PDF parser releases to the inference bindings; per request like
every PDF setting. */
+public class InferenceConfig implements Serializable {
+
+ private static final long serialVersionUID = 1L;
+
+ /** {@code TEXT}: the extracted text. {@code PAGES}: every page rendered,
when a binding wants pages. */
+ public enum Input {
+ TEXT, PAGES
+ }
+
+ private List<Input> input = List.of(Input.TEXT);
+
+ public List<Input> getInput() {
+ return input;
+ }
+
+ public void setInput(List<Input> input) {
+ this.input = input == null ? List.of() : List.copyOf(input);
+ }
+}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/OCR2XHTML.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/OCR2XHTML.java
index ae23a5e6e2..c3de03490b 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/OCR2XHTML.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/OCR2XHTML.java
@@ -97,9 +97,8 @@ class OCR2XHTML extends AbstractPDF2XHTML {
public void processPage(PDPage pdPage) throws IOException {
try {
startPage(pdPage);
- if (doOCROnCurrentPage(pdPage, OcrConfig.Strategy.OCR_ONLY) ==
PageOcr.SKIPPED) {
- annotateCurrentPage(pdPage);
- }
+ doOCROnCurrentPage(pdPage, OcrConfig.Strategy.OCR_ONLY);
+ // endPage annotates the page when no engine ran, and offers it to
inference
endPage(pdPage);
} catch (TikaException | SAXException e) {
throw new IOException(e);
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParserConfig.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParserConfig.java
index b83f449cad..239f19196e 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParserConfig.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/main/java/org/apache/tika/parser/pdf/PDFParserConfig.java
@@ -119,6 +119,8 @@ public class PDFParserConfig implements Serializable {
private OcrConfig ocr = new OcrConfig();
+ private InferenceConfig inference = new InferenceConfig();
+
/**
* Should the entire document be rendered?
*/
@@ -522,6 +524,15 @@ public class PDFParserConfig implements Serializable {
this.ocr = ocr;
}
+ /** What this parser releases to the inference bindings. */
+ public InferenceConfig getInference() {
+ return inference;
+ }
+
+ public void setInference(InferenceConfig inference) {
+ this.inference = inference == null ? new InferenceConfig() : inference;
+ }
+
// OCR settings are configured through the nested OcrConfig
(getOcr()/setOcr()).
// The flat ocr* convenience accessors (getOcrStrategy/setOcrDPI/...) were
removed in
// 4.x so that "ocr" is the single JSON spelling; use
getOcr().setStrategy(...) etc.
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/test/java/org/apache/tika/parser/pdf/PDFParserTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/test/java/org/apache/tika/parser/pdf/PDFParserTest.java
index 2852ce22cd..56333f89bd 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/test/java/org/apache/tika/parser/pdf/PDFParserTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/test/java/org/apache/tika/parser/pdf/PDFParserTest.java
@@ -21,13 +21,16 @@ import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertSame;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
import java.nio.file.Path;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
@@ -81,6 +84,9 @@ import
org.apache.tika.parser.enricher.CompositeContentEnricher;
import org.apache.tika.parser.enricher.ContentEnricher;
import org.apache.tika.parser.enricher.ContentEnrichers;
import org.apache.tika.parser.enricher.TextRecognizer;
+import org.apache.tika.parser.hook.ParseHook;
+import org.apache.tika.parser.hook.ParseHooks;
+import org.apache.tika.renderer.RenderingTracker;
import org.apache.tika.sax.BodyContentHandler;
import org.apache.tika.sax.ContentHandlerDecorator;
import org.apache.tika.sax.ToXMLContentHandler;
@@ -1238,6 +1244,8 @@ public class PDFParserTest extends TikaTest {
assertEquals(true, pdfParserConfig.isExtractInlineImages());
assertEquals(false, pdfParserConfig.isExtractUniqueInlineImagesOnly());
assertEquals(314, pdfParserConfig.getOcr().getDpi());
+ assertEquals(List.of(InferenceConfig.Input.PAGES),
+ pdfParserConfig.getInference().getInput());
assertEquals(2.1f, pdfParserConfig.getOcr().getImageQuality(), .01f);
assertEquals(OcrConfig.ImageFormat.JPEG,
pdfParserConfig.getOcr().getImageFormat());
assertEquals(524288000, pdfParserConfig.getMaxMainMemoryBytes());
@@ -1817,6 +1825,127 @@ public class PDFParserTest extends TikaTest {
assertContainsCount("MOCK_OCR_CONTENT", xml, 1);
}
+ /** OCR_ONLY with the budget spent still annotates a rendered page once. */
+ @Test
+ public void testOcrOnlyBudgetAnnotatesOnce() throws Exception {
+ PageEnricher recognizer = new PageEnricher("MOCK_OCR_CONTENT", true);
+ PageAnnotator annotator = new PageAnnotator();
+ CompositeContentEnricher enrichers =
+ new CompositeContentEnricher(List.of(recognizer, annotator));
+ PDFParser parser = new PDFParser();
+ parser.setContentEnrichers(enrichers);
+ PDFParserConfig config = new PDFParserConfig();
+ config.getOcr().setStrategy(OcrConfig.Strategy.OCR_ONLY);
+ config.getOcr().setMaxPagesToOcr(1);
+
config.setImageStrategy(PDFParserConfig.IMAGE_STRATEGY.RENDER_PAGES_AT_PAGE_END);
+ config.getOcr().setDpi(20);
+ ParseContext context = new ParseContext();
+ context.set(PDFParserConfig.class, config);
+ Metadata metadata = new Metadata();
+
+ String xml = parsePdfToXml(parser, "testPDF_bookmarks.pdf", metadata,
context);
+
+ assertEquals(1, recognizer.calls);
+ assertEquals(2, annotator.calls);
+ assertContainsCount("MOCK_OCR_CONTENT", xml, 1);
+ }
+
+ /** Records what the parse hooks are offered; wants pages when told to. */
+ private static class PageHook implements ParseHook {
+ final List<Integer> pages = new ArrayList<>();
+ final List<Long> sizes = new ArrayList<>();
+ Metadata document;
+ boolean wantsPages;
+ int wantsPagesCalls = 0;
+
+ @Override
+ public boolean wantsPages(MediaType renderType, Metadata document,
ParseContext context) {
+ wantsPagesCalls++;
+ return wantsPages;
+ }
+
+ @Override
+ public void offerPage(MediaType type, Metadata document, Metadata
parent, int page,
+ Path bytes, ParseContext context) throws
IOException {
+ assertEquals("image/png", type.toString());
+ pages.add(page);
+ sizes.add(Files.size(bytes));
+ this.document = document;
+ }
+ }
+
+ /** The PDF parser under the auto-detect parser, which seeds the hooks. */
+ private static AutoDetectParser hooked(PDFParser parser, PageHook hook) {
+ AutoDetectParser adp = new AutoDetectParser(parser);
+ adp.setParseHooks(new ParseHooks(List.of(hook)));
+ return adp;
+ }
+
+ private static PDFParserConfig pagesForInference(OcrConfig.Strategy
strategy) {
+ PDFParserConfig config = new PDFParserConfig();
+ config.getOcr().setStrategy(strategy);
+ config.getOcr().setDpi(20);
+ config.getInference().setInput(List.of(InferenceConfig.Input.PAGES));
+ return config;
+ }
+
+ /** One render per page feeds OCR and inference alike; every page is
offered once. */
+ @Test
+ public void testPagesOfferedOncePerPageFromOneRender() throws Exception {
+ for (OcrConfig.Strategy strategy :
List.of(OcrConfig.Strategy.OCR_AND_TEXT_EXTRACTION,
+ OcrConfig.Strategy.OCR_ONLY, OcrConfig.Strategy.NO_OCR)) {
+ PageEnricher recognizer = new PageEnricher("MOCK_OCR_CONTENT",
true);
+ PDFParser parser = new PDFParser();
+ parser.setContentEnrichers(new
CompositeContentEnricher(List.of(recognizer)));
+ PageHook hook = new PageHook();
+ hook.wantsPages = true;
+ ParseContext context = new ParseContext();
+ context.set(PDFParserConfig.class, pagesForInference(strategy));
+ RenderingTracker tracker = new RenderingTracker();
+ context.set(RenderingTracker.class, tracker);
+ Metadata metadata = new Metadata();
+ try (TikaInputStream tis =
getResourceAsStream("/test-documents/testPDF_bookmarks.pdf")) {
+ hooked(parser, hook).parse(tis, new ToXMLContentHandler(),
metadata, context);
+ }
+ String label = strategy.name();
+ assertEquals(List.of(1, 2), hook.pages, label);
+ assertSame(metadata, hook.document, label + ": the pages belong to
the PDF");
+ assertTrue(hook.sizes.get(0) > 0, label + ": the render has
bytes");
+ assertEquals(2, tracker.getNextId() - 1, label + ": one render per
page");
+ assertEquals(strategy == OcrConfig.Strategy.NO_OCR ? 0 : 2,
recognizer.calls, label);
+ }
+ }
+
+ /** Nothing renders for inference unless the config says PAGES and a hook
wants them. */
+ @Test
+ public void testPagesNotRenderedUnlessWanted() throws Exception {
+ PageHook unasked = new PageHook();
+ ParseContext context = new ParseContext();
+ PDFParserConfig config = new PDFParserConfig();
+ config.getOcr().setStrategy(OcrConfig.Strategy.NO_OCR);
+ context.set(PDFParserConfig.class, config);
+ RenderingTracker tracker = new RenderingTracker();
+ context.set(RenderingTracker.class, tracker);
+ try (TikaInputStream tis =
getResourceAsStream("/test-documents/testPDF_bookmarks.pdf")) {
+ hooked(new PDFParser(), unasked).parse(tis, new
ToXMLContentHandler(),
+ new Metadata(), context);
+ }
+ assertEquals(0, unasked.wantsPagesCalls, "the default input is TEXT:
hooks are not asked");
+ assertTrue(unasked.pages.isEmpty());
+
+ PageHook unwilling = new PageHook();
+ context = new ParseContext();
+ context.set(PDFParserConfig.class,
pagesForInference(OcrConfig.Strategy.NO_OCR));
+ context.set(RenderingTracker.class, tracker);
+ try (TikaInputStream tis =
getResourceAsStream("/test-documents/testPDF_bookmarks.pdf")) {
+ hooked(new PDFParser(), unwilling).parse(tis, new
ToXMLContentHandler(),
+ new Metadata(), context);
+ }
+ assertEquals(1, unwilling.wantsPagesCalls, "asked once per document");
+ assertTrue(unwilling.pages.isEmpty());
+ assertEquals(0, tracker.getNextId() - 1, "no binding wants pages:
nothing is rendered");
+ }
+
/** Annotating enricher for image/png: writes one chunk per render, never
text. */
private static class PageAnnotator implements Parser, ContentEnricher {
private static final long serialVersionUID = 1L;
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/test/resources/org/apache/tika/parser/pdf/tika-inline-config.json
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/test/resources/org/apache/tika/parser/pdf/tika-inline-config.json
index 0e326e2fdf..65690be7ed 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/test/resources/org/apache/tika/parser/pdf/tika-inline-config.json
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pdf-module/src/test/resources/org/apache/tika/parser/pdf/tika-inline-config.json
@@ -14,6 +14,7 @@
"imageQuality": 2.1,
"imageFormat": "JPEG"
},
+ "inference": { "input": ["PAGES"] },
"maxMainMemoryBytes": 524288000
}
}
diff --git
a/tika-serialization/src/test/java/org/apache/tika/config/loader/InferenceLoaderTest.java
b/tika-serialization/src/test/java/org/apache/tika/config/loader/InferenceLoaderTest.java
index 22a2cf2c7e..167e16d39f 100644
---
a/tika-serialization/src/test/java/org/apache/tika/config/loader/InferenceLoaderTest.java
+++
b/tika-serialization/src/test/java/org/apache/tika/config/loader/InferenceLoaderTest.java
@@ -32,9 +32,12 @@ import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
import org.apache.tika.exception.TikaConfigException;
+import org.apache.tika.metadata.Metadata;
import org.apache.tika.mime.MediaType;
import org.apache.tika.parser.AutoDetectParser;
+import org.apache.tika.parser.ParseContext;
import org.apache.tika.parser.inference.EngineRegistry;
+import org.apache.tika.parser.inference.InferenceBinding;
import org.apache.tika.parser.inference.InferenceDispatcher;
import org.apache.tika.parser.inference.InferenceSelection;
import org.apache.tika.parser.inference.InputKind;
@@ -87,6 +90,21 @@ public class InferenceLoaderTest {
"the dispatcher rides every parse as a hook");
}
+ @Test
+ public void testPagesBindingLoads() throws Exception {
+ TikaLoader loader = load("{" + ENGINES + ", \"inference\": ["
+ + " { \"id\": \"page-vectors\", \"engine\": \"one\",
\"input\": \"PAGES\","
+ + " \"tasks\": [\"test-task\"], \"maxChunks\": 50 } ] }");
+ InferenceDispatcher dispatcher = loader.get(InferenceDispatcher.class);
+ InferenceBinding pages = dispatcher.getBound().get(0).binding();
+ assertEquals(InputKind.PAGES, pages.getInput());
+ assertTrue(pages.accepts(InputKind.PAGES, MediaType.image("png")));
+ assertFalse(pages.accepts(InputKind.IMAGES, MediaType.image("png")),
+ "a PAGES binding never takes an image document");
+ assertTrue(dispatcher.wantsPages(MediaType.image("png"), new
Metadata(),
+ new ParseContext()));
+ }
+
@Test
public void testPerRequestSelectionResolves() throws Exception {
TikaLoader loader = load("{" + ENGINES + ", \"inference\": ["
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/InferenceBindingsWireTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/InferenceBindingsWireTest.java
index 0b04515ea4..ca674fe06f 100644
---
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/InferenceBindingsWireTest.java
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/InferenceBindingsWireTest.java
@@ -20,18 +20,30 @@ import static java.nio.charset.StandardCharsets.UTF_8;
import static org.junit.jupiter.api.Assertions.assertEquals;
import java.awt.image.BufferedImage;
+import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
+import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
import javax.imageio.ImageIO;
import jakarta.ws.rs.core.Response;
import org.apache.cxf.jaxrs.JAXRSServerFactoryBean;
import org.apache.cxf.jaxrs.client.WebClient;
+import org.apache.cxf.jaxrs.ext.multipart.Attachment;
+import org.apache.cxf.jaxrs.ext.multipart.ContentDisposition;
+import org.apache.cxf.jaxrs.ext.multipart.MultipartBody;
import org.apache.cxf.jaxrs.lifecycle.SingletonResourceProvider;
+import org.apache.pdfbox.pdmodel.PDDocument;
+import org.apache.pdfbox.pdmodel.PDPage;
+import org.apache.pdfbox.pdmodel.PDPageContentStream;
+import org.apache.pdfbox.pdmodel.common.PDRectangle;
+import org.apache.pdfbox.pdmodel.font.PDType1Font;
+import org.apache.pdfbox.pdmodel.font.Standard14Fonts;
import org.junit.jupiter.api.Test;
import org.apache.tika.metadata.Metadata;
@@ -65,6 +77,11 @@ public class InferenceBindingsWireTest extends CXFTestBase {
return
getClass().getResourceAsStream("/configs/tika-config-inference-bindings.json");
}
+ @Override
+ protected boolean isAllowPerRequestConfig() {
+ return true;
+ }
+
@Test
public void testBindingOutputSurvivesFork() throws Exception {
ByteArrayOutputStream png = new ByteArrayOutputStream();
@@ -82,4 +99,53 @@ public class InferenceBindingsWireTest extends CXFTestBase {
assertEquals("mock-images", metadata.get(MockTask.MARKER_KEY));
assertEquals("1", metadata.get(MockTask.UNITS_KEY));
}
+
+ /** A request asks for a PDF's pages; the PAGES binding's output lands on
the PDF. */
+ @Test
+ public void testPagesPerRequest() throws Exception {
+ String config = """
+ { "parse-context": {
+ "pdf-parser": { "ocr": { "strategy": "NO_OCR", "dpi": 20 },
+ "inference": { "input": ["PAGES"] } },
+ "inference": { "bindings": ["mock-pages"] } } }
+ """;
+ ContentDisposition fileCd = new ContentDisposition(
+ "form-data; name=\"file\"; filename=\"two-pages.pdf\"");
+ Attachment fileAtt = new Attachment("file", new
ByteArrayInputStream(twoPagePdf()), fileCd);
+ Attachment configAtt = new Attachment("config", "application/json",
+ new ByteArrayInputStream(config.getBytes(UTF_8)));
+
+ Response response = WebClient
+ .create(endPoint + META_PATH + "/config")
+ .type("multipart/form-data")
+ .accept("application/json")
+ .post(new MultipartBody(Arrays.asList(fileAtt, configAtt)));
+
+ assertEquals(200, response.getStatus());
+ Reader reader = new InputStreamReader((InputStream)
response.getEntity(), UTF_8);
+ List<Metadata> metadataList = JsonMetadataList.fromJson(reader);
+ assertEquals(1, metadataList.size());
+ Metadata metadata = metadataList.get(0);
+ assertEquals("mock-pages", metadata.get(MockTask.MARKER_KEY));
+ assertEquals("2", metadata.get(MockTask.UNITS_KEY));
+ }
+
+ private static byte[] twoPagePdf() throws IOException {
+ try (PDDocument document = new PDDocument()) {
+ for (int i = 1; i <= 2; i++) {
+ PDPage page = new PDPage(PDRectangle.LETTER);
+ document.addPage(page);
+ try (PDPageContentStream content = new
PDPageContentStream(document, page)) {
+ content.beginText();
+ content.setFont(new
PDType1Font(Standard14Fonts.FontName.HELVETICA), 12);
+ content.newLineAtOffset(72, 700);
+ content.showText("This is page " + i + " of the test
document.");
+ content.endText();
+ }
+ }
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ document.save(out);
+ return out.toByteArray();
+ }
+ }
}
diff --git
a/tika-server/tika-server-standard/src/test/resources/configs/tika-config-inference-bindings.json
b/tika-server/tika-server-standard/src/test/resources/configs/tika-config-inference-bindings.json
index 445057c367..601646c106 100644
---
a/tika-server/tika-server-standard/src/test/resources/configs/tika-config-inference-bindings.json
+++
b/tika-server/tika-server-standard/src/test/resources/configs/tika-config-inference-bindings.json
@@ -1,6 +1,9 @@
{
"engines": { "mock": { "mock-engine": {} } },
- "inference": [ { "id": "mock-images", "engine": "mock", "input": "IMAGES",
"tasks": ["mock-task"] } ],
+ "inference": [
+ { "id": "mock-images", "engine": "mock", "input": "IMAGES", "tasks":
["mock-task"] },
+ { "id": "mock-pages", "engine": "mock", "input": "PAGES", "tasks":
["mock-task"] }
+ ],
"auto-detect-parser": { "throwOnZeroBytes": false },
"parse-context": { "timeout-limits": { "progressTimeoutMillis": 30000 } }
}
