This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new d39c62b9fd improve skills (#3461)
d39c62b9fd is described below
commit d39c62b9fdec7819073895a361cec24793d84d76
Author: Tim Allison <[email protected]>
AuthorDate: Fri Oct 30 17:05:39 2026 -0400
improve skills (#3461)
---
.skills/users/file-to-markdown-docker/SKILL.md | 48 ++++----
CHANGES.txt | 5 +
.../server/standard/SkillConfigExamplesTest.java | 122 +++++++++++++++++++++
3 files changed, 152 insertions(+), 23 deletions(-)
diff --git a/.skills/users/file-to-markdown-docker/SKILL.md
b/.skills/users/file-to-markdown-docker/SKILL.md
index b43974e1b9..c798b6c774 100644
--- a/.skills/users/file-to-markdown-docker/SKILL.md
+++ b/.skills/users/file-to-markdown-docker/SKILL.md
@@ -123,29 +123,26 @@ fail inside the container even though it works fine
outside it.
Don't assume the `-full` image means OCR fired on a given file — confirm it,
especially the first time you stand one up.
-**For PDFs**, `pdf:ocr-page-count` is a verified, code-confirmed signal —
-non-zero means Tesseract processed that many pages:
+The signal is `tk:parsed-by-full-set`: it lists
+`org.apache.tika.parser.ocr.TesseractOCRParser` when Tesseract was run on the
+file (a PDF page or an image), and leaves it out when it wasn't:
```bash
-curl -T scanned.pdf http://localhost:9998/rmeta | jq
'.[0]."pdf:ocr-page-count"'
+curl -T scanned.pdf http://localhost:9998/rmeta | jq
'.[0]."tk:parsed-by-full-set"'
```
-Secondary evidence: the same entry's `tk:parsed-by-full-set` lists
-`org.apache.tika.parser.ocr.TesseractOCRParser` when Tesseract ran.
+Do **not** use `pdf:ocr-page-count` for this. It counts the pages that looked
+like they needed OCR, whether or not an engine ran, so it is non-zero with OCR
+turned off too. `tk:ocr-warnings` names input the engine was not given:
+`skipped:<reason>` from an image check, `skipped:max-pages` once a
+`pages.ocr.maxPages` cap is reached (4.3.0+).
-`0` or `null` means OCR didn't run on this PDF — check you're on `-full` (not
-minimal), that the PDF is actually image-only (a PDF with a real text layer
-correctly skips OCR — that's not a bug), and that no mounted config sets
-`skipOcr: true` (see below). If the key exists but under a **camelCase name**
-(`pdf:ocrPageCount`) — or you see `X-TIKA:*` keys — your image predates the
-4.0.0 metadata-key renames; upgrade the image rather than adapting to the old
-spellings.
-
-**For standalone images** (not embedded in a PDF), there's no single
-verified flag confirmed here — this skill doesn't have a code-checked answer
-for that case yet. The pragmatic fallback: compare extracted-text length
-against a file you know is a genuine scan; suspiciously empty output on a
-visibly text-bearing image is the same "wrong image" signal as above.
+No engine listed means OCR didn't run — check you're on `-full` (not
+minimal), that the file actually needs it (a PDF with a real text layer
+correctly skips OCR — that's not a bug), and that no mounted config turns it
+off (see below). If you see camelCase keys (`pdf:ocrPageCount`) or `X-TIKA:*`
+keys, your image predates the 4.0.0 metadata-key renames; upgrade the image
+rather than adapting to the old spellings.
## Configuration (turning OCR off, or anything else)
@@ -159,18 +156,23 @@ docker run -d -p 127.0.0.1:9998:9998 \
apache/tika:<version>-full -c /tika-config.json
```
+Through 4.2.x the server refuses a config file without a `server` block
+(`Couldn't find 'server' element`), so include one, even if empty; it is
+harmless on later versions.
+
To keep `-full`'s other parsers but disable OCR specifically (e.g. you only
-wanted GDAL):
+wanted GDAL), empty the `text-recognizers` list (4.1.0+):
```json
{
- "parsers": [
- { "default-parser": {} },
- { "tesseract-ocr-parser": { "skipOcr": true } }
- ]
+ "server": {},
+ "text-recognizers": []
}
```
+Since 4.2.0 an OCR engine under `parsers` fails the config load; engines go
+under `engines` and are named in `text-recognizers`.
+
## Memory
Size the container's `--memory` limit; don't pass `-Xmx` — the JVM sizes its
diff --git a/CHANGES.txt b/CHANGES.txt
index fee32d6a5b..77e7c60b46 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -63,6 +63,11 @@ Release 4.3.0 - ???
OTHER CHANGES
+ * The file-to-markdown-docker agent skill turns OCR off with an empty
text-recognizers
+ list and a server block (its tesseract-ocr-parser config failed to load
since 4.2.0),
+ and checks tk:parsed-by-full-set, not pdf:ocr-page-count, to confirm OCR
ran. A test
+ loads every config in the user skills in tika-app and tika-server
(TIKA-XXXX).
+
* A runtime fetcher or emitter save or delete reaches running pipes workers
at
their next fetch, through a change token on file and ignite stores
(ConfigStore.changeToken); before, a worker kept the instance it first
built
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/SkillConfigExamplesTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/SkillConfigExamplesTest.java
new file mode 100644
index 0000000000..7d68e385fa
--- /dev/null
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/SkillConfigExamplesTest.java
@@ -0,0 +1,122 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.server.standard;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Stream;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.DefaultParser;
+import org.apache.commons.cli.Options;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import org.apache.tika.config.loader.TikaLoader;
+import org.apache.tika.server.core.TikaServerConfig;
+
+/**
+ * The user skills ship in the source release and agents copy their configs
verbatim: every one
+ * must load in tika-app and tika-server.
+ */
+public class SkillConfigExamplesTest {
+
+ private static final String SKILLS_RELATIVE_PATH = ".skills/users";
+
+ private static final Pattern JSON_BLOCK =
Pattern.compile("```json\\n(.*?)```", Pattern.DOTALL);
+
+ // a block with none of these is an output sample, not a config
+ private static final Set<String> CONFIG_KEYS = Set.of("server", "parsers",
"engines",
+ "text-recognizers", "parse-context", "pipes", "detectors",
"inference");
+
+ @TempDir
+ Path tempDir;
+
+ @Test
+ public void testUserSkillConfigsLoad() throws Exception {
+ Map<String, String> configs = findConfigs();
+ assertFalse(configs.isEmpty(), "no configs found under " +
SKILLS_RELATIVE_PATH);
+ List<String> failures = new ArrayList<>();
+ int i = 0;
+ for (Map.Entry<String, String> e : configs.entrySet()) {
+ Path file = tempDir.resolve("config-" + i++ + ".json");
+ Files.writeString(file, e.getValue(), StandardCharsets.UTF_8);
+ try {
+ TikaLoader.load(file).loadParsers();
+ CommandLine cl = new DefaultParser().parse(
+ new Options().addOption("c", true, ""), new String[]
{"-c", file.toString()});
+ TikaServerConfig.load(cl);
+ } catch (Exception ex) {
+ failures.add(e.getKey() + ": " + ex.getMessage());
+ }
+ }
+ assertTrue(failures.isEmpty(), "skill configs that fail to load:\n" +
String.join("\n", failures));
+ }
+
+ private static Map<String, String> findConfigs() throws Exception {
+ ObjectMapper mapper = new ObjectMapper();
+ Path root = findSkillsDir();
+ Map<String, String> configs = new TreeMap<>();
+ try (Stream<Path> files = Files.walk(root)) {
+ for (Path p : (Iterable<Path>)
files.filter(Files::isRegularFile)::iterator) {
+ String name = root.relativize(p).toString();
+ if (!name.endsWith(".json") && !name.endsWith(".md")) {
+ continue;
+ }
+ String text = Files.readString(p, StandardCharsets.UTF_8);
+ if (name.endsWith(".json")) {
+ configs.put(name, text);
+ } else if (name.endsWith(".md")) {
+ Matcher m = JSON_BLOCK.matcher(text);
+ while (m.find()) {
+ JsonNode node = mapper.readTree(m.group(1));
+ if (node.isObject() &&
CONFIG_KEYS.stream().anyMatch(node::has)) {
+ int line = text.substring(0,
m.start()).split("\n", -1).length;
+ configs.put(name + ":" + line, m.group(1));
+ }
+ }
+ }
+ }
+ }
+ return configs;
+ }
+
+ private static Path findSkillsDir() {
+ Path dir = Path.of("").toAbsolutePath();
+ for (int i = 0; i < 6 && dir != null; i++, dir = dir.getParent()) {
+ Path candidate = dir.resolve(SKILLS_RELATIVE_PATH);
+ if (Files.isDirectory(candidate)) {
+ return candidate;
+ }
+ }
+ throw new IllegalStateException(
+ "could not locate " + SKILLS_RELATIVE_PATH + " above " +
Path.of("").toAbsolutePath());
+ }
+}
