This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new f499ebaeee TIKA-5027 provenance (#3525)
f499ebaeee is described below
commit f499ebaeee1608c9976c93c4ec28ecadbfbc282c
Author: Tim Allison <[email protected]>
AuthorDate: Wed Nov 4 15:44:23 2026 -0500
TIKA-5027 provenance (#3525)
* TIKA-5027: junk detector treats scripts without a usable table as
unmodeled, not junk
Scripts whose bigram table has fewer than 1000 bigrams add no z1 evidence;
all-unmodeled text takes the coherence fallback. TextQualityScore gains
getCoverage(); tika-eval reports languageness as unknown at coverage 0.
The fallback's z6 term is anchored so letter-free text scores -1, not +3.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* TIKA-5027: record junk-detector training provenance
TrainJunkModel and BuildJunkTrainingData take required paths and refuse a
corpus without manifest.tsv. The trainer writes <model>.provenance (model,
corpus and training-data digests, git commit, command line, every training
constant); ModelProvenanceTest fails if the bundled model and its
provenance diverge. train-junk-model.sh pins the corpus and keeps one run
directory per training run. Docs updated.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../ROOT/pages/advanced/junk-detection-build.adoc | 38 ++++-
.../ml/junkdetect/tools/BuildJunkTrainingData.java | 26 ++-
.../tika/ml/junkdetect/tools/TrainJunkModel.java | 56 +++++--
.../ml/junkdetect/tools/TrainingProvenance.java | 185 +++++++++++++++++++++
.../junkdetect/tools/TrainingProvenanceTest.java | 66 ++++++++
.../tika-ml-junkdetect-tools/train-junk-model.sh | 66 ++++++++
tika-ml/tika-ml-junkdetect/pom.xml | 1 +
.../tika/ml/junkdetect/junkdetect.bin.provenance | 4 +
.../tika/ml/junkdetect/ModelProvenanceTest.java | 56 +++++++
9 files changed, 464 insertions(+), 34 deletions(-)
diff --git a/docs/modules/ROOT/pages/advanced/junk-detection-build.adoc
b/docs/modules/ROOT/pages/advanced/junk-detection-build.adoc
index e163696419..696f68526b 100644
--- a/docs/modules/ROOT/pages/advanced/junk-detection-build.adoc
+++ b/docs/modules/ROOT/pages/advanced/junk-detection-build.adoc
@@ -76,6 +76,20 @@ self-contained jar by the `train` profile:
The result is `tika-ml-junkdetect-tools-*-tools.jar`.
+To rebuild the bundled model, run the driver script from the repository root
+rather than the tools one by one:
+
+[source,bash]
+----
+TIKA_JUNK_DATA=<dir holding the corpus>
tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
+----
+
+The script names the source corpus (its `CORPUS` constant is the committed
record
+of which data trained the bundled model), builds the tools, runs stages 1 and 3
+into a fresh run directory `<TIKA_JUNK_DATA>/<corpus>-runs/<time>-<commit>/`
(log,
+built data, and the exact uncommitted diff if the tree was dirty), and writes
the
+model to its classpath location.
+
[IMPORTANT]
====
Durable training parameters are *not* CLI flags. They are `public static
final`
@@ -166,8 +180,11 @@ java -cp tika-ml-junkdetect-tools-*-tools.jar \
--output-dir <workdir>/junkdetect
----
-`--data-dir`, `--output-dir`, and `--dry-run` (detect scripts and show the
budget
-without writing) are the only accepted arguments.
+`--data-dir` and `--output-dir` are required; `--dry-run` (detect scripts and
show
+the budget without writing) is the only other accepted argument. The source
+corpus must carry a `manifest.tsv`; the tool refuses a directory without one,
and
+records the corpus name and the manifest's SHA-256 in `source.tsv` in the
output
+directory.
=== Config constants
@@ -270,15 +287,18 @@ weights are fit in the same run.
[source,bash]
----
java -cp tika-ml-junkdetect-tools-*-tools.jar \
- org.apache.tika.ml.junkdetect.tools.TrainJunkModel
+ org.apache.tika.ml.junkdetect.tools.TrainJunkModel \
+ --data-dir <workdir>/junkdetect --output <model.bin> [--git-commit <sha>]
----
-The tool takes **no arguments**. Its input directory and its output path — the
-classpath resource
-`tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin`
-— are baked into `main`, deliberately: one bundled model in git, never a
training
-run to a scratch directory, never parallel A/B variants. Editing either means
-editing and committing the source.
+`--data-dir` (a `BuildJunkTrainingData` output directory, with its
`manifest.tsv`
+and `source.tsv`) and `--output` are required. Next to the model the tool
writes
+`<model>.provenance`: the model's SHA-256, the source corpus name and manifest
+digest, the training data's manifest digest, the git commit, the command line,
and
+every training constant. Commit the model and its provenance file together;
+`ModelProvenanceTest` fails the build if the bundled model's digest does not
match
+its provenance. There is one bundled model in git: train through
+`train-junk-model.sh`, never keep parallel model variants.
=== Calibration
diff --git
a/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/BuildJunkTrainingData.java
b/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/BuildJunkTrainingData.java
index 75b0a15ac1..f0637193f1 100644
---
a/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/BuildJunkTrainingData.java
+++
b/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/BuildJunkTrainingData.java
@@ -70,14 +70,15 @@ import java.util.zip.GZIPOutputStream;
* {script}.dev.gz — 10% split, used for calibration (mu/sigma)
* {script}.test.gz — 10% split, held out for final evaluation only
* manifest.tsv — per-script stats: entropy, budget, bytes written,
languages
+ * source.tsv — source corpus name and the SHA-256 of its
manifest.tsv
* </pre>
*
* <p>Usage:
* <pre>
* java BuildJunkTrainingData \
- * --data-dir ~/datasets/madlad/data \
- * --output-dir ~/datasets/madlad/junkdetect \
- * [--total-budget-bytes 50000000]
+ * --data-dir <corpus> \
+ * --output-dir <workdir>/junkdetect \
+ * [--dry-run]
* </pre>
*/
public class BuildJunkTrainingData {
@@ -101,8 +102,8 @@ public class BuildJunkTrainingData {
// -----------------------------------------------------------------------
public static void main(String[] args) throws IOException {
- Path dataDir = Paths.get(System.getProperty("user.home"), "datasets",
"madlad", "data");
- Path outputDir = Paths.get(System.getProperty("user.home"),
"datasets", "madlad", "junkdetect");
+ Path dataDir = null;
+ Path outputDir = null;
boolean dryRun = false;
// Bind config-controlled values into local variables. These are
@@ -155,6 +156,13 @@ public class BuildJunkTrainingData {
}
}
+ if (dataDir == null || outputDir == null) {
+ System.err.println("ERROR: --data-dir and --output-dir are
required");
+ printUsage();
+ System.exit(1);
+ }
+ TrainingProvenance.requireManifest(dataDir, "source corpus");
+
System.out.println("=== BuildJunkTrainingData ===");
System.out.println(" data-dir: " + dataDir);
System.out.println(" output-dir: " + outputDir);
@@ -446,6 +454,7 @@ public class BuildJunkTrainingData {
}
}
+ TrainingProvenance.writeSource(dataDir, outputDir);
System.out.println("\nWrote manifest: " + manifest);
System.out.println("Done.");
}
@@ -728,10 +737,9 @@ public class BuildJunkTrainingData {
private static void printUsage() {
System.err.println("Usage: BuildJunkTrainingData [options]");
- System.err.println(" --data-dir <path> MADLAD data root"
- + " (default: ~/datasets/madlad/data)");
- System.err.println(" --output-dir <path> Output directory"
- + " (default: ~/datasets/madlad/junkdetect)");
+ System.err.println(" --data-dir <path> Source corpus: <lang>/*.txt
plus manifest.tsv"
+ + " (required)");
+ System.err.println(" --output-dir <path> Output directory
(required)");
System.err.println(" --dry-run Detect scripts + show
budget,"
+ " skip file writing.");
System.err.println();
diff --git
a/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/TrainJunkModel.java
b/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/TrainJunkModel.java
index dc7e5f053e..c7ff4cd7ad 100644
---
a/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/TrainJunkModel.java
+++
b/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/TrainJunkModel.java
@@ -198,14 +198,43 @@ public class TrainJunkModel {
static final int CALIB_SAMPLES = 5000;
public static void main(String[] args) throws IOException {
- // Training data and output paths are intentionally NOT
CLI-configurable.
- // Single bundled model in git; never train to /tmp/; never maintain
- // parallel A/B variants. Edit and commit if these need to change.
- Path dataDir = Paths.get(System.getProperty("user.home"),
- "data", "junk-augmented-symbolboost");
- Path output = Paths.get(
- "tika-ml/tika-ml-junkdetect/src/main/resources",
- "org/apache/tika/ml/junkdetect/junkdetect.bin");
+ // Run through train-junk-model.sh, which pins the corpus; every run
writes
+ // <output>.provenance so a model can be traced to its data and code.
+ Path dataDir = null;
+ Path output = null;
+ String gitCommit = null;
+ for (int i = 0; i < args.length; i++) {
+ if (i + 1 >= args.length) {
+ System.err.println("ERROR: " + args[i] + " needs a value");
+ System.exit(1);
+ }
+ switch (args[i]) {
+ case "--data-dir":
+ dataDir = Paths.get(args[++i]);
+ break;
+ case "--output":
+ output = Paths.get(args[++i]);
+ break;
+ case "--git-commit":
+ gitCommit = args[++i];
+ break;
+ default:
+ System.err.println("Unknown argument: " + args[i]
+ + ". Training parameters live in
JunkDetectorTrainingConfig.");
+ System.exit(1);
+ }
+ }
+ if (dataDir == null || output == null) {
+ System.err.println("Usage: TrainJunkModel --data-dir
<BuildJunkTrainingData output>"
+ + " --output <model.bin> [--git-commit <sha>]");
+ System.exit(1);
+ }
+ TrainingProvenance.requireManifest(dataDir, "training data");
+ if
(!Files.isRegularFile(dataDir.resolve(TrainingProvenance.SOURCE_FILE))) {
+ System.err.println("ERROR: " + dataDir + " has no " +
TrainingProvenance.SOURCE_FILE
+ + "; rebuild it with BuildJunkTrainingData");
+ System.exit(1);
+ }
// Durable training parameters live in JunkDetectorTrainingConfig; this
// tool deliberately refuses CLI overrides so a built model file's
@@ -226,14 +255,6 @@ public class TrainJunkModel {
System.exit(1);
}
- if (args.length > 0) {
- System.err.println("TrainJunkModel takes no arguments. Data-dir
and"
- + " output path are baked in; edit the source if they need"
- + " to change. Other training parameters live in"
- + " JunkDetectorTrainingConfig.");
- System.exit(1);
- }
-
System.out.println("=== TrainJunkModel ===");
System.out.println(" data-dir: " + dataDir);
System.out.println(" output: " + output);
@@ -387,6 +408,9 @@ public class TrainJunkModel {
combiner, scriptBuckets, scriptTransTable, scriptTransCal,
output);
System.out.printf(Locale.ROOT, "Model size: %,d bytes (%.1f KB)%n",
Files.size(output), Files.size(output) / 1024.0);
+ TrainingProvenance.writeModelProvenance(output, dataDir, args,
gitCommit,
+ List.of(JunkDetectorTrainingConfig.class,
TrainJunkModel.class));
+ System.out.println("Wrote " + output +
TrainingProvenance.PROVENANCE_SUFFIX);
System.out.println("Done.");
}
diff --git
a/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/TrainingProvenance.java
b/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/TrainingProvenance.java
new file mode 100644
index 0000000000..eeec4e5c36
--- /dev/null
+++
b/tika-ml/tika-ml-junkdetect-tools/src/main/java/org/apache/tika/ml/junkdetect/tools/TrainingProvenance.java
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.ml.junkdetect.tools;
+
+import java.io.BufferedWriter;
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.reflect.Field;
+import java.lang.reflect.Modifier;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.security.MessageDigest;
+import java.security.NoSuchAlgorithmException;
+import java.time.Instant;
+import java.util.Arrays;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records which corpus, config and code produced a junk-detector model, so a
+ * model file can always be traced back to its inputs.
+ *
+ * <p>Chain: the source corpus carries a {@code manifest.tsv};
+ * {@link BuildJunkTrainingData} copies its name and digest into
+ * {@value #SOURCE_FILE} in the built directory; {@link TrainJunkModel} writes
+ * {@code <model>.provenance} with the model digest and everything upstream.
+ */
+public final class TrainingProvenance {
+
+ static final String MANIFEST = "manifest.tsv";
+ static final String SOURCE_FILE = "source.tsv";
+ public static final String PROVENANCE_SUFFIX = ".provenance";
+
+ private TrainingProvenance() {
+ }
+
+ /** Fails unless {@code dir} is a directory holding a {@value #MANIFEST}.
*/
+ static Path requireManifest(Path dir, String what) throws IOException {
+ Path m = dir.resolve(MANIFEST);
+ if (!Files.isRegularFile(m)) {
+ throw new IOException(what + " " + dir + " has no " + MANIFEST
+ + "; refusing to use a directory without a recorded
identity");
+ }
+ return m;
+ }
+
+ /** Writes {@value #SOURCE_FILE} into the built directory, naming the
source corpus. */
+ static void writeSource(Path sourceCorpus, Path builtDir) throws
IOException {
+ Path m = requireManifest(sourceCorpus, "source corpus");
+ Map<String, String> kv = new LinkedHashMap<>();
+ kv.put("source_corpus",
sourceCorpus.toAbsolutePath().normalize().getFileName().toString());
+ kv.put("source_manifest_sha256", sha256(m));
+ writeKv(builtDir.resolve(SOURCE_FILE), kv);
+ }
+
+ /** Reads a key/value file written by {@link #writeKv}. */
+ static Map<String, String> readKv(Path p) throws IOException {
+ Map<String, String> kv = new LinkedHashMap<>();
+ for (String line : Files.readAllLines(p, StandardCharsets.UTF_8)) {
+ int t = line.indexOf('\t');
+ if (t > 0 && !line.startsWith("#")) {
+ kv.put(line.substring(0, t), line.substring(t + 1));
+ }
+ }
+ return kv;
+ }
+
+ static void writeKv(Path p, Map<String, String> kv) throws IOException {
+ try (BufferedWriter w = Files.newBufferedWriter(p,
StandardCharsets.UTF_8)) {
+ for (Map.Entry<String, String> e : kv.entrySet()) {
+ w.write(e.getKey());
+ w.write('\t');
+ w.write(e.getValue().replace('\t', ' ').replace('\n', ' '));
+ w.write('\n');
+ }
+ }
+ }
+
+ /**
+ * Writes {@code <model>.provenance}: model digest, built-data and
source-corpus
+ * identity, every static training constant, the command line and git
state.
+ */
+ static void writeModelProvenance(Path model, Path builtDir, String[] args,
+ String gitCommit, List<Class<?>>
configClasses)
+ throws IOException {
+ Path builtManifest = requireManifest(builtDir, "training data");
+ Path source = builtDir.resolve(SOURCE_FILE);
+ if (!Files.isRegularFile(source)) {
+ throw new IOException("training data " + builtDir + " has no " +
SOURCE_FILE
+ + "; rebuild it with BuildJunkTrainingData");
+ }
+ Map<String, String> kv = new LinkedHashMap<>();
+ kv.put("model_sha256", sha256(model));
+ kv.put("trained_at", Instant.now().toString());
+ kv.putAll(readKv(source));
+ kv.put("training_data", shortPath(builtDir));
+ kv.put("training_data_manifest_sha256", sha256(builtManifest));
+ kv.put("git_commit", gitCommit == null ? "unrecorded" : gitCommit);
+ StringBuilder a = new StringBuilder();
+ for (int i = 0; i < args.length; i++) {
+ boolean path = i > 0 && (args[i - 1].equals("--data-dir") ||
args[i - 1].equals("--output"));
+ a.append(i == 0 ? "" : " ").append(path ?
shortPath(Path.of(args[i])) : args[i]);
+ }
+ kv.put("args", a.toString());
+ kv.put("java_version", System.getProperty("java.version"));
+ for (Class<?> c : configClasses) {
+ for (Field f : c.getDeclaredFields()) {
+ int mod = f.getModifiers();
+ if (!Modifier.isStatic(mod) || !Modifier.isFinal(mod) ||
f.isSynthetic()) {
+ continue;
+ }
+ try {
+ f.setAccessible(true);
+ kv.put("config." + c.getSimpleName() + "." + f.getName(),
render(f.get(null)));
+ } catch (ReflectiveOperationException | RuntimeException e) {
+ kv.put("config." + c.getSimpleName() + "." + f.getName(),
"?");
+ }
+ }
+ }
+ writeKv(model.resolveSibling(model.getFileName() + PROVENANCE_SUFFIX),
kv);
+ }
+
+ /** Last two path components: identifies a run without recording a machine
path. */
+ static String shortPath(Path p) {
+ Path n = p.toAbsolutePath().normalize();
+ int c = n.getNameCount();
+ return c >= 2 ? n.subpath(c - 2, c).toString() : n.toString();
+ }
+
+ private static String render(Object v) {
+ if (v instanceof int[]) {
+ return Arrays.toString((int[]) v);
+ }
+ if (v instanceof double[]) {
+ return Arrays.toString((double[]) v);
+ }
+ if (v instanceof float[]) {
+ return Arrays.toString((float[]) v);
+ }
+ if (v instanceof long[]) {
+ return Arrays.toString((long[]) v);
+ }
+ if (v instanceof Object[]) {
+ return Arrays.deepToString((Object[]) v);
+ }
+ return String.valueOf(v);
+ }
+
+ static String sha256(Path p) throws IOException {
+ MessageDigest md;
+ try {
+ md = MessageDigest.getInstance("SHA-256");
+ } catch (NoSuchAlgorithmException e) {
+ throw new IllegalStateException(e);
+ }
+ byte[] buf = new byte[1 << 16];
+ try (InputStream is = Files.newInputStream(p)) {
+ int n;
+ while ((n = is.read(buf)) > 0) {
+ md.update(buf, 0, n);
+ }
+ }
+ StringBuilder sb = new StringBuilder(64);
+ for (byte b : md.digest()) {
+ sb.append(String.format(java.util.Locale.ROOT, "%02x", b));
+ }
+ return sb.toString();
+ }
+}
diff --git
a/tika-ml/tika-ml-junkdetect-tools/src/test/java/org/apache/tika/ml/junkdetect/tools/TrainingProvenanceTest.java
b/tika-ml/tika-ml-junkdetect-tools/src/test/java/org/apache/tika/ml/junkdetect/tools/TrainingProvenanceTest.java
new file mode 100644
index 0000000000..92d1ba3dc5
--- /dev/null
+++
b/tika-ml/tika-ml-junkdetect-tools/src/test/java/org/apache/tika/ml/junkdetect/tools/TrainingProvenanceTest.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.ml.junkdetect.tools;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+public class TrainingProvenanceTest {
+
+ @Test
+ void chainRecordsCorpusDataAndModel(@TempDir Path tmp) throws IOException {
+ Path corpus = Files.createDirectories(tmp.resolve("corpus-x"));
+ Path built = Files.createDirectories(tmp.resolve("built"));
+ Path model = tmp.resolve("m.bin");
+
+ assertThrows(IOException.class, () ->
TrainingProvenance.writeSource(corpus, built));
+ Files.writeString(corpus.resolve("manifest.tsv"), "lang\tscript\n",
StandardCharsets.UTF_8);
+ TrainingProvenance.writeSource(corpus, built);
+
+ Files.writeString(model, "model", StandardCharsets.UTF_8);
+ assertThrows(IOException.class, () ->
TrainingProvenance.writeModelProvenance(
+ model, built, new String[0], "abc", List.of()));
+ Files.writeString(built.resolve("manifest.tsv"), "script\n",
StandardCharsets.UTF_8);
+ TrainingProvenance.writeModelProvenance(model, built,
+ new String[]{"--data-dir", built.toString(), "--git-commit",
"abc123"}, "abc123",
+ List.of(JunkDetectorTrainingConfig.class));
+
+ Map<String, String> kv =
TrainingProvenance.readKv(tmp.resolve("m.bin.provenance"));
+ assertEquals(TrainingProvenance.sha256(model), kv.get("model_sha256"));
+ assertEquals("corpus-x", kv.get("source_corpus"));
+ assertEquals(TrainingProvenance.sha256(corpus.resolve("manifest.tsv")),
+ kv.get("source_manifest_sha256"));
+ assertEquals("abc123", kv.get("git_commit"));
+ assertEquals("--data-dir " + TrainingProvenance.shortPath(built) + "
--git-commit abc123",
+ kv.get("args"));
+ assertTrue(!kv.get("args").contains(tmp.toString()), "machine path
leaked: " + kv.get("args"));
+
assertEquals(String.valueOf(JunkDetectorTrainingConfig.MIN_BIGRAM_COUNT),
+ kv.get("config.JunkDetectorTrainingConfig.MIN_BIGRAM_COUNT"));
+ assertTrue(kv.containsKey("training_data_manifest_sha256"));
+ }
+}
diff --git a/tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
b/tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
new file mode 100755
index 0000000000..77f9ebe07d
--- /dev/null
+++ b/tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
@@ -0,0 +1,66 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Rebuilds the bundled junk-detector model from a clean-text corpus.
+#
+# CORPUS is the committed record of which data trained the bundled model:
+# change it here and commit, never pass it on the command line. The run writes
+# junkdetect.bin plus junkdetect.bin.provenance (model digest, corpus manifest
+# digest, git state, every training constant); commit both together.
+#
+# Usage: TIKA_JUNK_DATA=<dir containing $CORPUS>
tika-ml/tika-ml-junkdetect-tools/train-junk-model.sh
+# Optional: JAVA_OPTS (default -Xmx12g), MAVEN_REPO_LOCAL (default
<repo>/.local_m2_repo).
+
+set -euo pipefail
+
+CORPUS=junk-clean-20261103
+
+: "${TIKA_JUNK_DATA:?set TIKA_JUNK_DATA to the directory that holds ${CORPUS}}"
+SRC="${TIKA_JUNK_DATA}/${CORPUS}"
+[ -f "${SRC}/manifest.tsv" ] || { echo "no manifest.tsv in ${SRC}" >&2; exit
1; }
+
+ROOT=$(git rev-parse --show-toplevel)
+cd "${ROOT}"
+
+# An uncommitted tree is allowed (experiments), but the exact diff is kept
beside the log.
+GIT=$(git rev-parse HEAD)
+if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
+ DIFF_SHA=$(git diff HEAD | sha256sum | cut -c1-12)
+ GIT="${GIT}-dirty-${DIFF_SHA}"
+fi
+RUN="${TIKA_JUNK_DATA}/${CORPUS}-runs/$(date +%Y%m%d-%H%M%S)-${GIT:0:12}"
+BUILT="${RUN}/built"
+mkdir -p "${RUN}"
+git diff HEAD > "${RUN}/uncommitted.diff"
+LOG="${RUN}/train.log"
+echo "corpus=${CORPUS} git=${GIT} run=${RUN}" | tee "${LOG}"
+
+REPO="${MAVEN_REPO_LOCAL:-${ROOT}/.local_m2_repo}"
+./mvnw clean install -pl tika-ml/tika-ml-junkdetect-tools -am -Pfast,train \
+ -Dmaven.repo.local="${REPO}" >> "${LOG}" 2>&1
+JAR=$(ls
tika-ml/tika-ml-junkdetect-tools/target/tika-ml-junkdetect-tools-*-tools.jar)
+OUT=tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin
+JOPTS=${JAVA_OPTS:--Xmx12g}
+
+# shellcheck disable=SC2086
+java ${JOPTS} -cp "${JAR}"
org.apache.tika.ml.junkdetect.tools.BuildJunkTrainingData \
+ --data-dir "${SRC}" --output-dir "${BUILT}" 2>&1 | tee -a "${LOG}"
+# shellcheck disable=SC2086
+java ${JOPTS} -cp "${JAR}" org.apache.tika.ml.junkdetect.tools.TrainJunkModel \
+ --data-dir "${BUILT}" --output "${OUT}" --git-commit "${GIT}" 2>&1 | tee -a
"${LOG}"
+cp "${OUT}.provenance" "${RUN}/"
+echo "model: ${OUT} provenance: ${OUT}.provenance log: ${LOG}" | tee -a
"${LOG}"
diff --git a/tika-ml/tika-ml-junkdetect/pom.xml
b/tika-ml/tika-ml-junkdetect/pom.xml
index 61199f0bb3..76d57a26f5 100644
--- a/tika-ml/tika-ml-junkdetect/pom.xml
+++ b/tika-ml/tika-ml-junkdetect/pom.xml
@@ -92,6 +92,7 @@
<configuration>
<inputExcludes>
<inputExclude>**/*.bin</inputExclude>
+ <inputExclude>**/*.provenance</inputExclude>
<inputExclude>**/*.txt</inputExclude>
</inputExcludes>
</configuration>
diff --git
a/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin.provenance
b/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin.provenance
new file mode 100644
index 0000000000..39aee43235
--- /dev/null
+++
b/tika-ml/tika-ml-junkdetect/src/main/resources/org/apache/tika/ml/junkdetect/junkdetect.bin.provenance
@@ -0,0 +1,4 @@
+model_sha256 509b4a2d0b5b6eee26f01d94d2373081dc628e31e27ea90e5fd280e2fdc613b2
+source_corpus junk-augmented-symbolboost (MADLAD-400 + Wikipedia +
symbol-boost augmentation; lost, language list not recorded)
+git_commit 8d9900e211 (last commit to change the model, TIKA-4745)
+note trained before provenance was recorded; retrain with train-junk-model.sh
diff --git
a/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/ModelProvenanceTest.java
b/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/ModelProvenanceTest.java
new file mode 100644
index 0000000000..b03cb5312f
--- /dev/null
+++
b/tika-ml/tika-ml-junkdetect/src/test/java/org/apache/tika/ml/junkdetect/ModelProvenanceTest.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.ml.junkdetect;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.security.MessageDigest;
+import java.util.HexFormat;
+
+import org.junit.jupiter.api.Test;
+
+/** The bundled model must ship with the provenance record its trainer wrote.
*/
+public class ModelProvenanceTest {
+
+ @Test
+ void provenanceMatchesBundledModel() throws Exception {
+ byte[] model;
+ try (InputStream is = getClass().getClassLoader()
+ .getResourceAsStream(JunkDetector.DEFAULT_MODEL_RESOURCE)) {
+ assertNotNull(is, "bundled model missing");
+ model = is.readAllBytes();
+ }
+ String provenance;
+ try (InputStream is = getClass().getClassLoader()
+ .getResourceAsStream(JunkDetector.DEFAULT_MODEL_RESOURCE +
".provenance")) {
+ assertNotNull(is, "junkdetect.bin.provenance missing; train with
train-junk-model.sh");
+ provenance = new String(is.readAllBytes(), StandardCharsets.UTF_8);
+ }
+ String recorded = null;
+ for (String line : provenance.split("\n")) {
+ if (line.startsWith("model_sha256\t")) {
+ recorded = line.substring("model_sha256\t".length()).trim();
+ }
+ }
+ String actual =
HexFormat.of().formatHex(MessageDigest.getInstance("SHA-256").digest(model));
+ assertEquals(actual, recorded,
+ "junkdetect.bin changed without its provenance; retrain with
train-junk-model.sh");
+ }
+}
