This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new bc1fbcaed7 TIKA-4681: unpack output is a Frictionless Data Package by
default (#3306)
bc1fbcaed7 is described below
commit bc1fbcaed769adaffeaab2c59167e27981792ee1
Author: Tim Allison <[email protected]>
AuthorDate: Wed Oct 7 14:31:18 2026 -0400
TIKA-4681: unpack output is a Frictionless Data Package by default (#3306)
---
.skills/users/file-forensics/SKILL.md | 20 +++--
.skills/users/file-to-markdown/SKILL.md | 10 +--
CHANGES.txt | 5 ++
.../advanced/integration-testing/tika-app.adoc | 21 +++---
.../advanced/integration-testing/tika-server.adoc | 4 +-
docs/modules/ROOT/pages/pipes/unpack-config.adoc | 80 ++++++++++++--------
docs/modules/ROOT/pages/using-tika/cli/index.adoc | 13 ++--
.../ROOT/pages/using-tika/server/index.adoc | 2 +-
.../src/main/java/org/apache/tika/cli/TikaCLI.java | 5 +-
.../test/java/org/apache/tika/cli/TikaCLITest.java | 87 +++++++++++++--------
.../org/apache/tika/async/cli/TikaAsyncCLI.java | 16 ++--
.../apache/tika/async/cli/AsyncProcessorTest.java | 5 +-
.../tika/pipes/core/extractor/UnpackConfig.java | 8 +-
.../extractor/UnpackConfigDeprecatedFlagsTest.java | 4 +-
.../tika/pipes/core/FrictionlessUnpackTest.java | 26 +++++++
.../org/apache/tika/pipes/core/UnpackModeTest.java | 53 ++++++++-----
.../server/core/resource/UnpackerResource.java | 27 +++----
.../tika/server/standard/ThumbnailPresetTest.java | 13 +++-
.../apache/tika/server/standard/TikaPipesTest.java | 27 ++++++-
.../server/standard/UnpackFrictionlessTest.java | 12 +--
...nlessTest.java => UnpackRegularConfigTest.java} | 88 +++++-----------------
.../standard/UnpackerResourceHandlerTest.java | 20 +++--
.../tika/server/standard/UnpackerResourceTest.java | 31 ++++----
23 files changed, 330 insertions(+), 247 deletions(-)
diff --git a/.skills/users/file-forensics/SKILL.md
b/.skills/users/file-forensics/SKILL.md
index 0bc3977554..0675d28b5d 100644
--- a/.skills/users/file-forensics/SKILL.md
+++ b/.skills/users/file-forensics/SKILL.md
@@ -106,7 +106,7 @@ java -jar tika-app.jar --config=file-forensics-config.json
-Z --extract-dir=evid
# mkdir -p evidence/suspect-embedded && unzip -q suspect-embedded.zip -d
evidence/suspect-embedded
jq 'map(del(."tk:content"))' suspect.rmeta.json > suspect.meta.json
-ls -lR evidence/suspect-embedded && sha256sum evidence/suspect-embedded/*/*
2>/dev/null || sha256sum evidence/suspect-embedded/*
+ls -lR evidence/suspect-embedded && sha256sum
evidence/suspect-embedded/*/unpacked/* 2>/dev/null || sha256sum
evidence/suspect-embedded/unpacked/*
```
`suspect.rmeta.json` is the full evidence record — a JSON array where entry
@@ -342,19 +342,23 @@ files is incomplete.
- Extracted files are **renumbered** (`00000001.jpg`, ...); original names
live in each rmeta entry's `tk:resource-name`, and `-z`/`-Z` writes a
sidecar `<name>.json` metadata dump for mapping numbers back to names.
-- Digest what you extracted (`sha256sum evidence/suspect-embedded/*/*`) so
- each artifact is pinned the same way the container file is.
+- The output is a Frictionless data package (Tika 4.2+): the files sit under
+ `<container>/unpacked/`, and `<container>/datapackage.json` already carries a
+ SHA-256 per file. Digest what you extracted
+ (`sha256sum evidence/suspect-embedded/*/unpacked/*`) and check it against the
+ manifest, so each artifact is pinned the same way the container file is.
- Mechanics and gotchas (Pipes-mode delay, chatty stderr) are in
`file-to-markdown`; the server-side equivalent is `/unpack`, which returns
- the embedded files as a zip over HTTP. **`/unpack` names differ from
- `-z`/`-Z`:** plain sequential names (`1.jpg`, `2.pdf`, ...) and **no
- sidecar JSON** — map names back via each rmeta entry's `tk:resource-name`
- yourself. Against the server started above (forensics config loaded):
+ the same package as one zip over HTTP. **`/unpack` names differ from
+ `-z`/`-Z`:** plain sequential names (`unpacked/1.jpg`, `unpacked/2.pdf`, ...)
+ and the rmeta rows live in the package's `metadata.json` rather than a
+ sidecar — map names back via each row's `tk:resource-name`. Against the
+ server started above (forensics config loaded):
```bash
curl -T suspect.file http://localhost:9998/unpack > suspect-embedded.zip
mkdir -p evidence/suspect-embedded && unzip -q suspect-embedded.zip -d
evidence/suspect-embedded
- sha256sum evidence/suspect-embedded/*
+ sha256sum evidence/suspect-embedded/unpacked/*
```
`/unpack/all` also includes the container's own text and metadata. Both
diff --git a/.skills/users/file-to-markdown/SKILL.md
b/.skills/users/file-to-markdown/SKILL.md
index 7ac6c83f6c..51c410007b 100644
--- a/.skills/users/file-to-markdown/SKILL.md
+++ b/.skills/users/file-to-markdown/SKILL.md
@@ -151,11 +151,11 @@ embedded content in: email attachments (`.eml`, `.msg`),
archive members
(`.zip`), PDF attachments, embedded objects/images in any office format
(an inline pasted picture counts), and so on.
-What lands on disk: a `<name>-embed/` directory of the embedded files
-**renumbered** (`00000001.jpg`, ...) — original names are not preserved on
-disk; they're in the rmeta output's `tk:resource-name` per entry, so keep
-the sibling `<name>.json` (an rmeta-shaped metadata dump `-z` also writes)
-if you need to map numbers back to names.
+What lands on disk: a `<name>/` Frictionless package — `datapackage.json`
+(SHA-256 per file), `metadata.json`, and the embedded files **renumbered**
+under `unpacked/` (`00000001.jpg`, ...). Original names are not preserved on
+disk; they are in each rmeta entry's `tk:resource-name`, so keep the sibling
+`<name>.json` (an rmeta-shaped dump `-z` also writes) to map numbers back.
`-z`/`-Z` route through Tika Pipes mode rather than the fast synchronous
path the other flags use — expect several seconds and a burst of
diff --git a/CHANGES.txt b/CHANGES.txt
index ba87b8d77f..dc63817378 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,10 @@
Release 4.2.0 - unreleased
+ * Unpack output is a Frictionless Data Package by default on every surface:
tika-server
+ /unpack, tika-app -z/-Z and pipes UNPACK mode, where it is one
<key>-frictionless.zip per
+ container unless outputMode is DIRECTORY. Set unpack-config.outputFormat
to REGULAR
+ (--unpack-format=REGULAR) for the pre-4.2 flat layout (TIKA-4681).
+
* Pipes plugins load only from the zips in plugin-roots; a plugin jar on
the classpath
needs -Dtika.plugins.classpath=true, which PipesForkParser sets on its
fork when it
has no plugins directory. Dev mode loads an exploded target/classes
(TIKA-4918).
diff --git a/docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc
b/docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc
index c1dc16d887..08b41395c1 100644
--- a/docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc
+++ b/docs/modules/ROOT/pages/advanced/integration-testing/tika-app.adoc
@@ -328,27 +328,30 @@ java -jar tika-app.jar -a
--config=/tmp/tika-app-test/my-config.json
*Expected:* Same as Test 21 but using explicit async flag.
-=== Test 23: Unpack with Frictionless Format
+=== Test 23: Unpack (Frictionless package, the default)
[source,bash]
----
mkdir -p /tmp/tika-app-test/frictionless-out
-java -jar tika-app.jar -Z --extract-dir=/tmp/tika-app-test/frictionless-out
--unpack-format=FRICTIONLESS --unpack-include-metadata
test_recursive_embedded.docx
-ls /tmp/tika-app-test/frictionless-out
+java -jar tika-app.jar -Z --extract-dir=/tmp/tika-app-test/frictionless-out
test_recursive_embedded.docx
+ls -R /tmp/tika-app-test/frictionless-out
----
-*Expected:* Extracts embedded files in Frictionless data package format with
metadata.json.
+*Expected:* `test_recursive_embedded.docx.json` plus a
`test_recursive_embedded.docx/`
+directory holding `datapackage.json`, `metadata.json` and `unpacked/`.
-=== Test 24: Unpack to Directory (not zipped)
+=== Test 24: Unpack zipped, and the pre-4.2 flat layout
[source,bash]
----
-mkdir -p /tmp/tika-app-test/unpack-dir-out
-java -jar tika-app.jar -Z --extract-dir=/tmp/tika-app-test/unpack-dir-out
--unpack-mode=DIRECTORY test_recursive_embedded.docx
-ls -R /tmp/tika-app-test/unpack-dir-out
+mkdir -p /tmp/tika-app-test/unpack-zip-out
/tmp/tika-app-test/unpack-regular-out
+java -jar tika-app.jar -Z --extract-dir=/tmp/tika-app-test/unpack-zip-out
--unpack-mode=ZIPPED test_recursive_embedded.docx
+java -jar tika-app.jar -Z --extract-dir=/tmp/tika-app-test/unpack-regular-out
--unpack-format=REGULAR test_recursive_embedded.docx
+ls -R /tmp/tika-app-test/unpack-zip-out /tmp/tika-app-test/unpack-regular-out
----
-*Expected:* Extracts embedded files to directory structure instead of zipped.
+*Expected:* One `test_recursive_embedded.docx-frictionless.zip` in the first
directory;
+loose files under `test_recursive_embedded.docx-embed/` with no manifest in
the second.
=== Test 25: Batch with Multiple Workers
diff --git
a/docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc
b/docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc
index 5c18e980a8..414bb770f1 100644
--- a/docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc
+++ b/docs/modules/ROOT/pages/advanced/integration-testing/tika-server.adoc
@@ -158,8 +158,8 @@ curl -s -X PUT -T test_recursive_embedded.docx
http://localhost:9998/unpack/all
unzip -l /tmp/unpack.zip
----
-*Expected:* ZIP file containing the extracted embedded files, a
`*.metadata.json` entry
-per file, and the original container document.
+*Expected:* A Frictionless package: `datapackage.json`, `metadata.json`, and
the embedded
+files plus the original container (`0.docx`) under `unpacked/`.
=== Test 13: GET /parsers
diff --git a/docs/modules/ROOT/pages/pipes/unpack-config.adoc
b/docs/modules/ROOT/pages/pipes/unpack-config.adoc
index af8560ae29..9ac0881ca7 100644
--- a/docs/modules/ROOT/pages/pipes/unpack-config.adoc
+++ b/docs/modules/ROOT/pages/pipes/unpack-config.adoc
@@ -50,13 +50,16 @@ live inside that block:
"parse-context": {
"unpack-config": {
"maxUnpackBytes": 104857600,
- "zipEmbeddedFiles": true
+ "outputMode": "ZIPPED"
}
}
}
----
-This extracts both metadata (like `RMETA` mode) and embedded document bytes.
+This extracts both metadata (like `RMETA` mode) and embedded document bytes.
Since 4.2 the
+output is a <<_frictionless_data_package_output,Frictionless Data Package>> on
every surface,
+one `<key>-frictionless.zip` per container unless `outputMode` is `DIRECTORY`;
+`"outputFormat": "REGULAR"` restores the pre-4.2 flat layout.
[NOTE]
====
@@ -95,7 +98,7 @@ immediately capped.
|`zipEmbeddedFiles`
|boolean
|`false`
-|Collect all embedded files into a single ZIP archive, emitted at the
container's emit key plus `-embedded.zip`.
+|`REGULAR` only: collect all embedded files into a single ZIP archive, emitted
at the container's emit key plus `-embedded.zip`. Frictionless packaging is
`outputMode`.
|`includeMetadata`
|boolean
@@ -133,9 +136,9 @@ immediately capped.
|Custom base path when `keyBaseStrategy=CUSTOM`.
|`outputFormat`
-|REGULAR, FRICTIONLESS
-|`REGULAR`
-|Output format for the ZIP archive. See
<<_frictionless_data_package_output,Frictionless Data Package Output>>.
+|FRICTIONLESS, REGULAR
+|`FRICTIONLESS`
+|`FRICTIONLESS` is a <<_frictionless_data_package_output,Frictionless Data
Package>>; `REGULAR` is the pre-4.2 flat layout (files at the root, no
manifest).
|`outputMode`
|ZIPPED, DIRECTORY
@@ -150,9 +153,24 @@ immediately capped.
== Examples
-=== ZIP Output with Metadata
+=== One package per container, with the original
-Collect all embedded files into a ZIP with metadata:
+[source,json]
+----
+{
+ "pipes": {
+ "parseMode": "UNPACK"
+ },
+ "parse-context": {
+ "unpack-config": {
+ "outputMode": "ZIPPED",
+ "includeOriginal": true
+ }
+ }
+}
+----
+
+=== The pre-4.2 flat ZIP with sidecars
[source,json]
----
@@ -162,6 +180,7 @@ Collect all embedded files into a ZIP with metadata:
},
"parse-context": {
"unpack-config": {
+ "outputFormat": "REGULAR",
"zipEmbeddedFiles": true,
"includeMetadata": true,
"includeOriginal": true
@@ -194,6 +213,9 @@ Collect all embedded files into a ZIP with metadata:
Produces names like `document-embed-00000001.pdf`. Under the `DEFAULT`
strategy the same
`zeroPadName`/`suffixStrategy` settings would produce
`<containerKey>-embed/00000001.pdf`.
+These keys are REGULAR output's; a Frictionless package sits at
`<containerKey>/` (or
+`<containerKey>-frictionless.zip`) and names its files
`unpacked/<id><suffix>`, so only
+`zeroPadName` and `suffixStrategy` show there.
== Suffix Strategies
@@ -225,9 +247,8 @@ The UNPACK mode can output files in
https://frictionlessdata.io/[Frictionless Da
a standard for packaging data files with their metadata. This format includes
a `datapackage.json`
manifest with file checksums and MIME types, making it easy to verify and
process extracted files.
-=== Enabling Frictionless Output
-
-Set `outputFormat` to `FRICTIONLESS` in your `unpack-config`:
+It is the default since 4.2 (TIKA-4681); 4.0 and 4.1 emitted the flat
`REGULAR` layout
+unless `outputFormat` said otherwise. To get that layout back:
[source,json]
----
@@ -237,8 +258,7 @@ Set `outputFormat` to `FRICTIONLESS` in your
`unpack-config`:
},
"parse-context": {
"unpack-config": {
- "outputFormat": "FRICTIONLESS",
- "includeMetadata": true
+ "outputFormat": "REGULAR"
}
}
}
@@ -246,7 +266,7 @@ Set `outputFormat` to `FRICTIONLESS` in your
`unpack-config`:
=== Output Structure
-When using Frictionless output format, the ZIP archive contains:
+A zipped package contains:
[source]
----
@@ -300,22 +320,23 @@ Naming the handler in both the path and a `config` part
is a `400`, as on `/tika
== What each surface produces
-The five knobs interact differently per surface. One default changes from
4.0.0: a
+The five knobs interact differently per surface. Two defaults changed since
4.0.0: a
Frictionless package carries `metadata.json` unless `includeMetadata` is
`false` (4.0.0 left
-it out unless `includeFullMetadata` was `true`). REGULAR output is as it was.
+it out unless `includeFullMetadata` was `true`), and since 4.2 Frictionless is
the format
+on every surface. REGULAR output is as it was.
[cols="1,2,2,2"]
|===
| |tika-server `/unpack` |tika-app `-z` / `-Z` |pipes JSON config
|format
-|REGULAR; set `outputFormat` in the server config for Frictionless
-|REGULAR; `--unpack-format=FRICTIONLESS`
+|Frictionless; `outputFormat: REGULAR` in the server config for the flat zip
+|Frictionless; `--unpack-format=REGULAR`
|`outputFormat`
|packaging
|always one zip (`ZIPPED` is pinned; an HTTP response is one body)
-|loose files; `DIRECTORY` when Frictionless is asked for without
`--unpack-mode`
+|loose files (`DIRECTORY`); `--unpack-mode=ZIPPED` for one zip
|`outputMode` governs Frictionless, `zipEmbeddedFiles` governs REGULAR
|depth
@@ -350,18 +371,18 @@ Output trees for `doc.docx` with three attachments:
[source]
----
-REGULAR, tika-server (one zip) REGULAR, tika-app -Z (loose)
+tika-server (one zip) tika-app -Z (loose)
+ datapackage.json out/doc.docx.json
+ metadata.json out/doc.docx/datapackage.json
+ unpacked/1.emf ... out/doc.docx/metadata.json
+ + with /all: unpacked/0.docx out/doc.docx/unpacked/1.emf ...
+
+REGULAR, tika-server (one zip) REGULAR, tika-app -Z
--unpack-format=REGULAR
1.emf out/doc.docx.json
2.zip out/doc.docx-embed/1.emf
3.bin out/doc.docx-embed/2.zip
+ with /all: 0.docx and one out/doc.docx-embed/3.bin
<name>.metadata.json per file
-
-FRICTIONLESS, tika-server (one zip) FRICTIONLESS, tika-app -Z
--unpack-format=FRICTIONLESS
- datapackage.json out/doc.docx.json
- metadata.json out/doc.docx/datapackage.json
- unpacked/1.emf ... out/doc.docx/metadata.json
- + with /all: unpacked/0.docx out/doc.docx/unpacked/1.emf ...
----
The top-level `doc.docx.json` on the command line is the normal pipes output
for the parse;
@@ -377,13 +398,12 @@ The `datapackage.json` file contains:
=== CLI Usage
-Extract files in Frictionless format using the CLI. The `-Z` flag turns on
recursive
-unpack (the Pipes-mode counterpart of standard-mode `-z`), and `-i`/`-o` are
the
-Pipes input/output directories:
+The `-Z` flag turns on recursive unpack (the Pipes-mode counterpart of
standard-mode `-z`),
+and `-i`/`-o` are the Pipes input/output directories:
[source,bash]
----
-java -jar tika-app.jar -Z --unpack-format=FRICTIONLESS -i /path/to/input -o
/path/to/output
+java -jar tika-app.jar -Z -i /path/to/input -o /path/to/output
----
NOTE: `-i` expects a directory of containers to unpack, not a single file. For
diff --git a/docs/modules/ROOT/pages/using-tika/cli/index.adoc
b/docs/modules/ROOT/pages/using-tika/cli/index.adoc
index 74fa7ad432..1835b30749 100644
--- a/docs/modules/ROOT/pages/using-tika/cli/index.adoc
+++ b/docs/modules/ROOT/pages/using-tika/cli/index.adoc
@@ -396,12 +396,12 @@ through every level. Both accept a single file or a
directory as input.
|Output directory for the extracted bytes; equivalent to `-o`.
|`--unpack-format=<format>`
-|Output format: `REGULAR` (default) or `FRICTIONLESS`.
+|Output format: `FRICTIONLESS` (default) or `REGULAR`, the pre-4.2 flat layout.
|`--unpack-mode=<mode>`
-|Output mode: `ZIPPED` (default) or `DIRECTORY`. With
`--unpack-format=FRICTIONLESS` and no
-mode, `DIRECTORY`: the package is laid out under `<out>/<input>/`
(`datapackage.json`,
-`unpacked/...`) rather than zipped.
+|Output mode: `DIRECTORY` (default) or `ZIPPED`. `DIRECTORY` lays the package
out under
+`<out>/<input>/` (`datapackage.json`, `metadata.json`, `unpacked/...`);
`ZIPPED` writes
+`<out>/<input>-frictionless.zip`.
|`--unpack-include-metadata`
|Metadata for every extracted file. A Frictionless package carries
`metadata.json` by default;
@@ -411,8 +411,9 @@ this adds a `.metadata.json` sidecar per file to REGULAR
zip output.
A `-c`/`--config` file's `unpack-config` is honoured and the flags above are
applied on top
of it. That is the only way to get the source document itself into the output
(`"unpack-config": {"includeOriginal": true}`); no flag does it. File naming
stays the
-CLI's own (`<input>-embed/00000001.<ext>`, detected suffixes) regardless of
the file's
-`suffixStrategy`, `zeroPadName`, `embeddedIdPrefix` or `keyBaseStrategy`. See
+CLI's own (`<input>/unpacked/00000001.<ext>`, or
`<input>-embed/00000001.<ext>` under
+`REGULAR`, detected suffixes, loose files unless `--unpack-mode=ZIPPED`)
regardless of the
+file's `suffixStrategy`, `zeroPadName`, `embeddedIdPrefix`, `keyBaseStrategy`
or `outputMode`. See
xref:pipes/unpack-config.adoc#_what_each_surface_produces[what each surface
produces].
=== Examples
diff --git a/docs/modules/ROOT/pages/using-tika/server/index.adoc
b/docs/modules/ROOT/pages/using-tika/server/index.adoc
index 8b1676a6f8..bc27b79b17 100644
--- a/docs/modules/ROOT/pages/using-tika/server/index.adoc
+++ b/docs/modules/ROOT/pages/using-tika/server/index.adoc
@@ -577,7 +577,7 @@ before the exception are discarded. Known gap.
By default exception detail is returned in full. Stack traces and their
messages can contain the
spooled file's path, the source filename, and fragments of the document. That
detail appears in
`tk:exception:*` metadata on **successful** parses as well as in error bodies
— including inside
-the `*.metadata.json` entries of an `/unpack` zip.
+the `metadata.json` of an `/unpack` zip.
One config setting governs every one of those channels — the metadata values,
every error body
built from an exception (`422`, `500`; a `400` echoes the caller's own request
problem, `415` and
diff --git a/tika-app/src/main/java/org/apache/tika/cli/TikaCLI.java
b/tika-app/src/main/java/org/apache/tika/cli/TikaCLI.java
index d1c3d823d2..f52cdfab27 100644
--- a/tika-app/src/main/java/org/apache/tika/cli/TikaCLI.java
+++ b/tika-app/src/main/java/org/apache/tika/cli/TikaCLI.java
@@ -995,9 +995,8 @@ public class TikaCLI {
out.println(" --content-only Output only extracted
content (no JSON wrapper); implies --concatenate");
out.println(" --on-exists Behavior when an output
file exists: exception (default), replace, skip");
out.println(" -Z Recursively unpack all the
attachments, too");
- out.println(" --unpack-format=<format> Output format: REGULAR
(default) or FRICTIONLESS");
- out.println(" --unpack-mode=<mode> Output mode: ZIPPED
(default) or DIRECTORY;");
- out.println(" DIRECTORY when
--unpack-format=FRICTIONLESS and no mode is given");
+ out.println(" --unpack-format=<format> Output format:
FRICTIONLESS (default) or REGULAR");
+ out.println(" --unpack-mode=<mode> Output mode: DIRECTORY
(default: loose files) or ZIPPED");
out.println(" --unpack-include-metadata Metadata for every
extracted file. A Frictionless package");
out.println(" carries metadata.json by
default; this adds per-file");
out.println(" sidecars to REGULAR zip
output");
diff --git a/tika-app/src/test/java/org/apache/tika/cli/TikaCLITest.java
b/tika-app/src/test/java/org/apache/tika/cli/TikaCLITest.java
index 61efb3eab6..c105d56c82 100644
--- a/tika-app/src/test/java/org/apache/tika/cli/TikaCLITest.java
+++ b/tika-app/src/test/java/org/apache/tika/cli/TikaCLITest.java
@@ -371,10 +371,10 @@ public class TikaCLITest {
String[] expectedChildren = new String[]{
"testPDFPackage.pdf.json",
//the first two test that the default single file config is
working
- "testPDFPackage.pdf-embed/00000001.bin",
- "testPDFPackage.pdf-embed/00000002.jpg",
- "testPDFPackage.pdf-embed/00000003.pdf",
- "testPDFPackage.pdf-embed/00000004.pdf"};
+ "testPDFPackage.pdf/unpacked/00000001.bin",
+ "testPDFPackage.pdf/unpacked/00000002.jpg",
+ "testPDFPackage.pdf/unpacked/00000003.pdf",
+ "testPDFPackage.pdf/unpacked/00000004.pdf"};
testRecursiveUnpack("testPDFPackage.pdf", expectedChildren, 2);
}
@@ -383,15 +383,15 @@ public class TikaCLITest {
// TODO: The .bin extensions for embedded .msg files are wrong - they
should be .msg
// CONTENT_TYPE is not being set for embedded documents - see
~/Desktop/unpack-discussion/mime-todo.txt
String[] expectedChildren = new String[]{"testPST.pst.json",
- "testPST.pst-embed/00000007.bin",
- "testPST.pst-embed/00000001.bin",
- "testPST.pst-embed/00000008.bin",
- "testPST.pst-embed/00000004.bin",
- "testPST.pst-embed/00000003.bin",
- "testPST.pst-embed/00000002.bin",
- "testPST.pst-embed/00000005.bin",
- "testPST.pst-embed/00000009.docx",
- "testPST.pst-embed/00000006.bin"};
+ "testPST.pst/unpacked/00000007.bin",
+ "testPST.pst/unpacked/00000001.bin",
+ "testPST.pst/unpacked/00000008.bin",
+ "testPST.pst/unpacked/00000004.bin",
+ "testPST.pst/unpacked/00000003.bin",
+ "testPST.pst/unpacked/00000002.bin",
+ "testPST.pst/unpacked/00000005.bin",
+ "testPST.pst/unpacked/00000009.docx",
+ "testPST.pst/unpacked/00000006.bin"};
testRecursiveUnpack("testPST.pst", expectedChildren, 2);
try (Reader reader =
Files.newBufferedReader(extractDir.resolve("testPST.pst.json"))) {
List<Metadata> metadataList = JsonMetadataList.fromJson(reader);
@@ -449,30 +449,30 @@ public class TikaCLITest {
@Test
public void testExtractSimple() throws Exception {
- assertShallow("coffee.xls", 9, "coffee.xls.json",
"coffee.xls-embed/00000001.emf",
- "coffee.xls-embed/00000006.cdx",
"coffee.xls-embed/00000005.png");
+ assertShallow("coffee.xls", 11, "coffee.xls.json",
"coffee.xls/unpacked/00000001.emf",
+ "coffee.xls/unpacked/00000006.cdx",
"coffee.xls/unpacked/00000005.png");
}
@Test
public void testExtractAbsolute() throws Exception {
- assertShallow("testZip_absolutePath.zip", 3,
"testZip_absolutePath.zip.json",
- "testZip_absolutePath.zip-embed/00000001.bin");
+ assertShallow("testZip_absolutePath.zip", 5,
"testZip_absolutePath.zip.json",
+ "testZip_absolutePath.zip/unpacked/00000001.bin");
}
@Test
public void testExtractRelative() throws Exception {
- assertShallow("testZip_relative.zip", 2, "testZip_relative.zip.json");
+ assertShallow("testZip_relative.zip", 4, "testZip_relative.zip.json");
}
@Test
public void testExtractOverlapping() throws Exception {
// overlapping names are handled by numbering
- assertShallow("testZip_overlappingNames.zip", 3,
"testZip_overlappingNames.zip.json");
+ assertShallow("testZip_overlappingNames.zip", 5,
"testZip_overlappingNames.zip.json");
}
@Test
public void testExtract0x00() throws Exception {
- assertShallow("testZip_zeroByte.zip", 2, "testZip_zeroByte.zip.json");
+ assertShallow("testZip_zeroByte.zip", 4, "testZip_zeroByte.zip.json");
}
private void testRecursiveUnpack(String targetFile, String[]
expectedChildrenFileNames) throws Exception {
@@ -563,8 +563,8 @@ public class TikaCLITest {
Set<String> fileNames = shallowOutputOf("testWithSubdirs.zip");
assertTrue(fileNames.stream().anyMatch(f -> f.endsWith(".json")),
"Should have a .json metadata file, got: " + fileNames);
- assertTrue(fileNames.stream().anyMatch(f -> f.contains("-embed/")),
- "Should have extracted embedded files in -embed/ directory,
got: " + fileNames);
+ assertTrue(fileNames.stream().anyMatch(f -> f.contains("/unpacked/")),
+ "Should have extracted embedded files under unpacked/, got: "
+ fileNames);
}
@Test
@@ -579,7 +579,7 @@ public class TikaCLITest {
private static Set<String> shallowOutputOf(String inputName) {
Set<String> mine = new HashSet<>();
for (String f : shallowOutput) {
- if (f.equals(inputName + ".json") || f.startsWith(inputName +
"-embed/")) {
+ if (f.equals(inputName + ".json") || f.startsWith(inputName +
"/")) {
mine.add(f);
}
}
@@ -626,13 +626,14 @@ public class TikaCLITest {
// Should have extracted files in the specified directory, not current
dir
assertTrue(fileNames.stream().anyMatch(f -> f.endsWith(".json")),
"Should have a .json metadata file in extractDir, got: " +
fileNames);
- assertTrue(fileNames.stream().anyMatch(f -> f.contains("-embed/")),
+ assertTrue(fileNames.stream().anyMatch(f -> f.contains("/unpacked/")),
"Should have extracted embedded files in extractDir, got: " +
fileNames);
}
+ /** TIKA-4681: the default is a Frictionless package laid out as loose
files. */
@Test
- public void testFrictionlessWithoutModeIsADirectory() throws Exception {
- Set<String> fileNames = unpack("-Z", "--unpack-format=FRICTIONLESS");
+ public void testDefaultIsALooseFrictionlessPackage() throws Exception {
+ Set<String> fileNames = unpack("-Z");
assertTrue(fileNames.stream().anyMatch(f ->
f.endsWith("/datapackage.json")),
"package should be laid out as a directory, got: " +
fileNames);
assertTrue(fileNames.stream().anyMatch(f -> f.contains("/unpacked/")),
fileNames.toString());
@@ -642,9 +643,35 @@ public class TikaCLITest {
"a Frictionless package carries metadata.json by default: " +
fileNames);
}
+ @Test
+ public void testZippedModeWritesOnePackageZip() throws Exception {
+ Set<String> fileNames = unpack("-Z", "--unpack-mode=ZIPPED");
+
assertTrue(fileNames.contains("test_recursive_embedded.docx-frictionless.zip"),
fileNames.toString());
+ assertFalse(fileNames.stream().anyMatch(f -> f.contains("/unpacked/")),
+ "ZIPPED must not also write loose files: " + fileNames);
+ }
+
+ @Test
+ public void testRegularFormatIsTheFlatLayout() throws Exception {
+ Set<String> fileNames = unpack("-Z", "--unpack-format=REGULAR");
+ assertTrue(fileNames.stream().anyMatch(f ->
f.matches("test_recursive_embedded\\.docx-embed/0+1\\.[^/]+")),
+ fileNames.toString());
+ assertFalse(fileNames.stream().anyMatch(f ->
f.endsWith("datapackage.json") || f.contains("/unpacked/")),
+ "REGULAR has no manifest and no unpacked/: " + fileNames);
+ }
+
+ /** --unpack-mode is the CLI's packaging knob in both formats. */
+ @Test
+ public void testRegularZippedWritesOneFlatZip() throws Exception {
+ Set<String> fileNames = unpack("-Z", "--unpack-format=REGULAR",
"--unpack-mode=ZIPPED");
+
assertTrue(fileNames.contains("test_recursive_embedded.docx-embedded.zip"),
fileNames.toString());
+ assertFalse(fileNames.stream().anyMatch(f -> f.contains("-embed/")),
+ "ZIPPED must not also write loose files: " + fileNames);
+ }
+
@Test
public void testFrictionlessIncludeMetadata() throws Exception {
- Set<String> fileNames = unpack("-Z", "--unpack-format=FRICTIONLESS",
"--unpack-include-metadata");
+ Set<String> fileNames = unpack("-Z", "--unpack-include-metadata");
String metadataJson = fileNames.stream().filter(f ->
f.endsWith("/metadata.json")).findFirst()
.orElseThrow(() -> new AssertionError("no metadata.json in " +
fileNames));
String content = Files.readString(extractDir.resolve(metadataJson));
@@ -657,10 +684,10 @@ public class TikaCLITest {
@Test
public void testConfigFileUnpackConfigIsHonoured() throws Exception {
Set<String> fileNames = unpack("-Z", "--config=" + CONFIGS_DIR +
"/tika-config-unpack-original.json");
- assertTrue(fileNames.stream().anyMatch(f ->
f.matches(".*-embed/0+\\.[^/]+")),
+ assertTrue(fileNames.stream().anyMatch(f ->
f.matches(".*/unpacked/0+\\.[^/]+")),
"includeOriginal from -c should add the container as
00000000.<ext>: " + fileNames);
- assertTrue(fileNames.stream().anyMatch(f ->
f.matches(".*-embed/0+1\\.[^/]+")),
- "-c must not change the CLI's file naming: " + fileNames);
+ assertTrue(fileNames.stream().anyMatch(f ->
f.matches(".*/unpacked/0+1\\.[^/]+")),
+ "-c must not change the CLI's file naming or packaging: " +
fileNames);
}
private Set<String> unpack(String... flags) throws Exception {
diff --git
a/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
b/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
index 5b10176425..9311b1abc6 100644
---
a/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
+++
b/tika-pipes/tika-async-cli/src/main/java/org/apache/tika/async/cli/TikaAsyncCLI.java
@@ -85,10 +85,9 @@ public class TikaAsyncCLI {
options.addOption(null, "concatenate", false, "concatenate content
from all embedded documents into a single content field");
options.addOption(null, "content-only", false, "output only extracted
content (no metadata, no JSON wrapper); implies --concatenate");
options.addOption(null, "unpack-format", true,
- "output format for unpacking: REGULAR (default) or
FRICTIONLESS");
+ "output format for unpacking: FRICTIONLESS (default) or
REGULAR");
options.addOption(null, "unpack-mode", true,
- "output mode for unpacking: ZIPPED (default) or DIRECTORY;
DIRECTORY when "
- + "--unpack-format FRICTIONLESS is given with no
mode");
+ "output mode for unpacking: DIRECTORY (default: loose files)
or ZIPPED");
options.addOption(null, "unpack-include-metadata", false,
"metadata for every extracted file: a Frictionless package
carries metadata.json "
+ "by default; this adds per-file sidecars to REGULAR
zip output");
@@ -406,18 +405,17 @@ public class TikaAsyncCLI {
config.setZeroPadName(8);
config.setKeyBaseStrategy(UnpackConfig.KEY_BASE_STRATEGY.DEFAULT);
- // Command-line flags win over the file
+ // Packaging is the CLI's too: loose files unless --unpack-mode asks
for a zip
+ config.setOutputMode(UnpackConfig.OUTPUT_MODE.DIRECTORY);
+
if (asyncConfig.getUnpackFormat() != null) {
config.setOutputFormat(UnpackConfig.OUTPUT_FORMAT.valueOf(asyncConfig.getUnpackFormat()));
- // Loose files are what -z otherwise writes; a zip only when asked
for
- if (asyncConfig.getUnpackMode() == null
- && config.getOutputFormat() ==
UnpackConfig.OUTPUT_FORMAT.FRICTIONLESS) {
- config.setOutputMode(UnpackConfig.OUTPUT_MODE.DIRECTORY);
- }
}
if (asyncConfig.getUnpackMode() != null) {
config.setOutputMode(UnpackConfig.OUTPUT_MODE.valueOf(asyncConfig.getUnpackMode()));
}
+ // REGULAR packages on zipEmbeddedFiles, not outputMode
+ config.setZipEmbeddedFiles(config.getOutputMode() ==
UnpackConfig.OUTPUT_MODE.ZIPPED);
if (asyncConfig.isUnpackIncludeMetadata()) {
config.setIncludeMetadata(true);
}
diff --git
a/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/AsyncProcessorTest.java
b/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/AsyncProcessorTest.java
index a0eb2d8bb7..c74045da87 100644
---
a/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/AsyncProcessorTest.java
+++
b/tika-pipes/tika-async-cli/src/test/java/org/apache/tika/async/cli/AsyncProcessorTest.java
@@ -125,6 +125,7 @@ public class AsyncProcessorTest extends TikaTest {
unpackConfig.setEmitter("fse-bytes");
unpackConfig.setSuffixStrategy(UnpackConfig.SUFFIX_STRATEGY.NONE);
unpackConfig.setEmbeddedIdPrefix("-");
+ unpackConfig.setOutputMode(UnpackConfig.OUTPUT_MODE.DIRECTORY);
ParseContext parseContext = new ParseContext();
parseContext.set(ParseMode.class, ParseMode.UNPACK);
parseContext.set(UnpackConfig.class, unpackConfig);
@@ -144,10 +145,10 @@ public class AsyncProcessorTest extends TikaTest {
}
processor.close();
- String container =
Files.readString(bytesOutputDir.resolve("emit-1-embed/0"));
+ String container =
Files.readString(bytesOutputDir.resolve("emit-1/unpacked/0"));
assertContains("\"dc:creator\">Nikolai Lobachevsky", container);
- String xmlEmbedded =
Files.readString(bytesOutputDir.resolve("emit-1-embed/1"));
+ String xmlEmbedded =
Files.readString(bytesOutputDir.resolve("emit-1/unpacked/1"));
assertContains("name=\"dc:creator\"", xmlEmbedded);
assertContains(">embeddedAuthor</metadata>", xmlEmbedded);
diff --git
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackConfig.java
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackConfig.java
index c235daf1a0..bc02d5ce6a 100644
---
a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackConfig.java
+++
b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/extractor/UnpackConfig.java
@@ -83,7 +83,7 @@ public class UnpackConfig implements Serializable {
*/
public enum OUTPUT_FORMAT {
/**
- * Regular output - embedded files emitted individually or as simple
zip
+ * The pre-4.2 layout: embedded files emitted individually or as a
flat zip
*/
REGULAR,
/**
@@ -149,7 +149,7 @@ public class UnpackConfig implements Serializable {
private long maxUnpackBytes = DEFAULT_MAX_UNPACK_BYTES;
// Frictionless Data Package options
- private OUTPUT_FORMAT outputFormat = OUTPUT_FORMAT.REGULAR;
+ private OUTPUT_FORMAT outputFormat = OUTPUT_FORMAT.FRICTIONLESS;
private OUTPUT_MODE outputMode = OUTPUT_MODE.ZIPPED;
/**
@@ -321,8 +321,8 @@ public class UnpackConfig implements Serializable {
/**
* Get the output format for UNPACK mode.
- * REGULAR is the default (existing behavior).
- * FRICTIONLESS creates a Frictionless Data Package with datapackage.json
manifest.
+ * FRICTIONLESS (the default since 4.2) is a Frictionless Data Package
with a datapackage.json manifest.
+ * REGULAR is the pre-4.2 flat layout.
*/
public OUTPUT_FORMAT getOutputFormat() {
return outputFormat;
diff --git
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/extractor/UnpackConfigDeprecatedFlagsTest.java
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/extractor/UnpackConfigDeprecatedFlagsTest.java
index 0cdeeab749..45bf43a9f7 100644
---
a/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/extractor/UnpackConfigDeprecatedFlagsTest.java
+++
b/tika-pipes/tika-pipes-core/src/test/java/org/apache/tika/pipes/core/extractor/UnpackConfigDeprecatedFlagsTest.java
@@ -65,7 +65,9 @@ public class UnpackConfigDeprecatedFlagsTest {
UnpackConfig c = MAPPER.readValue("{\"outputFormat\":
\"FRICTIONLESS\"}", UnpackConfig.class);
assertTrue(c.writesMetadata());
assertFalse(MAPPER.readValue("{\"outputFormat\": \"REGULAR\"}",
UnpackConfig.class).writesMetadata());
- assertFalse(new UnpackConfig().writesMetadata());
+ // TIKA-4681: a fresh config is a Frictionless package
+ assertEquals(UnpackConfig.OUTPUT_FORMAT.FRICTIONLESS, new
UnpackConfig().getOutputFormat());
+ assertTrue(new UnpackConfig().writesMetadata());
UnpackConfig optOut = MAPPER.readValue(
"{\"outputFormat\": \"FRICTIONLESS\", \"includeMetadata\":
false}", UnpackConfig.class);
diff --git
a/tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/FrictionlessUnpackTest.java
b/tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/FrictionlessUnpackTest.java
index c64ab81bad..19e6d7cc0a 100644
---
a/tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/FrictionlessUnpackTest.java
+++
b/tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/FrictionlessUnpackTest.java
@@ -17,6 +17,7 @@
package org.apache.tika.pipes.core;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -126,6 +127,31 @@ public class FrictionlessUnpackTest {
return parseContext;
}
+ /** TIKA-4681: an UnpackConfig that says nothing about format is a
Frictionless package. */
+ @Test
+ public void testDefaultFormatIsFrictionless() throws Exception {
+ ParseContext parseContext = new ParseContext();
+ parseContext.set(ParseMode.class, ParseMode.UNPACK);
+ UnpackConfig unpackConfig = new UnpackConfig();
+ unpackConfig.setEmitter(EMITTER_NAME);
+ parseContext.set(UnpackConfig.class, unpackConfig);
+
+ PipesResult pipesResult = process("defaultFormat",
TEST_DOC_WITH_EMBEDDED, parseContext);
+ assertTrue(pipesResult.isSuccess(), "Status: " + pipesResult.status()
+ + ", Message: " + pipesResult.message());
+
+ Path outputDir = outputDir("defaultFormat");
+ List<String> names;
+ try (Stream<Path> files = Files.list(outputDir)) {
+ names = files.map(p -> p.getFileName().toString()).toList();
+ }
+ assertTrue(names.contains(TEST_DOC_WITH_EMBEDDED +
"-frictionless.zip"), names.toString());
+ assertFalse(names.contains(TEST_DOC_WITH_EMBEDDED + "-embedded.zip"),
names.toString());
+ Set<String> entries = zipEntries(onlyFrictionlessZip(outputDir));
+ assertTrue(entries.contains("datapackage.json"), entries.toString());
+ assertTrue(entries.contains("metadata.json"), "a package carries
metadata.json by default: " + entries);
+ }
+
@Test
public void testFrictionlessZippedOutput() throws Exception {
// ZIPPED structure, datapackage.json schema, and SHA256 correctness,
from one parse
diff --git
a/tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/UnpackModeTest.java
b/tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/UnpackModeTest.java
index fe82539b7f..755d10fe44 100644
---
a/tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/UnpackModeTest.java
+++
b/tika-pipes/tika-pipes-integration-tests/src/test/java/org/apache/tika/pipes/core/UnpackModeTest.java
@@ -29,7 +29,9 @@ import java.nio.file.Path;
import java.util.Enumeration;
import java.util.HashSet;
import java.util.List;
+import java.util.Map;
import java.util.Set;
+import java.util.TreeMap;
import java.util.stream.Stream;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
@@ -120,10 +122,10 @@ public class UnpackModeTest {
for (Metadata m : metadataList) {
assertNotNull(m.get("Content-Type"));
}
- List<Path> embedded = embeddedFiles(outputDir, testDocWithEmbedded);
+ Map<String, Long> embedded = embeddedFiles(outputDir,
testDocWithEmbedded);
assertEquals(4, embedded.size(), "embedded bytes: " + embedded);
- for (Path p : embedded) {
- assertTrue(Files.size(p) > 0, p.toString());
+ for (Map.Entry<String, Long> e : embedded.entrySet()) {
+ assertTrue(e.getValue() > 0, e.toString());
}
}
@@ -168,8 +170,7 @@ public class UnpackModeTest {
assertTrue(pipesResult.isSuccess(),
"UNPACK with custom UnpackConfig should succeed. Status: " +
pipesResult.status());
- List<String> names = embeddedFiles(outputDir("custom"),
testDocWithEmbedded).stream()
- .map(f -> f.getFileName().toString()).toList();
+ Set<String> names = embeddedFiles(outputDir("custom"),
testDocWithEmbedded).keySet();
assertEquals(4, names.size(), names.toString());
for (String name : names) {
assertTrue(name.matches("\\d{8}\\..+"), "zero-padded with detected
suffix: " + name);
@@ -191,7 +192,7 @@ public class UnpackModeTest {
assertTrue(pipesResult.isSuccess(),
"UNPACK with includeOriginal should succeed. Status: " +
pipesResult.status());
- List<Path> embedded = embeddedFiles(outputDir("includeOriginal"),
testDocWithEmbedded);
+ Map<String, Long> embedded =
embeddedFiles(outputDir("includeOriginal"), testDocWithEmbedded);
assertEquals(5, embedded.size(), "4 embedded + the original: " +
embedded);
}
@@ -260,6 +261,7 @@ public class UnpackModeTest {
UnpackConfig unpackConfig = new UnpackConfig();
unpackConfig.setEmitter(emitterName);
+ unpackConfig.setOutputFormat(UnpackConfig.OUTPUT_FORMAT.REGULAR);
unpackConfig.setZipEmbeddedFiles(true);
unpackConfig.setSuffixStrategy(UnpackConfig.SUFFIX_STRATEGY.DETECTED);
parseContext.set(UnpackConfig.class, unpackConfig);
@@ -286,6 +288,7 @@ public class UnpackModeTest {
UnpackConfig unpackConfig = new UnpackConfig();
unpackConfig.setEmitter(emitterName);
+ unpackConfig.setOutputFormat(UnpackConfig.OUTPUT_FORMAT.REGULAR);
unpackConfig.setZipEmbeddedFiles(true);
unpackConfig.setIncludeMetadataInZip(true);
unpackConfig.setSuffixStrategy(UnpackConfig.SUFFIX_STRATEGY.DETECTED);
@@ -315,6 +318,7 @@ public class UnpackModeTest {
UnpackConfig unpackConfig = new UnpackConfig();
unpackConfig.setEmitter(emitterName);
+ unpackConfig.setOutputFormat(UnpackConfig.OUTPUT_FORMAT.REGULAR);
unpackConfig.setZipEmbeddedFiles(true);
unpackConfig.setIncludeOriginal(true);
unpackConfig.setSuffixStrategy(UnpackConfig.SUFFIX_STRATEGY.DETECTED);
@@ -338,6 +342,7 @@ public class UnpackModeTest {
UnpackConfig unpackConfig = new UnpackConfig();
unpackConfig.setEmitter(emitterName);
+ unpackConfig.setOutputFormat(UnpackConfig.OUTPUT_FORMAT.REGULAR);
unpackConfig.setZipEmbeddedFiles(true);
parseContext.set(UnpackConfig.class, unpackConfig);
@@ -359,6 +364,7 @@ public class UnpackModeTest {
UnpackConfig unpackConfig = new UnpackConfig();
unpackConfig.setEmitter(emitterName);
unpackConfig.setMaxUnpackBytes(10L);
+ unpackConfig.setOutputMode(UnpackConfig.OUTPUT_MODE.DIRECTORY);
parseContext.set(UnpackConfig.class, unpackConfig);
PipesResult pipesResult = process("limited", testDocWithEmbedded,
testDocWithEmbedded + "-limited",
@@ -371,7 +377,7 @@ public class UnpackModeTest {
try (Stream<Path> files = Files.walk(outputDir("limited"))) {
totalBytesWritten = files
.filter(Files::isRegularFile)
- .filter(p -> !p.toString().endsWith(".json"))
+ .filter(p -> p.toString().replace('\\',
'/').contains("/unpacked/"))
.mapToLong(p -> {
try {
return Files.size(p);
@@ -403,10 +409,10 @@ public class UnpackModeTest {
assertTrue(pipesResult.isSuccess(),
"UNPACK with default maxUnpackBytes should succeed. Status: "
+ pipesResult.status());
- List<Path> embedded = embeddedFiles(outputDir("default"),
testDocWithEmbedded + "-default");
+ Map<String, Long> embedded = embeddedFiles(outputDir("default"),
testDocWithEmbedded + "-default");
assertEquals(4, embedded.size(), embedded.toString());
- for (Path p : embedded) {
- assertEquals(146, Files.size(p), p.toString());
+ for (Map.Entry<String, Long> e : embedded.entrySet()) {
+ assertEquals(146, e.getValue(), e.toString());
}
}
@@ -426,10 +432,10 @@ public class UnpackModeTest {
assertTrue(pipesResult.isSuccess(),
"UNPACK with unlimited maxUnpackBytes should succeed. Status:
" + pipesResult.status());
- List<Path> embedded = embeddedFiles(outputDir("unlimited"),
testDocWithEmbedded + "-unlimited");
+ Map<String, Long> embedded = embeddedFiles(outputDir("unlimited"),
testDocWithEmbedded + "-unlimited");
assertEquals(4, embedded.size(), embedded.toString());
- for (Path p : embedded) {
- assertEquals(146, Files.size(p), p.toString());
+ for (Map.Entry<String, Long> e : embedded.entrySet()) {
+ assertEquals(146, e.getValue(), e.toString());
}
}
@@ -463,13 +469,22 @@ public class UnpackModeTest {
}
}
- private static List<Path> embeddedFiles(Path outputDir, String emitKey)
throws IOException {
- Path dir = outputDir.resolve(emitKey + "-embed");
- if (!Files.isDirectory(dir)) {
- return List.of();
+ /** The default package is a ZIPPED Frictionless one: entry name under
unpacked/ -> size. */
+ private static Map<String, Long> embeddedFiles(Path outputDir, String
emitKey) throws IOException {
+ Path zipFile = outputDir.resolve(emitKey + "-frictionless.zip");
+ Map<String, Long> embedded = new TreeMap<>();
+ if (!Files.isRegularFile(zipFile)) {
+ return embedded;
}
- try (Stream<Path> files = Files.list(dir)) {
- return files.sorted().toList();
+ try (ZipFile zip = new ZipFile(zipFile.toFile())) {
+ Enumeration<? extends ZipEntry> entries = zip.entries();
+ while (entries.hasMoreElements()) {
+ ZipEntry entry = entries.nextElement();
+ if (entry.getName().startsWith("unpacked/") &&
!entry.isDirectory()) {
+
embedded.put(entry.getName().substring("unpacked/".length()), entry.getSize());
+ }
+ }
}
+ return embedded;
}
}
diff --git
a/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/resource/UnpackerResource.java
b/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/resource/UnpackerResource.java
index 321d9673ab..cdf96de978 100644
---
a/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/resource/UnpackerResource.java
+++
b/tika-server/tika-server-core/src/main/java/org/apache/tika/server/core/resource/UnpackerResource.java
@@ -89,23 +89,10 @@ import org.apache.tika.pipes.core.extractor.UnpackConfig;
* }
* </pre>
* <p>
- * <b>Frictionless Data Package Format:</b>
+ * <b>Output format:</b>
* <p>
- * To receive output in Frictionless Data Package format (with
datapackage.json manifest,
- * SHA256 hashes, and files in unpacked/ subdirectory), use:
- * <pre>
- * {
- * "parse-context": {
- * "unpack-config": {
- * "outputFormat": "FRICTIONLESS",
- * "outputMode": "ZIPPED",
- * "includeMetadata": true
- * }
- * }
- * }
- * </pre>
- * <p>
- * The Frictionless zip structure:
+ * The response is a Frictionless Data Package (datapackage.json manifest with
SHA256 hashes,
+ * files under unpacked/):
* <pre>
* output.zip
* ├── datapackage.json # Manifest with file list, SHA256 hashes,
mimetypes
@@ -115,6 +102,14 @@ import org.apache.tika.pipes.core.extractor.UnpackConfig;
* ├── 00000002.png
* └── ...
* </pre>
+ * The pre-4.2 flat zip (files at the root, no manifest) is still available:
+ * <pre>
+ * {
+ * "parse-context": {
+ * "unpack-config": { "outputFormat": "REGULAR" }
+ * }
+ * }
+ * </pre>
* <p>
* <b>Breaking Changes from Pre-4.0:</b>
* <ul>
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/ThumbnailPresetTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/ThumbnailPresetTest.java
index 7ea8881665..4ac707aa42 100644
---
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/ThumbnailPresetTest.java
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/ThumbnailPresetTest.java
@@ -29,6 +29,7 @@ import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.zip.ZipEntry;
@@ -163,10 +164,20 @@ public class ThumbnailPresetTest extends CXFTestBase {
.getStatus(), inner);
}
+ /**
+ * @return the embedded documents of the Frictionless package, without its
+ * datapackage.json and metadata.json
+ */
private Map<String, byte[]> unpack(String path, String file) throws
IOException {
Response response = put(path, fixture(file));
assertEquals(200, response.getStatus(), file);
- return readZipArchiveBytes((InputStream) response.getEntity());
+ Map<String, byte[]> unpacked = new LinkedHashMap<>();
+ readZipArchiveBytes((InputStream) response.getEntity()).forEach((name,
bytes) -> {
+ if (name.startsWith("unpacked/")) {
+ unpacked.put(name, bytes);
+ }
+ });
+ return unpacked;
}
private Response put(String path, InputStream body) {
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaPipesTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaPipesTest.java
index a3cf4bb072..f544e50c54 100644
---
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaPipesTest.java
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/TikaPipesTest.java
@@ -18,6 +18,7 @@ package org.apache.tika.server.standard;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
import java.io.ByteArrayInputStream;
import java.io.IOException;
@@ -32,9 +33,14 @@ import java.nio.file.Path;
import java.nio.file.StandardCopyOption;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Enumeration;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import java.util.Set;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipFile;
import jakarta.ws.rs.core.Response;
import org.apache.commons.io.FileUtils;
@@ -320,9 +326,26 @@ public class TikaPipesTest extends CXFTestBase {
assertContains("When in the Course", metadataList
.get(6)
.get(TikaCoreProperties.TIKA_CONTENT));
- Map<String, Long> expected = loadExpected();
+ // TIKA-4681: the bytes emitter writes one Frictionless package, the
embedded
+ // documents under unpacked/ in it
Map<String, Long> byteFileNames = getFileNames(outputBytesDir);
- assertEquals(expected, byteFileNames);
+ assertEquals(Set.of(TEST_RECURSIVE_DOC + "-frictionless.zip"),
byteFileNames.keySet());
+ List<Long> unpackedSizes = new ArrayList<>();
+ try (ZipFile zip = new ZipFile(
+ outputBytesDir.resolve(TEST_RECURSIVE_DOC +
"-frictionless.zip").toFile())) {
+ assertNotNull(zip.getEntry("datapackage.json"));
+ assertNotNull(zip.getEntry("metadata.json"));
+ for (Enumeration<? extends ZipEntry> e = zip.entries();
e.hasMoreElements(); ) {
+ ZipEntry entry = e.nextElement();
+ if (entry.getName().startsWith("unpacked/")) {
+ unpackedSizes.add(entry.getSize());
+ }
+ }
+ }
+ List<Long> expectedSizes = new ArrayList<>(loadExpected().values());
+ Collections.sort(expectedSizes);
+ Collections.sort(unpackedSizes);
+ assertEquals(expectedSizes, unpackedSizes);
}
private Map<String, Long> loadExpected() {
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackFrictionlessTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackFrictionlessTest.java
index e05b658021..7c671b29e2 100644
---
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackFrictionlessTest.java
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackFrictionlessTest.java
@@ -50,7 +50,7 @@ import org.apache.tika.server.core.resource.UnpackerResource;
import org.apache.tika.server.core.writer.TarWriter;
import org.apache.tika.server.core.writer.ZipWriter;
-/** /unpack under a server-level Frictionless config, with no per-request
config. */
+/** /unpack with no unpack-config anywhere: a Frictionless package since 4.2
(TIKA-4681). */
public class UnpackFrictionlessTest extends CXFTestBase {
private static final String TEST_DOC =
"test-documents/test_recursive_embedded.docx";
@@ -58,10 +58,7 @@ public class UnpackFrictionlessTest extends CXFTestBase {
private static final String SERVER_CONFIG = """
{
- "parsers": [ { "default-parser": {} } ],
- "parse-context": {
- "unpack-config": { "outputFormat": "FRICTIONLESS",
"outputMode": "ZIPPED" }
- }
+ "parsers": [ { "default-parser": {} } ]
}
""";
@@ -113,11 +110,14 @@ public class UnpackFrictionlessTest extends CXFTestBase {
}
@Test
- public void testServerConfigMakesUnpackFrictionless() throws Exception {
+ public void testDefaultIsAFrictionlessPackage() throws Exception {
Map<String, byte[]> entries = unpack("/unpack");
assertTrue(entries.containsKey("datapackage.json"),
entries.keySet().toString());
assertTrue(entries.keySet().stream().anyMatch(k ->
k.startsWith("unpacked/")),
entries.keySet().toString());
+ assertTrue(entries.keySet().stream().allMatch(k ->
k.startsWith("unpacked/")
+ || k.equals("datapackage.json") ||
k.equals("metadata.json")),
+ "nothing but the package at the root: " + entries.keySet());
}
/** A Frictionless package carries metadata.json by default; /all adds
only the original. */
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackFrictionlessTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackRegularConfigTest.java
similarity index 53%
copy from
tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackFrictionlessTest.java
copy to
tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackRegularConfigTest.java
index e05b658021..07bf63c0aa 100644
---
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackFrictionlessTest.java
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackRegularConfigTest.java
@@ -24,10 +24,10 @@ import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -37,9 +37,6 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import jakarta.ws.rs.core.Response;
import org.apache.cxf.jaxrs.JAXRSServerFactoryBean;
import org.apache.cxf.jaxrs.client.WebClient;
-import org.apache.cxf.jaxrs.ext.multipart.Attachment;
-import org.apache.cxf.jaxrs.ext.multipart.ContentDisposition;
-import org.apache.cxf.jaxrs.ext.multipart.MultipartBody;
import org.apache.cxf.jaxrs.lifecycle.SingletonResourceProvider;
import org.junit.jupiter.api.Test;
@@ -50,8 +47,8 @@ import org.apache.tika.server.core.resource.UnpackerResource;
import org.apache.tika.server.core.writer.TarWriter;
import org.apache.tika.server.core.writer.ZipWriter;
-/** /unpack under a server-level Frictionless config, with no per-request
config. */
-public class UnpackFrictionlessTest extends CXFTestBase {
+/** A server-level unpack-config.outputFormat of REGULAR restores the pre-4.2
flat zip. */
+public class UnpackRegularConfigTest extends CXFTestBase {
private static final String TEST_DOC =
"test-documents/test_recursive_embedded.docx";
private static final ObjectMapper MAPPER = new ObjectMapper();
@@ -60,7 +57,7 @@ public class UnpackFrictionlessTest extends CXFTestBase {
{
"parsers": [ { "default-parser": {} } ],
"parse-context": {
- "unpack-config": { "outputFormat": "FRICTIONLESS",
"outputMode": "ZIPPED" }
+ "unpack-config": { "outputFormat": "REGULAR" }
}
}
""";
@@ -95,7 +92,7 @@ public class UnpackFrictionlessTest extends CXFTestBase {
@Override
protected InputStream getPipesConfigInputStream() throws IOException {
- unpackTempDir = sharedUnpackDir();
+ unpackTempDir = Files.createTempDirectory("tika-unpack-regular-");
Map<String, Object> replacements = new HashMap<>();
replacements.put("UNPACK_EMITTER_BASE_PATH",
unpackTempDir.toAbsolutePath().toString());
replacements.put("PLUGINS_PATHS",
@@ -113,74 +110,23 @@ public class UnpackFrictionlessTest extends CXFTestBase {
}
@Test
- public void testServerConfigMakesUnpackFrictionless() throws Exception {
+ public void testServerConfigRestoresTheFlatLayout() throws Exception {
Map<String, byte[]> entries = unpack("/unpack");
- assertTrue(entries.containsKey("datapackage.json"),
entries.keySet().toString());
- assertTrue(entries.keySet().stream().anyMatch(k ->
k.startsWith("unpacked/")),
- entries.keySet().toString());
- }
-
- /** A Frictionless package carries metadata.json by default; /all adds
only the original. */
- @Test
- public void testPlainUnpackHasMetadataJsonByDefault() throws Exception {
- Map<String, byte[]> entries = unpack("/unpack");
- assertTrue(entries.containsKey("metadata.json"),
entries.keySet().toString());
- assertFalse(entries.keySet().stream().anyMatch(k ->
k.startsWith("unpacked/0.")),
- "plain /unpack must not pack the container's bytes: " +
entries.keySet());
- }
-
- /** DIRECTORY from a request contradicts the one-body transport: 400, not
a silent pin. */
- @Test
- public void testDirectoryModeInRequestIs400() throws Exception {
- ContentDisposition cd = new ContentDisposition(
- "form-data; name=\"file\";
filename=\"test_recursive_embedded.docx\"");
- Attachment fileAtt = new Attachment("file",
ClassLoader.getSystemResourceAsStream(TEST_DOC), cd);
- Attachment configAtt = new Attachment("config", "application/json",
new ByteArrayInputStream(
- "{\"unpack-config\": {\"outputMode\":
\"DIRECTORY\"}}".getBytes(StandardCharsets.UTF_8)));
- Response response = WebClient.create(endPoint + "/unpack")
- .type("multipart/form-data")
- .accept("application/zip")
- .post(new MultipartBody(Arrays.asList(fileAtt, configAtt)));
- assertEquals(400, response.getStatus());
+ assertFalse(entries.containsKey("datapackage.json"),
entries.keySet().toString());
+ assertFalse(entries.containsKey("metadata.json"),
entries.keySet().toString());
+ assertTrue(entries.keySet().stream().noneMatch(k -> k.contains("/")),
+ "REGULAR puts every file at the zip root: " +
entries.keySet());
+ assertFalse(entries.isEmpty());
}
+ /** Under REGULAR, /all is where the per-file sidecars and the original
come from. */
@Test
- public void testUnpackAllCarriesMetadataAndTheOriginal() throws Exception {
+ public void testUnpackAllAddsSidecarsAndTheOriginal() throws Exception {
Map<String, byte[]> entries = unpack("/unpack/all");
- assertTrue(entries.containsKey("metadata.json"),
entries.keySet().toString());
- assertTrue(entries.keySet().stream().anyMatch(k ->
k.startsWith("unpacked/0.")),
- "includeOriginal should add the container: " +
entries.keySet());
- }
-
- /** The RMETA list's row 0 is the container -- its own metadata and its
own text. */
- @Test
- public void testMetadataJsonCarriesTheContainerRow() throws Exception {
- Map<String, byte[]> entries = unpack("/unpack");
- JsonNode rows = MAPPER.readTree(entries.get("metadata.json"));
- assertTrue(rows.isArray() && rows.size() > 1, "expected an RMETA list,
got: " + rows);
-
- JsonNode container = rows.get(0);
- assertEquals("0", container.path("tk:embedded-depth").asText(),
- "row 0 should be the container: " + container);
- assertTrue(container.hasNonNull("tk:content"),
- "the container row should carry its own extracted text: " +
container);
- assertTrue(container.hasNonNull("dcterms:created"),
- "the container row should carry its own metadata: " +
container);
- }
-
- /** Every row can be joined to its file: id, tree position and logical
path all travel. */
- @Test
- public void testRowsCarryEmbeddedPathData() throws Exception {
- Map<String, byte[]> entries = unpack("/unpack/all");
- JsonNode rows = MAPPER.readTree(entries.get("metadata.json"));
- JsonNode embedded = rows.get(1);
- for (String key : new String[]{"tk:embedded-id", "tk:embedded-id-path",
- "tk:embedded-resource-path", "tk:resource-name"}) {
- assertTrue(embedded.hasNonNull(key), key + " missing from: " +
embedded);
- }
- String id = embedded.get("tk:embedded-id").asText();
- assertTrue(entries.keySet().stream().anyMatch(k ->
k.startsWith("unpacked/" + id + ".")),
- "row's tk:embedded-id should name a packed file: " +
entries.keySet());
+ assertTrue(entries.keySet().stream().anyMatch(k ->
k.endsWith(".metadata.json")),
+ entries.keySet().toString());
+ assertTrue(entries.keySet().stream().anyMatch(k -> k.startsWith("0.")),
+ "includeOriginal should add the container at the root: " +
entries.keySet());
}
private Map<String, byte[]> unpack(String path) throws Exception {
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackerResourceHandlerTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackerResourceHandlerTest.java
index 699edf794b..c1169f14ee 100644
---
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackerResourceHandlerTest.java
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackerResourceHandlerTest.java
@@ -54,11 +54,21 @@ import org.apache.tika.server.core.writer.ZipWriter;
/**
* /unpack has no format segment, so the config part is the only place a
handler is named --
* and it reaches the RMETA list written to metadata.json.
+ * Per-file sidecars exist only in the REGULAR layout, so the server config
pins it; the
+ * metadata.json probes ask for FRICTIONLESS explicitly.
*/
public class UnpackerResourceHandlerTest extends CXFTestBase {
private static final String TEST_DOC =
"test-documents/test_recursive_embedded.docx";
private static final ObjectMapper MAPPER = new ObjectMapper();
+ private static final String SERVER_CONFIG = """
+ {
+ "parsers": [ { "default-parser": {} } ],
+ "parse-context": {
+ "unpack-config": { "outputFormat": "REGULAR" }
+ }
+ }
+ """;
private static final String FRICTIONLESS =
"\"unpack-config\": {\"outputFormat\": \"FRICTIONLESS\",
\"outputMode\": \"ZIPPED\", "
+ "\"includeFullMetadata\": true}";
@@ -88,7 +98,7 @@ public class UnpackerResourceHandlerTest extends CXFTestBase {
@Override
protected InputStream getTikaConfigInputStream() throws IOException {
- return
this.getClass().getResourceAsStream("/configs/tika-config-unpacker.json");
+ return new
ByteArrayInputStream(SERVER_CONFIG.getBytes(StandardCharsets.UTF_8));
}
@Override
@@ -171,10 +181,10 @@ public class UnpackerResourceHandlerTest extends
CXFTestBase {
/** includeMetadataInZip writes a sidecar per extracted file; it carries
tk:content too. */
@Test
public void testPerFileSidecarsFollowTheConfigPartHandler() throws
Exception {
- assertTrue(perFileSidecar("{\"unpack-config\":
{\"includeMetadataInZip\": true}}")
+ assertTrue(perFileSidecar("{\"unpack-config\": {\"outputFormat\":
\"REGULAR\", \"includeMetadataInZip\": true}}")
.contains("\"tk:content-handler-type\":\"MARKDOWN\""));
- String xml = perFileSidecar("{\"unpack-config\":
{\"includeMetadataInZip\": true}, "
+ String xml = perFileSidecar("{\"unpack-config\": {\"outputFormat\":
\"REGULAR\", \"includeMetadataInZip\": true}, "
+ "\"basic-content-handler-factory\": {\"type\": \"XML\"}}");
assertTrue(xml.contains("\"tk:content-handler-type\":\"XML\""), xml);
assertTrue(xml.contains("<html xmlns="), xml);
@@ -183,8 +193,8 @@ public class UnpackerResourceHandlerTest extends
CXFTestBase {
/** IGNORE is the opt-out: sidecars keep their metadata but carry no
extracted text. */
@Test
public void testIgnoreHandlerLeavesSidecarsWithoutContent() throws
Exception {
- String withText = "{\"unpack-config\": {\"includeMetadataInZip\":
true}}";
- String ignore = "{\"unpack-config\": {\"includeMetadataInZip\": true},
"
+ String withText = "{\"unpack-config\": {\"outputFormat\": \"REGULAR\",
\"includeMetadataInZip\": true}}";
+ String ignore = "{\"unpack-config\": {\"outputFormat\": \"REGULAR\",
\"includeMetadataInZip\": true}, "
+ "\"basic-content-handler-factory\": {\"type\": \"IGNORE\"}}";
assertTrue(sidecarsWithContent(withText) > 0, "baseline should carry
tk:content");
diff --git
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackerResourceTest.java
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackerResourceTest.java
index 00006e6e2d..26ffa7680f 100644
---
a/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackerResourceTest.java
+++
b/tika-server/tika-server-standard/src/test/java/org/apache/tika/server/standard/UnpackerResourceTest.java
@@ -172,11 +172,8 @@ public class UnpackerResourceTest extends CXFTestBase {
.toList();
assertEquals(2, wavFiles.size(), "Should have 2 WAV files");
- // With saveAll=true, metadata JSON files should be included
- List<String> metadataFiles = data.keySet().stream()
- .filter(k -> k.endsWith(".metadata.json"))
- .toList();
- assertTrue(metadataFiles.size() >= 2, "Should have metadata JSON files
for each embedded doc");
+ assertTrue(data.containsKey("metadata.json"),
+ "/all carries the parse as metadata.json. Entries: " +
data.keySet());
}
@Test
@@ -271,7 +268,6 @@ public class UnpackerResourceTest extends CXFTestBase {
@Test
public void testMetadataJsonIncluded() throws Exception {
- // Test that /unpack/all includes metadata JSON files
Response response = WebClient
.create(endPoint + ALL_PATH)
.type(APPLICATION_MSWORD)
@@ -280,14 +276,9 @@ public class UnpackerResourceTest extends CXFTestBase {
Map<String, byte[]> data = readZipArchiveBytes((InputStream)
response.getEntity());
- // Should have metadata JSON files
- List<String> metadataFiles = data.keySet().stream()
- .filter(k -> k.endsWith(".metadata.json"))
- .toList();
- assertFalse(metadataFiles.isEmpty(), "Should have metadata JSON
files");
-
- // Verify the JSON contains expected metadata fields
- String metadataJson = new String(data.get(metadataFiles.get(0)),
StandardCharsets.UTF_8);
+ byte[] metadata = data.get("metadata.json");
+ assertNotNull(metadata, "Should have metadata.json. Entries: " +
data.keySet());
+ String metadataJson = new String(metadata, StandardCharsets.UTF_8);
assertTrue(metadataJson.contains("Content-Type"), "Metadata JSON
should contain Content-Type");
}
@@ -424,8 +415,9 @@ public class UnpackerResourceTest extends CXFTestBase {
assertEquals(200, response.getStatus());
Map<String, String> data = readZipArchive((InputStream)
response.getEntity());
- // With maxCount=1, should only have 1 embedded document
- assertEquals(1, data.size(), "Should have exactly 1 embedded document
with maxCount=1");
+ long unpacked = data.keySet().stream().filter(k ->
k.startsWith("unpacked/")).count();
+ assertEquals(1, unpacked, "Should have exactly 1 embedded document
with maxCount=1. Entries: "
+ + data.keySet());
}
/**
@@ -595,15 +587,20 @@ public class UnpackerResourceTest extends CXFTestBase {
*/
@Test
public void testRegularAllContainerAppearsOnce() throws Exception {
+ String configJson = """
+ { "parse-context": { "unpack-config": { "outputFormat":
"REGULAR" } } }
+ """;
ContentDisposition fileCd = new ContentDisposition("form-data;
name=\"file\"; filename=\"Doc1_ole.doc\"");
Attachment fileAtt = new Attachment("file",
ClassLoader.getSystemResourceAsStream(TEST_DOC_WAV), fileCd);
+ Attachment configAtt = new Attachment("config", "application/json",
+ new
ByteArrayInputStream(configJson.getBytes(StandardCharsets.UTF_8)));
Response response = WebClient
.create(endPoint + ALL_PATH)
.type("multipart/form-data")
.accept("application/zip")
- .post(new MultipartBody(Arrays.asList(fileAtt)));
+ .post(new MultipartBody(Arrays.asList(fileAtt, configAtt)));
assertEquals(200, response.getStatus());
Map<String, String> data = readZipArchive((InputStream)
response.getEntity());
