This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new e2940fd226 TIKA-4845 - improve BAT estimation (#3080)
e2940fd226 is described below
commit e2940fd22606b0a1cf109f9d0fdac2320f430784
Author: Tim Allison <[email protected]>
AuthorDate: Thu Aug 27 13:59:45 2026 -0400
TIKA-4845 - improve BAT estimation (#3080)
---
CHANGES.txt | 158 +++++++++++++++++++++
.../detect/microsoft/POIFSContainerDetector.java | 36 ++++-
.../detect/microsoft/POIFSDeclaredSizeTest.java | 131 +++++++++++++++--
3 files changed, 308 insertions(+), 17 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 4953061798..1ca08ba230 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,163 @@
Release 4.1.0 - unreleased
+ * Stop spooling OLE2 objects whose header over-reserves BAT capacity
+ (TIKA-4845).
+
+ * Add Micrometer reporting and opt-in endpoint for tika-server (TIKA-4839).
+
+ * Improve spooling/decrease number of spills to disk (TIKA-4835).
+
+ * Fixed a bug that made per-request (parse-context) configuration unusable
+ for parsers that lock some config fields against caller modification --
+ Tess4J, the VLM parsers and the OpenAI image-embedding parser. Any such
+ config threw, including an empty one: the defaults were deep-copied
+ through their own setters, which the runtime config overrides to reject
+ caller input, so the copy tripped the parser's own guards before the
+ caller's JSON was read. Locked fields are still rejected when a caller
+ actually sets them. Configuration supplied at initialization time (the
+ "parsers" section) was never affected (TIKA-4843).
+
+ * OOXML parsers flag package parts that are unreachable through the OPC
+ relationship graph: msoffice:has-unreferenced-parts (boolean) and
+ msoffice:unreferenced-part-names. Purely structural (no bytes are
+ inspected; content types come from [Content_Types].xml by extension), so
+ expect false positives from tools that leave orphan parts behind. A hiding
+ place a raw-ZIP scanner can still see, not a statement about what Tika
+ parsed. Applies to Word, Excel, PowerPoint and Visio OOXML (including
+ macro-enabled variants); XPS links content by markup rather than
+ relationships and is not checked (TIKA-4837).
+
+ * Shared pipes server (useSharedServer: true, not the default): a client
whose
+ in-flight parse was killed by another client's restart could restart the
+ healthy replacement. ensureRunning holds its lock across the whole fork,
so
+ siblings cannot report a dead worker until after the replacement is up,
and
+ the pending-restart flag carried no process identity -- so a report about
+ the process that just died was applied to its successor, which was then
+ destroyed and re-forked. One worker death produced two restarts and a
second
+ round of destroyed in-flight work; under sustained concurrent load it
+ sustained itself at one spurious restart per round, appearing as periodic
+ unexplained worker churn and intermittent parse failures that succeed on
+ retry. Each fork now carries a generation that clients capture when they
+ connect and hand back with every report, and reports about a superseded
+ process are dropped. Also fixed in shared mode: ensureRunning could fork a
+ replacement after shutdown() that nothing owned and nothing would ever
+ destroy, and an interrupt during process teardown left the process handle
+ pointing at a killed process and leaked the temp directory. Affects 4.0.0
+ and earlier (TIKA-4844).
+
+ * tika-pipes: the cache memory budget (how much rewindable content a forked
+ worker keeps in memory before spilling to disk; new since 4.0.0, which had
+ no budget at all) defaults to a quarter of the fork's heap, so raising
+ -Xmx raises it. It is one pool per forked JVM shared by all of its
threads.
+ -Dtika.pipes.cacheMemoryBudgetBytes in forkedJvmArgs overrides it (below
+ the quarter-heap ceiling; <=0 disables); the fork logs the value and its
+ source at startup. TikaInputStream.hasFile() now also reports content the
+ stream cache spilled on its own, not only content a getPath() call put on
+ disk; note getPath() may still have to drain the rest of the source into
+ that file. TikaInputStream.toString() no longer forces a spill, so logging
+ or debugger-inspecting a stream is side-effect-free.
+ TikaInputStream.inMemoryContent(channel) gives a zero-copy read-only view
+ of cached content for consumers that need random access. Digester
+ gains digestSink(), a DigestSink that digests as it is written; nothing is
+ written to the metadata unless the producer calls commit(), so any failure
+ -- exception, Error, or a producer that closes the sink itself --
publishes
+ no digest rather than a digest of the bytes that happened to arrive. A
+ translator that claims a stream and writes nothing likewise publishes
+ nothing: embedded PST mail items, whose translator is still a stub, no
+ longer carry the digest of zero bytes (the same value for every one of
+ them) and instead carry no digest at all. DigestHelper uses it for
+ translated embedded streams, which no longer touch a temp file when the
+ digester implements digestSink (all of Tika's do; one that only implements
+ digest() still buffers).
+ TemporaryResources.closeAll(Closeable...) closes every argument even when
+ one throws unchecked; TemporaryResources, CachingSource,
CachingInputStream
+ and CompositeDigester use it (TIKA-4835).
+
+ * Documentation: corrected a batch of pages and javadocs that contradicted
+ the code. Notably: the ES/OpenSearch attachmentStrategy has no default
+ (unset means embedded documents get neither the parent field nor the
+ parent/child relation); Kafka's connectionsMaxIdleMs is passed to the
+ producer, not ignored; jdbc queryTimeoutSeconds is applied only when > 0,
+ so 0 does not mean "no limit"; the Solr emitter/iterator support only
+ basic auth, not ntlm, and only when a userName is set; pipes-reporters
+ silently loads zero reporters when given a JSON array, and
+ pipes-iterator/pipes-reporters instances are built at config load rather
+ than lazily; under CONTENT_ONLY only a parse-context filter replaces the
+ built-in one, not the top-level metadata-filters chain;
+ _mime-include/_mime-exclude also accept a bare string; Tess4J locks
+ poolSize and maxImagePixels as well as the two paths; and pdf:trapped and
+ xmp:pdf:Trapped are new 4.x keys rather than renames (3.x captured the
+ flag only as pdf:docinfo:trapped and dropped the XMP value). Also
+ corrected the config nesting shown in every pipes-plugin fetcher/emitter
+ javadoc -- 23 of them, which had it inverted (the instance id is the
+ outer key, the component name the inner) -- and removed references to a
+ TesseractOCRConfig.properties file that 4.x does not load (TIKA-4842).
+
+ * Pipes plugins no longer bundle their own Jackson: jackson-core, -databind
+ and -annotations are provided by the host (tika-serialization) and the
+ plugins parent pom now bans bundling them, so a mapper can cross the
+ plugin boundary without a second copy of the Jackson classes (seven plugin
+ zips shipped one). Plugin configuration JSON is parsed by one shared
+ mapper, PluginJson (tika-plugins-core), which rejects unknown keys,
+ numbers for enums and duplicate keys, and accepts
+ // and /* */ comments; the 33 per-plugin *Config classes use it instead of
+ their own bare ObjectMapper (TIKA-4840).
+
+ * tika-server and tika-async-cli now start from a config that contains
+ // or /* */ comments, as the configuration docs have always said they
+ may. The main loader accepted them; the steps that re-read the user's
+ file to merge in server/CLI overrides (ConfigMerger, ensurePluginRoots)
+ used their own bare parser and refused the whole file; they now use the
+ shared TikaObjectMapperFactory mapper (TIKA-4834).
+
+ * The Kafka pipes iterator no longer stops at the first empty poll. A newly
+ subscribed consumer spends its first poll(s) joining the group and returns
+ empty even when the topic has a backlog, so the iterator could enqueue
zero
+ files and report success. It now waits for a partition assignment (bounded
+ by the new assignmentTimeoutMs, default 30s) and requires a continuous
quiet
+ window (drainIdleMs, default 1s) before concluding the topic is drained.
+ groupInitialRebalanceDelayMs is deprecated and no longer sent to the
+ consumer: it is a broker setting that Kafka has always ignored
(TIKA-4833).
+
+ * Pipes IPC: carry inline document bytes as a raw binary field beside the
+ tuple in the request envelope -- never inside the tuple or its
+ ParseContext -- and disable Smile's 7-bit binary encoding. Tuple JSON
+ serialized by 4.0.0 with an "inline-bytes" parse-context entry no longer
+ loads; it is rejected with a tailored message (TIKA-4829).
+
+ * Digesting embedded documents no longer buffers each embedded object to a
+ temp file. Zip entries are re-read from the parent archive on rewind, and
+ a new process-wide CacheMemoryBudget (seeded by the pipes forked server;
+ default 256MB, clamped to a quarter of the fork's heap; tunable via
+ -Dtika.pipes.cacheMemoryBudgetBytes in the config's forkedJvmArgs, <=0
+ disables) lets embedded objects stay in memory past the per-object 1MB
+ threshold. New public API on TikaInputStream: get(IOSupplier,...),
+ enableRewind(CacheMemoryBudget), getSeekableByteChannel(). Zip/7z/epub/odf
+ parsing and zip container detection now read through seekable channels, so
+ after detection/parsing a TikaInputStream may no longer be file-backed
+ (hasFile() false); getPath()/getFile() still work and spool on demand
+ (TIKA-4828).
+
+ * Pipes now carries the caller-supplied Content-Type across the worker's
+ fresh-metadata boundary as a soft detection hint, so every forked-parse
+ endpoint (/tika, /meta, /rmeta, /unpack, /async, /pipes, plus tika-grpc
+ and embedded PipesForkParser) can route on a client Content-Type, not
+ only on the filename. Detection keeps the hint only when it equals or
+ specializes the content-detected type (e.g. refining image/tiff to
+ image/x-canon-cr2); for bytes with no magic it can select any type,
+ matching the routing power the filename already had. The
+ CONTENT_TYPE_USER_OVERRIDE key is deliberately not carried, so the hint
+ cannot force an unrelated type (TIKA-4825).
+
+ * OneNote extraction now follows document order, omits superseded page
+ revisions, sorts author metadata, extracts embedded object BLOBs, and
+ bounds malformed-input recursion and file-derived allocations. Parse
+ warnings and embedded relationship IDs are exposed in metadata. Malformed
+ or truncated files that cannot be fully parsed, and files whose walk
+ yields no content, now fall back to the legacy string dump instead of
+ failing or returning empty output. The legacy MS-ONESTORE walker bounds
+ its recursion (depth caps plus file-node-list and fragment-chain cycle
+ guards) and now honors shouldParseEmbedded for embedded file data
* PDF: extractFontNames threw NullPointerException on a page with no
/Resources dictionary (TIKA-4842).
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/detect/microsoft/POIFSContainerDetector.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/detect/microsoft/POIFSContainerDetector.java
index cb046e3780..7438e94f44 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/detect/microsoft/POIFSContainerDetector.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/detect/microsoft/POIFSContainerDetector.java
@@ -605,6 +605,11 @@ public class POIFSContainerDetector implements Detector {
* in memory, the budget has no room for the copy, or POI's stream loader
rejects the
* object (it is stricter than the file loader on truncated objects).
* <p>
+ * None of this would be needed if POI could be handed the bytes it
already has. Its API
+ * takes a File or FileChannel (no copy) or an InputStream (copy sized by
the header) with
+ * nothing in between, and POIFSFileSystem._data is protected while
readCoreContents() is
+ * private, so a subclass cannot supply a ByteArrayBackedDataSource either.
+ * <p>
* With no budget in the context there is no accounting at all, and a
byte[]-backed
* stream has no spill threshold to bound it either, so the copy is capped
outright:
* above {@link #MAX_UNBUDGETED_COPY} the file path -- which is what 4.0.0
always did --
@@ -617,6 +622,16 @@ public class POIFSContainerDetector implements Detector {
*/
private static final long MAX_UNBUDGETED_COPY = 16L * 1024 * 1024;
+ /**
+ * BAT blocks a small object may reserve regardless of its length. Every
valid header
+ * declares at least one, and writers round up; the unit is the sector
size, so this is
+ * 256KB for 512-byte sectors and 16MB for 4K ones.
+ */
+ private static final int MIN_DECLARED_BAT_BLOCKS = 4;
+
+ /** Past this multiple of the bytes in hand the header is not describing
this object. */
+ private static final int MAX_DECLARED_AMPLIFICATION = 16;
+
private Set<String> getTopLevelNamesInMemory(TikaInputStream stream,
ParseContext context)
throws IOException {
CacheMemoryBudget budget = context == null ? null :
context.get(CacheMemoryBudget.class);
@@ -674,10 +689,15 @@ public class POIFSContainerDetector implements Detector {
/**
* The heap POI's stream loader would allocate for this object -- sized
from the header's
* declared BAT count, not the content -- or -1 when the header cannot be
read or declares
- * more than the content can account for. A valid header covers at most
one BAT block of
- * unused entries beyond the actual size; anything past that is a
malformed or hostile
- * header (a 512-byte object can declare hundreds of MB) and must not be
opened from a
- * stream at all.
+ * an implausible multiple of the bytes in hand.
+ * <p>
+ * The bound is on amplification, over a floor of a few BAT blocks. Real
writers reserve
+ * BAT capacity ahead of use -- SolidWorks declares 26 BAT blocks where 16
would do, and
+ * small objects routinely declare several times their own length -- so
"one BAT block of
+ * slack" rejects valid files. The floor is in sectors rather than bytes
because a 4K-sector
+ * object cannot declare less than 4MB even when it is nearly empty. What
must not get
+ * through is a 512-byte object declaring hundreds of MB, four orders of
magnitude past
+ * anything a writer produces.
*/
static long honestDeclaredSize(SeekableByteChannel channel) throws
IOException {
byte[] header = new byte[POIFSConstants.SMALLER_BIG_BLOCK_SIZE];
@@ -698,10 +718,12 @@ public class POIFSContainerDetector implements Detector {
HeaderBlock hb = new HeaderBlock(
UnsynchronizedByteArrayInputStream.builder().setByteArray(header).get());
long declared = BATBlock.calculateMaximumSize(hb);
- long oneBatSpan = (long) hb.getBigBlockSize().getBigBlockSize() *
- hb.getBigBlockSize().getBATEntriesPerBlock();
+ long sector = hb.getBigBlockSize().getBigBlockSize();
+ long batSpan = sector *
hb.getBigBlockSize().getBATEntriesPerBlock();
long actual = channel.size();
- return declared > actual + oneBatSpan ? -1 : declared;
+ long ceiling = Math.max(sector + MIN_DECLARED_BAT_BLOCKS * batSpan,
+ actual * MAX_DECLARED_AMPLIFICATION);
+ return declared > ceiling ? -1 : declared;
} catch (IOException | RuntimeException e) {
return -1;
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/detect/microsoft/POIFSDeclaredSizeTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/detect/microsoft/POIFSDeclaredSizeTest.java
index 8450153bec..a002d71a2b 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/detect/microsoft/POIFSDeclaredSizeTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/detect/microsoft/POIFSDeclaredSizeTest.java
@@ -17,6 +17,7 @@
package org.apache.tika.detect.microsoft;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -32,6 +33,8 @@ import java.nio.file.StandardOpenOption;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
import org.apache.tika.TikaTest;
import org.apache.tika.io.CacheMemoryBudget;
@@ -43,8 +46,8 @@ import org.apache.tika.parser.ParseContext;
/**
* POI sizes its in-memory OLE2 buffer from the header's declared BAT count
rather than the
* actual length, so a 512-byte object can demand hundreds of MB. The
in-memory detection path
- * only believes a header the bytes in hand can account for, and reserves that
from the
- * budget before POI allocates it.
+ * refuses a header that declares an implausible multiple of the bytes in
hand, and reserves
+ * what it does believe from the budget before POI allocates it.
*/
public class POIFSDeclaredSizeTest extends TikaTest {
@@ -56,15 +59,27 @@ public class POIFSDeclaredSizeTest extends TikaTest {
/** A bare 512-byte OLE2 header declaring {@code batCount} BAT blocks and
nothing else. */
private static byte[] header(int batCount) {
+ return header(batCount, 9);
+ }
+
+ /** As above, with an explicit sector shift: 9 for 512-byte sectors, 12
for 4K. */
+ private static byte[] header(int batCount, int sectorShift) {
byte[] data = new byte[512];
byte[] magic = {(byte) 0xd0, (byte) 0xcf, 0x11, (byte) 0xe0,
(byte) 0xa1, (byte) 0xb1, 0x1a, (byte) 0xe1};
System.arraycopy(magic, 0, data, 0, magic.length);
- data[SECTOR_SHIFT_OFFSET] = 9; // 2^9 = 512-byte blocks
+ data[SECTOR_SHIFT_OFFSET] = (byte) sectorShift;
ByteBuffer.wrap(data).order(ByteOrder.LITTLE_ENDIAN).putInt(BAT_COUNT_OFFSET,
batCount);
return data;
}
+ /** The same header followed by zeros out to {@code totalLen}. */
+ private static byte[] headerPaddedTo(int batCount, int totalLen) {
+ byte[] data = new byte[totalLen];
+ System.arraycopy(header(batCount), 0, data, 0, 512);
+ return data;
+ }
+
private SeekableByteChannel channelFor(byte[] bytes, String name) throws
Exception {
Path p = tempDir.resolve(name);
Files.write(p, bytes);
@@ -82,18 +97,87 @@ public class POIFSDeclaredSizeTest extends TikaTest {
}
}
+ /**
+ * Small objects may reserve up to four BAT blocks regardless of length:
at that size the
+ * ratio to a few hundred bytes of header says nothing about whether the
header is honest.
+ */
+ @Test
+ public void testSmallDeclarationsAreBelievedUpToTheBatFloor() throws
Exception {
+ // 4 BAT blocks == 262_656 bytes declared by 512 bytes of content:
exactly the floor
+ try (SeekableByteChannel channel = channelFor(header(4),
"at-floor.ole")) {
+ assertEquals((1 + 4 * 128) * 512L,
+ POIFSContainerDetector.honestDeclaredSize(channel));
+ }
+ // 5 BAT blocks == 328_192: the first value past it
+ try (SeekableByteChannel channel = channelFor(header(5),
"over-floor.ole")) {
+ assertEquals(-1,
POIFSContainerDetector.honestDeclaredSize(channel));
+ }
+ }
+
+ /**
+ * The floor is counted in sectors, not bytes. A 4K-sector object cannot
declare less than
+ * one 4MB BAT block however empty it is, so a byte-valued floor tuned for
512-byte sectors
+ * would reject every small one of them -- a case no file in the test
corpus exercises.
+ */
@Test
- public void testHonestHeaderIsBelievedWithinOneBatBlock() throws Exception
{
- // header only, declaring one BAT block: 129 sectors, 512 bytes
present -- within slack
- try (SeekableByteChannel channel = channelFor(header(1),
"modest.ole")) {
- assertEquals((1 + 128) * 512L,
POIFSContainerDetector.honestDeclaredSize(channel));
+ public void testTheBatFloorScalesWithSectorSize() throws Exception {
+ // 4 BAT blocks of 4K sectors == 16_781_312 bytes: the floor, not a
byte constant
+ try (SeekableByteChannel channel = channelFor(header(4, 12),
"4k-at-floor.ole")) {
+ assertEquals((1 + 4 * 1024) * 4096L,
+ POIFSContainerDetector.honestDeclaredSize(channel));
}
- // two BAT blocks declared by 512 bytes: one block past what the
content covers
- try (SeekableByteChannel channel = channelFor(header(2),
"twoblocks.ole")) {
+ try (SeekableByteChannel channel = channelFor(header(5, 12),
"4k-over-floor.ole")) {
assertEquals(-1,
POIFSContainerDetector.honestDeclaredSize(channel));
}
}
+ /** Past the floor the bound is a multiple of the bytes actually in hand.
*/
+ @Test
+ public void testLargeDeclarationsAreBoundedByAmplification() throws
Exception {
+ int actual = 128 * 1024; // ceiling is 16x this == 2_097_152
+ // 31 BAT blocks == 2_032_128 declared: inside the ceiling
+ try (SeekableByteChannel channel =
+ channelFor(headerPaddedTo(31, actual),
"under-ceiling.ole")) {
+ assertEquals((1 + 31 * 128) * 512L,
+ POIFSContainerDetector.honestDeclaredSize(channel));
+ }
+ // 32 BAT blocks == 2_097_664 declared: the first value past it
+ try (SeekableByteChannel channel =
+ channelFor(headerPaddedTo(32, actual),
"over-ceiling.ole")) {
+ assertEquals(-1,
POIFSContainerDetector.honestDeclaredSize(channel));
+ }
+ }
+
+ /**
+ * The regression this bound exists to avoid: real writers reserve BAT
capacity ahead of
+ * use, so these all declare well past their own length -- SolidWorks by
26 BAT blocks
+ * where 16 would do, the encrypted workbook by 12x. Under a one-BAT-block
rule every one
+ * of them fell back to spooling the object to a temp file.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {
+ "testEXCEL_protected_passtika_2.xlsx",
+ "testPPT_comment.ppt",
+ "testPPT_macros.ppt",
+ "testPPT_oleWorkbook.ppt",
+ "testsolidworksAssembly2014SP0.SLDASM",
+ "testsolidworksDrawing2014SP0.SLDDRW",
+ "testsolidworksPart2013SP2.SLDPRT",
+ "testsolidworksPart2014SP0.SLDPRT"})
+ public void testOverDeclaringRealFilesAreBelieved(String name) throws
Exception {
+ byte[] bytes;
+ try (InputStream is = getResourceAsStream("/test-documents/" + name)) {
+ bytes = is.readAllBytes();
+ }
+ try (SeekableByteChannel channel = channelFor(bytes, name)) {
+ long declared = POIFSContainerDetector.honestDeclaredSize(channel);
+ assertTrue(declared > 0, name + " must not be rejected: declared "
+ declared +
+ " against " + bytes.length + " bytes");
+ assertTrue(declared > bytes.length,
+ name + " is only a regression fixture while it
over-declares");
+ }
+ }
+
@Test
public void testRealDocumentHeaderIsHonest() throws Exception {
byte[] bytes;
@@ -102,7 +186,7 @@ public class POIFSDeclaredSizeTest extends TikaTest {
}
try (SeekableByteChannel channel = channelFor(bytes, "real.doc")) {
long declared = POIFSContainerDetector.honestDeclaredSize(channel);
- assertTrue(declared >= bytes.length && declared <= bytes.length +
128 * 512L,
+ assertTrue(declared >= bytes.length && declared <= bytes.length *
16L,
"a real header declares about its own size: " + declared +
" vs " + bytes.length);
}
}
@@ -114,6 +198,33 @@ public class POIFSDeclaredSizeTest extends TikaTest {
}
}
+ /**
+ * End to end, and the point of the whole change: an over-declaring file
detected from
+ * memory is opened in memory and never touches disk. Before the
amplification bound these
+ * fell to the file path, which materialised a temp file for every one of
them.
+ */
+ @ParameterizedTest
+ @ValueSource(strings = {"testPPT_comment.ppt",
"testsolidworksPart2013SP2.SLDPRT"})
+ public void testOverDeclaringFileIsOpenedFromMemory(String name) throws
Exception {
+ byte[] bytes;
+ try (InputStream is = getResourceAsStream("/test-documents/" + name)) {
+ bytes = is.readAllBytes();
+ }
+ ParseContext context = new ParseContext();
+ CacheMemoryBudget budget = new CacheMemoryBudget(1024L * 1024 * 1024);
+ context.set(CacheMemoryBudget.class, budget);
+ Metadata metadata = new Metadata();
+ try (TemporaryResources tmp = new TemporaryResources()) {
+ TikaInputStream tis =
+ TikaInputStream.get(new ByteArrayInputStream(bytes), tmp,
metadata);
+ new POIFSContainerDetector().detect(tis, metadata, context);
+ assertNotNull(tis.getOpenContainer(), name + " must be opened from
memory");
+ assertFalse(tis.hasFile(), name + " must not have been spooled to
disk");
+ assertTrue(budget.getReservedBytes() > 0, "the copy must be
charged while open");
+ }
+ assertEquals(0, budget.getReservedBytes(), "released when the stream
closes");
+ }
+
/**
* End to end. NOTE: this asserts only that the crafted object does not
become an open
* container and leaves nothing charged -- both of which also hold if the
declared-size
