This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new a49480cda2 TIKA-4814 -- OneNote follow-ups: fall back to legacy string
dump when… (#3045)
a49480cda2 is described below
commit a49480cda27483c020f4d4ad3bd65903610bc63b
Author: Tim Allison <[email protected]>
AuthorDate: Fri Aug 21 21:24:53 2026 -0400
TIKA-4814 -- OneNote follow-ups: fall back to legacy string dump when…
(#3045)
---
.skills/oss-fuzz/SKILL.md | 20 ++
CHANGES.txt | 17 +-
.../apache/tika/parser/microsoft/onenote/GUID.java | 18 +-
.../parser/microsoft/onenote/OneNoteDocument.java | 3 +
.../parser/microsoft/onenote/OneNoteParser.java | 147 +++++++++-----
.../tika/parser/microsoft/onenote/OneNotePtr.java | 44 ++++-
.../microsoft/onenote/OneNoteTreeWalker.java | 46 ++++-
.../onenote/fsshttpb/MSOneStorePackage.java | 161 +++++++++------
.../property/PrtArrayOfPropertyValues.java | 13 +-
.../PrtFourBytesOfLengthFollowedByData.java | 6 +
.../onenote/fsshttpb/streamobj/ObjectDataBLOB.java | 19 --
.../streamobj/ObjectDataBLOBDataElementData.java | 15 --
.../onenote/fsshttpb/streamobj/PropertySet.java | 23 ++-
.../streamobj/RevisionStoreObjectGroup.java | 12 +-
.../space/ObjectSpaceObjectStreamOfContextIDs.java | 8 +-
.../space/ObjectSpaceObjectStreamOfOIDs.java | 8 +-
.../space/ObjectSpaceObjectStreamOfOSIDs.java | 6 +
.../tika/parser/microsoft/onenote/GUIDTest.java | 19 ++
.../microsoft/onenote/OneNoteParserTest.java | 219 ++++++++++++++++++++-
.../onenote/fsshttpb/MSOneStoreBlobTest.java | 17 +-
.../onenote/fsshttpb/MSOneStorePackageTest.java | 112 ++++++++++-
.../PropertyDeserializationBoundsTest.java | 184 +++++++++++++++++
22 files changed, 934 insertions(+), 183 deletions(-)
diff --git a/.skills/oss-fuzz/SKILL.md b/.skills/oss-fuzz/SKILL.md
index ae3d6ea84b..f18098381c 100644
--- a/.skills/oss-fuzz/SKILL.md
+++ b/.skills/oss-fuzz/SKILL.md
@@ -312,6 +312,26 @@ and is safe — it runs a single testcase, not a corpus dir,
so the
- **New target:** add `FooParserFuzzer.java` next to the others following the
`ParserFuzzer.parseOne` + swallow-expected-exceptions pattern.
+## Cleanup — the container leaves root-owned files in your working tree
+
+`build_fuzzers` with a local `--mount_path` compiles your working tree *inside
+the container as root*, so `target/` dirs (and other build outputs) in the
+mounted checkout come back **root-owned** — a later host-side `./mvnw clean`
+then fails with permission errors. When you are done fuzzing, run the clean
+**from the container** (root can delete its own files) against the same mount:
+
+```bash
+docker run --rm --platform linux/amd64 \
+ -v /path/to/tika:/src/project-parent/tika \
+ gcr.io/oss-fuzz/apache-tika \
+ bash -c 'cd /src/project-parent/tika && ./mvnw clean -Pfast
-Dmaven.repo.local=/tmp/m2'
+```
+
+Use a throwaway in-container repo path for `-Dmaven.repo.local` (as above) so
+the clean itself does not write root-owned files into the host
`.local_m2_repo`.
+Verify nothing is left behind: `find /path/to/tika -user root | head` should
+print nothing.
+
## Disclosure caveat (read before touching the public project)
The `apache-tika` project on Google's infra **auto-files bugs and discloses on
diff --git a/CHANGES.txt b/CHANGES.txt
index 1b59350a8c..5c32de6a3d 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -30,6 +30,17 @@ Release 4.1.0 - unreleased
CONTENT_TYPE_USER_OVERRIDE key is deliberately not carried, so the hint
cannot force an unrelated type (TIKA-4825).
+ * OneNote extraction now follows document order, omits superseded page
+ revisions, sorts author metadata, extracts embedded object BLOBs, and
+ bounds malformed-input recursion and file-derived allocations. Parse
+ warnings and embedded relationship IDs are exposed in metadata. Malformed
+ or truncated files that cannot be fully parsed, and files whose walk
+ yields no content, now fall back to the legacy string dump instead of
+ failing or returning empty output. The legacy MS-ONESTORE walker bounds
+ its recursion (depth caps plus file-node-list and fragment-chain cycle
+ guards) and now honors shouldParseEmbedded for embedded file data
+ (TIKA-4814).
+
* RawTiffParser extracts the camera-generated JPEG previews embedded in
TIFF-based raw images (Nikon NEF/NRW, Sony ARW/SRF/SR2, Pentax PEF/PTX,
Adobe DNG and Canon CR2, including BigTIFF DNG containers) as thumbnail
@@ -510,12 +521,6 @@ Release 4.0.0 - 8/18/2026
OTHER CHANGES
- * OneNote extraction now follows document order, omits superseded page
- revisions, sorts author metadata, extracts embedded object BLOBs, and
- bounds malformed-input recursion and per-file-node-list property
allocation.
- Malformed truncated property arrays now fail rather than return partial
data;
- parse warnings and embedded relationship IDs are exposed in metadata
(TIKA-4814).
-
* Dependency upgrades since 4.0.0-beta-1, including Jetty 12.1.12, CXF
4.2.3 and SolrJ 10.0.0 (TIKA-4327).
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java
index fe63cc9bf4..55e788bb4a 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/GUID.java
@@ -56,8 +56,8 @@ public class GUID implements Comparable<GUID> {
throw new TikaException("Invalid GUID string");
}
for (int i = 0; i < hex.length(); i += 2) {
- int high = Character.digit(hex.charAt(i), 16);
- int low = Character.digit(hex.charAt(i + 1), 16);
+ int high = asciiHexDigit(hex.charAt(i));
+ int low = asciiHexDigit(hex.charAt(i + 1));
if (high < 0 || low < 0) {
throw new TikaException("Invalid GUID string");
}
@@ -66,6 +66,20 @@ public class GUID implements Comparable<GUID> {
return new GUID(intGuid);
}
+ // Character.digit accepts non-ASCII Unicode digits; GUIDs are ASCII hex
only
+ private static int asciiHexDigit(char c) {
+ if (c >= '0' && c <= '9') {
+ return c - '0';
+ }
+ if (c >= 'a' && c <= 'f') {
+ return c - 'a' + 10;
+ }
+ if (c >= 'A' && c <= 'F') {
+ return c - 'A' + 10;
+ }
+ return -1;
+ }
+
public static int memcmp(int[] b1, int[] b2, int sz) {
for (int i = 0; i < sz; i++) {
if (b1[i] != b2[i]) {
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java
index 936228f25c..cd76a58a55 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteDocument.java
@@ -34,6 +34,9 @@ class OneNoteDocument {
Map<ExtendedGUID, Pair<Long, ExtendedGUID>> revisionRoleMap = new
HashMap<>();
ExtendedGUID currentRevision = ExtendedGUID.nil();
FileNodeList root = new FileNodeList();
+ // set when the root file node list could not be fully parsed (e.g. a
truncated file);
+ // the header and any structure parsed so far remain usable
+ Exception structureParseException;
public OneNoteDocument() {
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java
index b7e498cebf..14fd500a50 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteParser.java
@@ -129,39 +129,56 @@ public class OneNoteParser implements Parser {
metadata.set(OneNote.RGB_PLACEHOLDER,
"0x" +
Long.toHexString(oneNoteDocument.header.rgbPlaceholder));
- Pair<Long, ExtendedGUID> roleAndContext = Pair.of(1L,
ExtendedGUID.nil());
- OneNoteTreeWalker oneNoteTreeWalker =
- new OneNoteTreeWalker(options, oneNoteDocument,
oneNoteDirectFileResource,
- xhtml, metadata, context, roleAndContext);
+ Exception structureFailure =
oneNoteDocument.structureParseException;
+ boolean walked = false;
+ if (structureFailure == null) {
+ try {
+ Pair<Long, ExtendedGUID> roleAndContext = Pair.of(1L,
ExtendedGUID.nil());
+ OneNoteTreeWalker oneNoteTreeWalker =
+ new OneNoteTreeWalker(options, oneNoteDocument,
+ oneNoteDirectFileResource, xhtml,
metadata, context,
+ roleAndContext);
- oneNoteTreeWalker.walkTree();
+ oneNoteTreeWalker.walkTree();
- if (!oneNoteTreeWalker.getAuthors().isEmpty()) {
- metadata.set(TikaCoreProperties.CREATOR,
- sortedValues(oneNoteTreeWalker.getAuthors()));
- }
- if (!oneNoteTreeWalker.getMostRecentAuthors().isEmpty()) {
- metadata.set(OneNote.MOST_RECENT_AUTHORS,
-
sortedValues(oneNoteTreeWalker.getMostRecentAuthors()));
- }
- if (!oneNoteTreeWalker.getOriginalAuthors().isEmpty()) {
- metadata.set(OneNote.ORIGINAL_AUTHORS,
-
sortedValues(oneNoteTreeWalker.getOriginalAuthors()));
- }
- if (!Instant.MAX.equals(
-
Instant.ofEpochMilli(oneNoteTreeWalker.getCreationTimestamp()))) {
- metadata.set(OneNote.CREATION_TIMESTAMP,
-
String.valueOf(oneNoteTreeWalker.getCreationTimestamp()));
- }
- if
(!Instant.MIN.equals(oneNoteTreeWalker.getLastModifiedTimestamp())) {
- metadata.set(OneNote.LAST_MODIFIED_TIMESTAMP,
String.valueOf(
-
oneNoteTreeWalker.getLastModifiedTimestamp().toEpochMilli()));
+ if (!oneNoteTreeWalker.getAuthors().isEmpty()) {
+ metadata.set(TikaCoreProperties.CREATOR,
+
sortedValues(oneNoteTreeWalker.getAuthors()));
+ }
+ if
(!oneNoteTreeWalker.getMostRecentAuthors().isEmpty()) {
+ metadata.set(OneNote.MOST_RECENT_AUTHORS,
+
sortedValues(oneNoteTreeWalker.getMostRecentAuthors()));
+ }
+ if (!oneNoteTreeWalker.getOriginalAuthors().isEmpty())
{
+ metadata.set(OneNote.ORIGINAL_AUTHORS,
+
sortedValues(oneNoteTreeWalker.getOriginalAuthors()));
+ }
+ if (!Instant.MAX.equals(
+
Instant.ofEpochMilli(oneNoteTreeWalker.getCreationTimestamp()))) {
+ metadata.set(OneNote.CREATION_TIMESTAMP,
+
String.valueOf(oneNoteTreeWalker.getCreationTimestamp()));
+ }
+ if
(!Instant.MIN.equals(oneNoteTreeWalker.getLastModifiedTimestamp())) {
+ metadata.set(OneNote.LAST_MODIFIED_TIMESTAMP,
String.valueOf(
+
oneNoteTreeWalker.getLastModifiedTimestamp().toEpochMilli()));
+ }
+ if (oneNoteTreeWalker.getLastModified() >
Long.MIN_VALUE) {
+ metadata.set(TikaCoreProperties.MODIFIED,
+
String.valueOf(oneNoteTreeWalker.getLastModified()));
+ }
+ walked = true;
+ } catch (Exception e) {
+ rethrowIfLimitReached(e);
+ structureFailure = e;
+ }
}
- if (oneNoteTreeWalker.getLastModified() > Long.MIN_VALUE) {
- metadata.set(TikaCoreProperties.MODIFIED,
-
String.valueOf(oneNoteTreeWalker.getLastModified()));
+ if (!walked) {
+ legacyFallbackDump("OneNote parse failed; falling back to
legacy text dump: " +
+ failureMessage(structureFailure),
structureFailure, metadata,
+ xhtml, oneNoteDirectFileResource);
}
} else if (header.isLegacyOrAlternativePackaging()) {
+ MSOneStorePackage pkg = null;
try {
AlternativePackaging alternatePackageOneStoreFile = new
AlternativePackaging();
byte[] bytes;
@@ -172,27 +189,18 @@ public class OneNoteParser implements Parser {
alternatePackageOneStoreFile.doDeserializeFromByteArray(bytes, 0);
MSOneStoreParser onenoteParser = new MSOneStoreParser();
- MSOneStorePackage pkg =
-
onenoteParser.parse(alternatePackageOneStoreFile.dataElementPackage);
+ pkg =
onenoteParser.parse(alternatePackageOneStoreFile.dataElementPackage);
pkg.walkTree(options, metadata, xhtml, context);
} catch (Exception e) {
- WriteLimitReachedException.throwIfWriteLimitReached(e);
- if (e instanceof EmbeddedLimitReachedException) {
- throw (EmbeddedLimitReachedException) e;
- }
- String failure = e.getMessage() == null ?
e.getClass().getSimpleName() :
- e.getMessage();
- LOG.warn("OneNote FSSHTTPB parse failed; falling back to
legacy text dump: {}",
- failure);
- LOG.debug("OneNote FSSHTTPB parse failure", e);
-
metadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING,
+ rethrowIfLimitReached(e);
+ legacyFallbackDump(
"OneNote FSSHTTPB parse failed; falling back to
legacy text dump: " +
- failure);
- OneNoteLegacyDumpStrings dumpStrings =
- new
OneNoteLegacyDumpStrings(oneNoteDirectFileResource, xhtml);
- dumpStrings.dump();
+ failureMessage(e), e, metadata, xhtml,
+ oneNoteDirectFileResource);
+ pkg = null;
}
+ legacyFallbackIfNoContent(pkg, metadata, xhtml,
oneNoteDirectFileResource);
} else {
throw new TikaException("Invalid OneStore document - could not
parse headers");
}
@@ -208,6 +216,42 @@ public class OneNoteParser implements Parser {
return sorted;
}
+ private static void rethrowIfLimitReached(Exception e) throws
TikaException, SAXException {
+ WriteLimitReachedException.throwIfWriteLimitReached(e);
+ if (e instanceof EmbeddedLimitReachedException) {
+ throw (EmbeddedLimitReachedException) e;
+ }
+ }
+
+ private static String failureMessage(Exception e) {
+ return e.getMessage() == null ? e.getClass().getSimpleName() :
e.getMessage();
+ }
+
+ // the walk completed but every page dangled - without this a degraded
+ // file would yield empty output where the dump still finds its text
+ static void legacyFallbackIfNoContent(MSOneStorePackage pkg, Metadata
metadata,
+ XHTMLContentHandler xhtml,
+ OneNoteDirectFileResource
oneNoteDirectFileResource)
+ throws TikaException, SAXException {
+ if (pkg != null && !pkg.hasEmittedContent()) {
+ legacyFallbackDump("OneNote FSSHTTPB parse produced no content; " +
+ "falling back to legacy text dump", null,
metadata, xhtml,
+ oneNoteDirectFileResource);
+ }
+ }
+
+ private static void legacyFallbackDump(String warning, Exception cause,
Metadata metadata,
+ XHTMLContentHandler xhtml,
+ OneNoteDirectFileResource
oneNoteDirectFileResource)
+ throws TikaException, SAXException {
+ LOG.warn(warning);
+ if (cause != null) {
+ LOG.debug("OneNote parse failure", cause);
+ }
+ metadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING, warning);
+ new OneNoteLegacyDumpStrings(oneNoteDirectFileResource, xhtml).dump();
+ }
+
/**
* Create a OneNoteDocument object.
* <p>
@@ -241,7 +285,8 @@ public class OneNoteParser implements Parser {
* content.
* @return A parsed one note document. This document does not contain any
of the binary data,
* rather it just contains
- * the data pointers and metadata.
+ * the data pointers and metadata. A failure while parsing the root file
node list is not
+ * thrown; it is recorded in the returned document's {@code
structureParseException}.
* @throws IOException Will throw IOException in typical IO issue
situations.
*/
public OneNoteDocument createOneNoteDocumentFromDirectFileResource(
@@ -253,9 +298,15 @@ public class OneNoteParser implements Parser {
if (oneNoteDocument.header.isMsOneStoreFormat()) {
// Now that we parsed the header, the "root file node list"
- oneNotePtr.reposition(oneNoteDocument.header.fcrFileNodeListRoot);
- FileNodePtr curPath = new FileNodePtr();
- oneNotePtr.deserializeFileNodeList(oneNoteDocument.root, curPath);
+ try {
+
oneNotePtr.reposition(oneNoteDocument.header.fcrFileNodeListRoot);
+ FileNodePtr curPath = new FileNodePtr();
+ oneNotePtr.deserializeFileNodeList(oneNoteDocument.root,
curPath);
+ } catch (TikaException | IOException | RuntimeException e) {
+ // a truncated or malformed root list is recorded, not thrown,
so the
+ // caller can fall back to the legacy string dump
+ oneNoteDocument.structureParseException = e;
+ }
}
return oneNoteDocument;
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNotePtr.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNotePtr.java
index 30ec931658..916b9bbb5a 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNotePtr.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNotePtr.java
@@ -21,8 +21,10 @@ import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.Arrays;
+import java.util.HashSet;
import java.util.List;
import java.util.Map;
+import java.util.Set;
import org.apache.commons.codec.binary.Hex;
import org.apache.commons.io.EndianUtils;
@@ -55,13 +57,26 @@ class OneNotePtr {
"{638DE92F-A6D4-4BC1-9A36-B3FC2511A5B7}";
private static final int MAX_PROPERTY_VALUES = 100_000;
private static final int MAX_PROPERTY_SET_DEPTH = 1000;
+ // the format nests file node lists only a handful of levels deep; 100
leaves wide
+ // margin while keeping the recursion far from any stack limit
+ private static final int MAX_FILE_NODE_LIST_DEPTH = 100;
private static final class PropertyValueBudget {
private long remaining = MAX_PROPERTY_VALUES;
}
+ /**
+ * Recursion state for the baseType-2 file-node-list nesting, shared by
every pointer
+ * copied while parsing one document.
+ */
+ private static final class FileNodeListRecursion {
+ private int depth;
+ private final Set<Long> activeListOffsets = new HashSet<>();
+ }
+
int indentLevel = 0;
private PropertyValueBudget propertyValueBudget = new
PropertyValueBudget();
+ private FileNodeListRecursion fileNodeListRecursion = new
FileNodeListRecursion();
long offset;
long end;
@@ -84,6 +99,7 @@ class OneNotePtr {
this.end = oneNotePtr.end;
this.indentLevel = oneNotePtr.indentLevel;
this.propertyValueBudget = oneNotePtr.propertyValueBudget;
+ this.fileNodeListRecursion = oneNotePtr.fileNodeListRecursion;
}
public OneNoteHeader deserializeHeader() throws IOException, TikaException
{
@@ -269,6 +285,9 @@ class OneNotePtr {
throws IOException, TikaException {
OneNotePtr localPtr = new OneNotePtr(ptr);
FileNodePtrBackPush bp = new FileNodePtrBackPush(curPath);
+ // a next-fragment reference pointing at an already-seen fragment
would loop forever
+ Set<Long> seenFragmentOffsets = new HashSet<>();
+ seenFragmentOffsets.add(ptr.offset);
try {
while (true) {
FileChunkReference next = FileChunkReference.nil();
@@ -276,6 +295,11 @@ class OneNotePtr {
if (FileChunkReference.nil().equals(next)) {
break;
}
+ if (!seenFragmentOffsets.add(next.stp)) {
+ throw new TikaException(
+ "OneNote file node list fragment cycle detected at
offset " +
+ next.stp);
+ }
localPtr.reposition(next);
ptr = localPtr;
}
@@ -292,8 +316,24 @@ class OneNotePtr {
*/
public OneNotePtr deserializeFileNodeList(FileNodeList fileNodeList,
FileNodePtr curPath)
throws IOException, TikaException {
- propertyValueBudget = new PropertyValueBudget();
- return internalDeserializeFileNodeList(this, fileNodeList, curPath);
+ if (fileNodeListRecursion.depth >= MAX_FILE_NODE_LIST_DEPTH) {
+ throw new TikaMemoryLimitException(
+ "OneNote file node list nesting exceeds depth limit " +
+ MAX_FILE_NODE_LIST_DEPTH);
+ }
+ long listOffset = offset;
+ if (!fileNodeListRecursion.activeListOffsets.add(listOffset)) {
+ throw new TikaException(
+ "OneNote file node list cycle detected at offset " +
listOffset);
+ }
+ fileNodeListRecursion.depth++;
+ try {
+ propertyValueBudget = new PropertyValueBudget();
+ return internalDeserializeFileNodeList(this, fileNodeList,
curPath);
+ } finally {
+ fileNodeListRecursion.depth--;
+ fileNodeListRecursion.activeListOffsets.remove(listOffset);
+ }
}
/**
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteTreeWalker.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteTreeWalker.java
index c98c41dbd1..943520e37f 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteTreeWalker.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteTreeWalker.java
@@ -60,6 +60,8 @@ class OneNoteTreeWalker {
private static final Logger LOG =
LoggerFactory.getLogger(OneNoteTreeWalker.class);
private static final int MAX_CYCLE_WARNINGS = 100;
+ // low enough that the ~3 stack frames per level cannot overflow a default
thread stack
+ private static final int MAX_WALK_DEPTH = 500;
private static final String P = "p";
/**
* See spec MS-ONE - 2.3.1 - TIME32 - epoch of jan 1 1980 UTC.
@@ -115,6 +117,8 @@ class OneNoteTreeWalker {
Collections.newSetFromMap(new IdentityHashMap<>());
private int cycleWarningCount;
private boolean cycleWarningsSuppressed;
+ private int walkDepth;
+ private boolean depthWarningRecorded;
/**
* Create a one tree walker.
@@ -307,17 +311,37 @@ class OneNoteTreeWalker {
if (fileNode == null) {
return Collections.emptyMap();
}
+ if (walkDepth >= MAX_WALK_DEPTH) {
+ // the cycle guard cannot catch a long acyclic chain - cap the
recursion depth
+ // so a crafted file cannot overflow the stack
+ recordDepthWarning();
+ return Collections.emptyMap();
+ }
if (!activeFileNodes.add(fileNode)) {
recordCycleWarning(fileNode);
return Collections.emptyMap();
}
+ walkDepth++;
try {
return walkFileNodeInternal(fileNode, parentPropertyId);
} finally {
+ walkDepth--;
activeFileNodes.remove(fileNode);
}
}
+ private void recordDepthWarning() {
+ if (depthWarningRecorded) {
+ return;
+ }
+ depthWarningRecorded = true;
+ String warning = "OneNote file-node traversal exceeded depth limit " +
MAX_WALK_DEPTH;
+ LOG.warn(warning);
+ if (parentMetadata != null) {
+ parentMetadata.add(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING,
warning);
+ }
+ }
+
private void recordCycleWarning(FileNode fileNode) {
if (reportedCycleNodes.contains(fileNode)) {
return;
@@ -368,7 +392,7 @@ class OneNoteTreeWalker {
if (fileNode.subType.fileDataStoreObjectReference.ref != null &&
!FileChunkReference.nil()
.equals(fileNode.subType.fileDataStoreObjectReference.ref.fileData)) {
structure.put("fileDataStoreObjectReference",
walkFileDataStoreObjectReference(
- fileNode.subType.fileDataStoreObjectReference));
+ fileNode.subType.fileDataStoreObjectReference,
fileNode.gosid));
}
return structure;
}
@@ -381,7 +405,7 @@ class OneNoteTreeWalker {
* @throws IOException Can throw these when manipulating the seekable byte
channel.
*/
private Map<String, Object> walkFileDataStoreObjectReference(
- FileDataStoreObjectReference fileDataStoreObjectReference)
+ FileDataStoreObjectReference fileDataStoreObjectReference,
ExtendedGUID gosid)
throws IOException, SAXException, TikaException {
Map<String, Object> structure = new HashMap<>();
OneNotePtr content = new OneNotePtr(oneNoteDocument, dif);
@@ -391,12 +415,13 @@ class OneNoteTreeWalker {
"File data store cb " +
fileDataStoreObjectReference.ref.fileData.cb +
" exceeds document size: " + dif.size());
}
- handleEmbedded((int) fileDataStoreObjectReference.ref.fileData.cb);
+ handleEmbedded((int) fileDataStoreObjectReference.ref.fileData.cb,
gosid);
structure.put("fileDataStoreObjectMetadata",
fileDataStoreObjectReference);
return structure;
}
- private void handleEmbedded(int length) throws TikaException, IOException,
SAXException {
+ private void handleEmbedded(int length, ExtendedGUID gosid)
+ throws TikaException, IOException, SAXException {
TikaInputStream tis = null;
ByteBuffer buf;
try {
@@ -408,14 +433,21 @@ class OneNoteTreeWalker {
return;
}
Metadata embeddedMetadata = Metadata.newInstance(this.parseContext);
+ embeddedMetadata.set(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE,
+ TikaCoreProperties.EmbeddedResourceType.ATTACHMENT.toString());
+ if (gosid != null && !ExtendedGUID.nil().equals(gosid)) {
+ embeddedMetadata.set(TikaCoreProperties.EMBEDDED_RELATIONSHIP_ID,
gosid.toString());
+ }
try {
AttributesImpl attributes = new AttributesImpl();
attributes.addAttribute("", "class", "class", "CDATA", "embedded");
xhtml.startElement("div", attributes);
xhtml.endElement("div");
- tis = TikaInputStream.get(buf.array());
- embeddedDocumentExtractor.parseEmbedded(tis, new
EmbeddedContentHandler(xhtml),
- embeddedMetadata, this.parseContext, false);
+ if
(embeddedDocumentExtractor.shouldParseEmbedded(embeddedMetadata, parseContext))
{
+ tis = TikaInputStream.get(buf.array());
+ embeddedDocumentExtractor.parseEmbedded(tis, new
EmbeddedContentHandler(xhtml),
+ embeddedMetadata, this.parseContext, false);
+ }
} finally {
IOUtils.closeQuietly(tis);
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
index b8c9bd0b31..0ad1d277dd 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
@@ -28,6 +28,7 @@ import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.HashSet;
+import java.util.IdentityHashMap;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -142,6 +143,11 @@ public class MSOneStorePackage {
private final Set<String> recordedParseWarningKeys = new HashSet<>();
private boolean parseWarningsSuppressed;
private boolean storageMappingsIndexed;
+ private boolean contentEmitted;
+ // flattened property actions per object; objects can be re-flattened many
times during
+ // picture/resource-name resolution, which is quadratic without this cache
+ private final Map<RevisionStoreObject, List<PropertyAction>>
objectActionsCache =
+ new IdentityHashMap<>();
private final Map<CellID, StorageIndexCellMapping>
storageIndexCellMappingsById =
new HashMap<>();
private final Map<ExGuid, StorageIndexRevisionMapping>
storageIndexRevisionMappingsById =
@@ -305,8 +311,7 @@ public class MSOneStorePackage {
}
}
for (RevisionStoreCell cell : remainingCells.values()) {
- if (cell.cellID == null || cell.cellID.extendGUID2 == null ||
- !coveredObjectSpaces.contains(cell.cellID.extendGUID2)) {
+ if (!coveredObjectSpaces.contains(cell.cellID.extendGUID2)) {
// not an older version of one of the pages - keep it so no
content is lost
otherCells.add(cell);
}
@@ -350,7 +355,7 @@ public class MSOneStorePackage {
if (object.objectID != null && !visited.add(object.objectID)) {
return;
}
- List<PropertyAction> actions = collectObjectActions(object, depth);
+ List<PropertyAction> actions = collectObjectActions(object);
for (PropertyAction action : actions) {
if (action.spaceReference != null) {
out.add(action.spaceReference);
@@ -373,19 +378,26 @@ public class MSOneStorePackage {
// Only objects reachable from the root objects of the current
revision are part of
// the current content. The object groups may also contain older,
superseded versions
// of objects (under a different object ID); those are intentionally
not walked.
- boolean resolvedRoot = false;
+ // A blob-only root (no property set) cannot reach the page body, so
it does not
+ // count as a resolved content root: if the content root declare
dangles next to it,
+ // the walk-everything fallback must still fire or the page body is
silently lost.
+ boolean resolvedContentRoot = false;
+ boolean unresolvedRoot = false;
for (RevisionManifestRootDeclare rootDeclare : cell.rootDeclares) {
RevisionStoreObject rootObject =
objectsById.get(rootDeclare.objectExGuid);
if (rootObject == null) {
recordParseWarning("OneNote cell root object " +
rootDeclare.objectExGuid +
" could not be resolved");
+ unresolvedRoot = true;
} else {
- resolvedRoot = true;
+ if (rootObject.propertySet != null) {
+ resolvedContentRoot = true;
+ }
walkObject(rootObject, objectsById, visited, AuthorRole.NONE,
options,
metadata, xhtml, 0);
}
}
- if (!resolvedRoot) {
+ if (cell.rootDeclares.isEmpty() || (unresolvedRoot &&
!resolvedContentRoot)) {
if (cell.rootDeclares.isEmpty()) {
recordParseWarning("OneNote cell has no declared root objects;
walking all objects");
} else {
@@ -481,11 +493,15 @@ public class MSOneStorePackage {
return;
}
if (object.objectID != null && !visited.add(object.objectID)) {
+ // one object may be referenced under several author roles (e.g.
the same author
+ // as both AuthorOriginal and AuthorMostRecent) - record the role
even though
+ // the subtree is not walked again
+ recordAuthors(object, authorRole);
return;
}
List<PropertyAction> actions = object.propertySet != null &&
object.propertySet.objectSpaceObjectPropSet != null ?
- collectObjectActions(object, depth) : Collections.emptyList();
+ collectObjectActions(object) : Collections.emptyList();
EmbeddedResourceInfo resourceInfo = embeddedResourceInfo(actions);
if (resourceInfo == null) {
resourceInfo = inheritedResourceInfo;
@@ -552,7 +568,7 @@ public class MSOneStorePackage {
if (object.propertySet == null ||
object.propertySet.objectSpaceObjectPropSet == null) {
return false;
}
- for (PropertyAction action : collectObjectActions(object, depth)) {
+ for (PropertyAction action : collectObjectActions(object)) {
if (action.isChildReference && action.childReference != null &&
hasUsablePicture(action.childReference, objectsById,
visited, depth + 1)) {
return true;
@@ -572,7 +588,7 @@ public class MSOneStorePackage {
candidate.propertySet.objectSpaceObjectPropSet == null) {
continue;
}
- List<PropertyAction> candidateActions =
collectObjectActions(candidate, depth);
+ List<PropertyAction> candidateActions =
collectObjectActions(candidate);
for (PropertyAction action : candidateActions) {
if (action.isChildReference &&
target.objectID.equals(action.childReference) &&
(action.oneNotePropertyEnum ==
OneNotePropertyEnum.PictureContainer ||
@@ -643,22 +659,24 @@ public class MSOneStorePackage {
}
/**
- * Flattens the properties of an object, in order, into a list of actions.
+ * Flattens the properties of an object, in order, into a list of actions.
The result is
+ * cached per object; callers must not mutate the returned list.
*/
private List<PropertyAction> collectObjectActions(RevisionStoreObject
object) {
- return collectObjectActions(object, 0);
- }
-
- private List<PropertyAction> collectObjectActions(RevisionStoreObject
object, int depth) {
+ List<PropertyAction> actions = objectActionsCache.get(object);
+ if (actions != null) {
+ return actions;
+ }
List<ExGuid> referencedObjects =
object.referencedObjectID == null ||
object.referencedObjectID.content == null ?
Collections.emptyList() :
object.referencedObjectID.content;
List<CellID> referencedSpaces = object.referencedObjectSpacesID ==
null ||
object.referencedObjectSpacesID.content == null ?
Collections.emptyList() :
object.referencedObjectSpacesID.content;
- List<PropertyAction> actions = new ArrayList<>();
+ actions = new ArrayList<>();
collectActions(object.propertySet.objectSpaceObjectPropSet.body,
referencedObjects,
- new int[]{0}, referencedSpaces, new int[]{0}, actions, depth);
+ new int[]{0}, referencedSpaces, new int[]{0}, actions, 0);
+ objectActionsCache.put(object, actions);
return actions;
}
@@ -670,13 +688,6 @@ public class MSOneStorePackage {
* section 2.7.8), so the references must be assigned here, in property
order, no matter
* in which order the actions are processed later.
*/
- private void collectActions(PropertySet propertySet, List<ExGuid>
referencedObjects,
- int[] referenceCursor, List<CellID>
referencedSpaces,
- int[] spaceCursor, List<PropertyAction>
actions) {
- collectActions(propertySet, referencedObjects, referenceCursor,
referencedSpaces,
- spaceCursor, actions, 0);
- }
-
private void collectActions(PropertySet propertySet, List<ExGuid>
referencedObjects,
int[] referenceCursor, List<CellID>
referencedSpaces,
int[] spaceCursor, List<PropertyAction>
actions, int depth) {
@@ -862,17 +873,8 @@ public class MSOneStorePackage {
}
metadata.set(TikaCoreProperties.MODIFIED,
String.valueOf(lastModified));
} else if (oneNotePropertyEnum == OneNotePropertyEnum.Author) {
- String author = decodeOneNoteText(
- ((PrtFourBytesOfLengthFollowedByData) property).data);
- if (authorRole == AuthorRole.MOST_RECENT) {
- mostRecentAuthors.add(author);
- } else if (authorRole == AuthorRole.ORIGINAL) {
- originalAuthors.add(author);
- // the original authors are the creators of the content
- authors.add(author);
- } else {
- authors.add(author);
- }
+ recordAuthor(decodeOneNoteText(
+ ((PrtFourBytesOfLengthFollowedByData) property).data),
authorRole);
} else if (propertyType ==
PropertyType.FourBytesOfLengthFollowedByData) {
boolean isBinary = propertyIsBinary(oneNotePropertyEnum);
PrtFourBytesOfLengthFollowedByData dataProperty =
@@ -880,28 +882,15 @@ public class MSOneStorePackage {
if ((dataProperty.data.length & 1) == 0 &&
oneNotePropertyEnum !=
OneNotePropertyEnum.TextExtendedAscii && !isBinary) {
if
(options.getUtf16PropertiesToPrint().contains(oneNotePropertyEnum)) {
- xhtml.startElement(P);
- try {
- xhtml.characters(new String(dataProperty.data,
StandardCharsets.UTF_16LE));
- } finally {
- xhtml.endElement(P);
- }
+ emitParagraph(xhtml, new String(dataProperty.data,
+ StandardCharsets.UTF_16LE));
}
} else if (oneNotePropertyEnum ==
OneNotePropertyEnum.TextExtendedAscii) {
- xhtml.startElement(P);
- try {
- xhtml.characters(new String(dataProperty.data,
StandardCharsets.US_ASCII));
- } finally {
- xhtml.endElement(P);
- }
+ emitParagraph(xhtml, new String(dataProperty.data,
StandardCharsets.US_ASCII));
} else if (!isBinary) {
if
(options.getUtf16PropertiesToPrint().contains(oneNotePropertyEnum)) {
- xhtml.startElement(P);
- try {
- xhtml.characters(new String(dataProperty.data,
StandardCharsets.UTF_16LE));
- } finally {
- xhtml.endElement(P);
- }
+ emitParagraph(xhtml, new String(dataProperty.data,
+ StandardCharsets.UTF_16LE));
}
} else {
if (oneNotePropertyEnum ==
OneNotePropertyEnum.RichEditTextUnicode) {
@@ -920,6 +909,36 @@ public class MSOneStorePackage {
return new String(bytes, StandardCharsets.UTF_16LE).replace("\u0000",
"");
}
+ private void recordAuthor(String author, AuthorRole role) {
+ if (role == AuthorRole.MOST_RECENT) {
+ mostRecentAuthors.add(author);
+ } else if (role == AuthorRole.ORIGINAL) {
+ originalAuthors.add(author);
+ // the original authors are the creators of the content
+ authors.add(author);
+ } else {
+ authors.add(author);
+ }
+ }
+
+ /**
+ * Records the Author properties of an already-visited object under the
given role.
+ */
+ private void recordAuthors(RevisionStoreObject object, AuthorRole role) {
+ // NONE also records (into authors) so the result is visit-order
independent
+ if (object.propertySet == null ||
+ object.propertySet.objectSpaceObjectPropSet == null) {
+ return;
+ }
+ for (PropertyAction action : collectObjectActions(object)) {
+ if (action.oneNotePropertyEnum == OneNotePropertyEnum.Author &&
+ action.property instanceof
PrtFourBytesOfLengthFollowedByData) {
+ recordAuthor(decodeOneNoteText(
+ ((PrtFourBytesOfLengthFollowedByData)
action.property).data), role);
+ }
+ }
+ }
+
private EmbeddedResourceInfo embeddedResourceInfo(List<PropertyAction>
actions) {
for (PropertyAction action : actions) {
if (action.property instanceof PrtFourBytesOfLengthFollowedByData
&&
@@ -939,10 +958,11 @@ public class MSOneStorePackage {
return null;
}
- private String sanitizeResourceName(String name) {
+ static String sanitizeResourceName(String name) {
name = name.replace('\\', '/');
+ // a drive-relative name like C:pic.png has no slash, so strip the
prefix separately
if (name.length() > 1 && name.charAt(1) == ':') {
- return "";
+ name = name.substring(2);
}
int slash = name.lastIndexOf('/');
name = slash >= 0 ? name.substring(slash + 1) : name;
@@ -980,6 +1000,9 @@ public class MSOneStorePackage {
xhtml.endElement("div");
try (TikaInputStream tis = TikaInputStream.get(data)) {
if
(embeddedDocumentExtractor.shouldParseEmbedded(embeddedMetadata, parseContext))
{
+ // only counts as content when the extractor accepts it, so a
declined
+ // embedded object in an otherwise-empty file still triggers
the fallback
+ contentEmitted = true;
embeddedDocumentExtractor.parseEmbedded(tis, new
EmbeddedContentHandler(xhtml),
embeddedMetadata, this.parseContext, false);
}
@@ -1011,14 +1034,32 @@ public class MSOneStorePackage {
} finally {
xhtml.endElement("a");
}
+ contentEmitted = true;
} else {
- xhtml.startElement(P);
- try {
- xhtml.characters(txt);
- } finally {
- xhtml.endElement(P);
- }
+ emitParagraph(xhtml, txt);
+ }
+ }
+
+ private void emitParagraph(XHTMLContentHandler xhtml, String text) throws
SAXException {
+ xhtml.startElement(P);
+ try {
+ xhtml.characters(text);
+ } finally {
+ xhtml.endElement(P);
}
+ // blank text does not count as content, so it cannot suppress the
legacy fallback
+ if (!text.isBlank()) {
+ contentEmitted = true;
+ }
+ }
+
+ /**
+ * Whether the tree walk emitted any content - text or an embedded object.
When it did
+ * not, the caller can fall back to the legacy string dump so a degraded
file still
+ * yields its text.
+ */
+ public boolean hasEmittedContent() {
+ return contentEmitted;
}
private long getScalar(IProperty property) throws TikaException,
IOException {
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtArrayOfPropertyValues.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtArrayOfPropertyValues.java
index 68b1dab169..7220f999bf 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtArrayOfPropertyValues.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtArrayOfPropertyValues.java
@@ -41,16 +41,27 @@ public class PrtArrayOfPropertyValues implements IProperty {
* @return
*/
public int doDeserializeFromByteArray(byte[] byteArray, int startIndex)
throws IOException {
+ return doDeserializeFromByteArray(byteArray, startIndex, 0);
+ }
+
+ public int doDeserializeFromByteArray(byte[] byteArray, int startIndex,
int depth)
+ throws IOException {
int index = startIndex;
this.cProperties = BitConverter.toInt32(byteArray, index);
index += 4;
this.propertyID = new PropertyID();
int len = this.propertyID.doDeserializeFromByteArray(byteArray, index);
index += len;
+ // each PropertySet consumes at least its 2-byte count, so a valid
count cannot
+ // exceed the remaining bytes / 2; bounds the allocation on malformed
counts
+ if (this.cProperties < 0 || this.cProperties > (byteArray.length -
index) / 2) {
+ throw new IOException("prtArrayOfPropertyValues count " +
this.cProperties +
+ " exceeds remaining data " + (byteArray.length - index));
+ }
this.data = new PropertySet[this.cProperties];
for (int i = 0; i < this.cProperties; i++) {
this.data[i] = new PropertySet();
- int length = this.data[i].doDeserializeFromByteArray(byteArray,
index);
+ int length = this.data[i].doDeserializeFromByteArray(byteArray,
index, depth + 1);
index += length;
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtFourBytesOfLengthFollowedByData.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtFourBytesOfLengthFollowedByData.java
index 8518be09b5..4eb0ca7b83 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtFourBytesOfLengthFollowedByData.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/property/PrtFourBytesOfLengthFollowedByData.java
@@ -43,6 +43,12 @@ public class PrtFourBytesOfLengthFollowedByData implements
IProperty {
int index = startIndex;
this.cb = (int) BitConverter.toUInt32(byteArray, startIndex);
index += 4;
+ // copyOfRange zero-fills past the source end, so cb must be bounded by
+ // the remaining bytes or a malformed length allocates cb bytes of heap
+ if (this.cb < 0 || this.cb > byteArray.length - index) {
+ throw new IOException("prtFourBytesOfLengthFollowedByData length "
+ this.cb +
+ " exceeds remaining data " + (byteArray.length - index));
+ }
this.data = Arrays.copyOfRange(byteArray, index, index + this.cb);
index += this.cb;
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOB.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOB.java
index e92d04f6e9..2873e1d013 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOB.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOB.java
@@ -30,14 +30,8 @@ import
org.apache.tika.parser.microsoft.onenote.fsshttpb.util.ByteUtil;
* e.g. an embedded image or file. See MS-FSSHTTPB section 2.2.1.12.8.
*/
public class ObjectDataBLOB extends StreamObject {
- /**
- * A binary item that holds the opaque binary data.
- */
public BinaryItem data;
- /**
- * Initializes a new instance of the ObjectDataBLOB class.
- */
public ObjectDataBLOB() {
super(StreamObjectTypeHeaderStart.ObjectDataBLOB);
this.data = new BinaryItem();
@@ -53,13 +47,6 @@ public class ObjectDataBLOB extends StreamObject {
return ByteUtil.toByteArray(this.data.content);
}
- /**
- * Used to de-serialize the element.
- *
- * @param byteArray A Byte array
- * @param currentIndex Start position
- * @param lengthOfItems The length of the items
- */
@Override
protected void deserializeItemsFromByteArray(byte[] byteArray,
AtomicInteger currentIndex,
int lengthOfItems)
@@ -75,12 +62,6 @@ public class ObjectDataBLOB extends StreamObject {
currentIndex.set(index.get());
}
- /**
- * Used to convert the element into a byte List
- *
- * @param byteList A Byte list
- * @return The number of elements actually contained in the list
- */
@Override
protected int serializeItemsToByteList(List<Byte> byteList) throws
IOException {
int startPoint = byteList.size();
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOBDataElementData.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOBDataElementData.java
index 17828cf0d7..ff28e9d096 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOBDataElementData.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/ObjectDataBLOBDataElementData.java
@@ -29,20 +29,10 @@ import org.apache.tika.exception.TikaException;
public class ObjectDataBLOBDataElementData extends DataElementData {
public ObjectDataBLOB objectDataBLOB;
- /**
- * Initializes a new instance of the ObjectDataBLOBDataElementData class.
- */
public ObjectDataBLOBDataElementData() {
this.objectDataBLOB = new ObjectDataBLOB();
}
- /**
- * Deserializes this element from the supplied byte array.
- *
- * @param byteArray A byte array
- * @param startIndex Start position
- * @return The number of bytes consumed
- */
@Override
public int deserializeDataElementDataFromByteArray(byte[] byteArray, int
startIndex)
throws TikaException, IOException {
@@ -51,11 +41,6 @@ public class ObjectDataBLOBDataElementData extends
DataElementData {
return index.get() - startIndex;
}
- /**
- * Used to convert the element into a byte List.
- *
- * @return The Byte list
- */
@Override
public List<Byte> serializeToByteList() throws TikaException, IOException {
return this.objectDataBLOB.serializeToByteList();
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/PropertySet.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/PropertySet.java
index 45e1278f42..64276e5aa5 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/PropertySet.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/PropertySet.java
@@ -37,6 +37,10 @@ import
org.apache.tika.parser.microsoft.onenote.fsshttpb.util.BitConverter;
* This class is used to represent a PropertySet.
*/
public class PropertySet implements IProperty {
+ // a nesting level costs ~14 file bytes, so unbounded depth lets a small
+ // crafted file drive a StackOverflowError that escapes catch(Exception)
+ public static final int MAX_PROPERTY_NESTING = 100;
+
public int cProperties;
public PropertyID[] rgPrids;
@@ -72,6 +76,14 @@ public class PropertySet implements IProperty {
* @return Return the length in byte of the PropertySet.
*/
public int doDeserializeFromByteArray(byte[] byteArray, int startIndex)
throws IOException {
+ return doDeserializeFromByteArray(byteArray, startIndex, 0);
+ }
+
+ public int doDeserializeFromByteArray(byte[] byteArray, int startIndex,
int depth)
+ throws IOException {
+ if (depth > MAX_PROPERTY_NESTING) {
+ throw new IOException("PropertySet nesting exceeds " +
MAX_PROPERTY_NESTING);
+ }
int index = startIndex;
this.cProperties = BitConverter.toInt16(byteArray, startIndex);
@@ -124,7 +136,16 @@ public class PropertySet implements IProperty {
break;
}
if (property != null) {
- int len = property.doDeserializeFromByteArray(byteArray,
index);
+ int len;
+ if (property instanceof PropertySet) {
+ len = ((PropertySet) property)
+ .doDeserializeFromByteArray(byteArray, index,
depth + 1);
+ } else if (property instanceof PrtArrayOfPropertyValues) {
+ len = ((PrtArrayOfPropertyValues) property)
+ .doDeserializeFromByteArray(byteArray, index,
depth + 1);
+ } else {
+ len = property.doDeserializeFromByteArray(byteArray,
index);
+ }
this.rgData.add(property);
index += len;
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreObjectGroup.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreObjectGroup.java
index 1381fdbdbe..fb16a8402a 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreObjectGroup.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/RevisionStoreObjectGroup.java
@@ -18,7 +18,7 @@ package
org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj;
import java.io.IOException;
import java.util.ArrayList;
-import java.util.HashMap;
+import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
@@ -42,11 +42,16 @@ public class RevisionStoreObjectGroup {
Map<ExGuid,
DataElement> blobElements)
throws IOException {
RevisionStoreObjectGroup objectGroup = new
RevisionStoreObjectGroup(objectGroupId);
- Map<ExGuid, RevisionStoreObject> objectDict = new HashMap<>();
+ // insertion-ordered so the object list (and any degraded-file
fallback walk) is
+ // deterministic
+ Map<ExGuid, RevisionStoreObject> objectDict = new LinkedHashMap<>();
if (!isEncryption) {
RevisionStoreObject revisionObject = null;
for (int i = 0; i <
dataObject.objectGroupDeclarations.objectDeclarationList.size();
i++) {
+ if (i >=
dataObject.objectGroupData.objectGroupObjectDataList.size()) {
+ throw new IOException("Missing object data for object
declaration " + i);
+ }
ObjectGroupObjectDeclare objectDeclaration =
dataObject.objectGroupDeclarations.objectDeclarationList.get(i);
ObjectGroupObjectData objectData =
@@ -108,6 +113,9 @@ public class RevisionStoreObjectGroup {
} else {
for (int i = 0; i <
dataObject.objectGroupDeclarations.objectDeclarationList.size();
i++) {
+ if (i >=
dataObject.objectGroupData.objectGroupObjectDataList.size()) {
+ throw new IOException("Missing object data for object
declaration " + i);
+ }
ObjectGroupObjectDeclare objectDeclaration =
dataObject.objectGroupDeclarations.objectDeclarationList.get(i);
ObjectGroupObjectData objectData =
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfContextIDs.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfContextIDs.java
index fe6d288330..7e1e348e7d 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfContextIDs.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfContextIDs.java
@@ -57,10 +57,16 @@ public class ObjectSpaceObjectStreamOfContextIDs {
int headerCount = this.header.doDeserializeFromByteArray(byteArray,
index);
index += headerCount;
+ // each CompactID consumes 4 bytes, so a valid count cannot exceed the
remaining
+ // bytes / 4; bounds the allocation on malformed counts
+ if (this.header.count > (byteArray.length - index) / 4L) {
+ throw new IOException("ObjectSpaceObjectStreamOfContextIDs count "
+
+ this.header.count + " exceeds remaining data " +
(byteArray.length - index));
+ }
this.body = new CompactID[(int) this.header.count];
for (int i = 0; i < this.header.count; i++) {
CompactID compactID = new CompactID();
- int count = compactID.doDeserializeFromByteArray(byteArray,
startIndex);
+ int count = compactID.doDeserializeFromByteArray(byteArray, index);
this.body[i] = compactID;
index += count;
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOIDs.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOIDs.java
index 7901dd015e..53398663c3 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOIDs.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOIDs.java
@@ -58,10 +58,16 @@ public class ObjectSpaceObjectStreamOfOIDs {
int headerCount = this.header.doDeserializeFromByteArray(byteArray,
index);
index += headerCount;
+ // each CompactID consumes 4 bytes, so a valid count cannot exceed the
remaining
+ // bytes / 4; bounds the allocation on malformed counts
+ if (this.header.count > (byteArray.length - index) / 4L) {
+ throw new IOException("ObjectSpaceObjectStreamOfOIDs count " +
this.header.count +
+ " exceeds remaining data " + (byteArray.length - index));
+ }
this.body = new CompactID[(int) this.header.count];
for (int i = 0; i < this.header.count; i++) {
CompactID compactID = new CompactID();
- int count = compactID.doDeserializeFromByteArray(byteArray,
startIndex);
+ int count = compactID.doDeserializeFromByteArray(byteArray, index);
this.body[i] = compactID;
index += count;
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOSIDs.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOSIDs.java
index 61f7deda90..f37de26f76 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOSIDs.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/streamobj/space/ObjectSpaceObjectStreamOfOSIDs.java
@@ -57,6 +57,12 @@ public class ObjectSpaceObjectStreamOfOSIDs {
int headerCount = this.header.doDeserializeFromByteArray(byteArray,
index);
index += headerCount;
+ // each CompactID consumes 4 bytes, so a valid count cannot exceed the
remaining
+ // bytes / 4; bounds the allocation on malformed counts
+ if (this.header.count > (byteArray.length - index) / 4L) {
+ throw new IOException("ObjectSpaceObjectStreamOfOSIDs count " +
this.header.count +
+ " exceeds remaining data " + (byteArray.length - index));
+ }
this.body = new CompactID[(int) this.header.count];
for (int i = 0; i < this.header.count; i++) {
CompactID compactID = new CompactID();
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/GUIDTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/GUIDTest.java
index 6e7f3b70d1..0f21f861db 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/GUIDTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/GUIDTest.java
@@ -16,6 +16,7 @@
*/
package org.apache.tika.parser.microsoft.onenote;
+import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import java.nio.charset.StandardCharsets;
@@ -26,6 +27,24 @@ import org.apache.tika.exception.TikaException;
public class GUIDTest {
+ @Test
+ public void testParsesValidGuidRoundTrip() throws Exception {
+ byte[] valid = "{638DE92F-a6d4-4BC1-9A36-4AFC2511A5B7}"
+ .getBytes(StandardCharsets.UTF_16LE);
+
+ assertEquals("{638DE92F-A6D4-4BC1-9A36-4AFC2511A5B7}",
+ GUID.fromCurlyBraceUTF16Bytes(valid).toString());
+ }
+
+ @Test
+ public void testRejectsNonAsciiUnicodeDigits() {
+ // fullwidth '6' is a Unicode digit that Character.digit(c, 16) would
accept
+ byte[] malformed = "{６38DE92F-A6D4-4BC1-9A36-4AFC2511A5B7}"
+ .getBytes(StandardCharsets.UTF_16LE);
+
+ assertThrows(TikaException.class, () ->
GUID.fromCurlyBraceUTF16Bytes(malformed));
+ }
+
@Test
public void testRejectsMalformedCurlyBraceGuid() {
byte[] malformed = "{638DE92F-A6D4-4BC1-9IJ36-4AFC2511A5B7}"
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
index 90e1ada59e..f064b50c4a 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
@@ -25,10 +25,12 @@ import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.IOException;
import java.io.InputStream;
+import java.io.StringWriter;
import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;
import java.nio.ByteBuffer;
import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.time.Instant;
@@ -49,6 +51,7 @@ import org.apache.tika.io.TikaInputStream;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.TikaCoreProperties;
import org.apache.tika.parser.ParseContext;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.MSOneStorePackage;
import org.apache.tika.sax.ToTextContentHandler;
import org.apache.tika.sax.XHTMLContentHandler;
@@ -57,7 +60,9 @@ public class OneNoteParserTest extends TikaTest {
//test recursive parser wrapper for image files
@Test
- public void testFuzzerRegressionInputsFailWithTikaExceptions() throws
Exception {
+ public void testFuzzerRegressionInputsFallBackToLegacyDump() throws
Exception {
+ // a structural failure falls back to the legacy string dump; the
failure is
+ // pinned in the parse-warning metadata
String[][] resources = {
{"testOneNote-fuzz1.one", "Missing dependent revision"},
{"testOneNote-fuzz2.one", "unified property count"},
@@ -68,15 +73,73 @@ public class OneNoteParserTest extends TikaTest {
assertNotNull(input, resource[0]);
try (InputStream stream = input;
TikaInputStream tis = TikaInputStream.get(stream)) {
- TikaException exception = assertThrows(TikaException.class,
- () -> new OneNoteParser().parse(tis, new
ToTextContentHandler(),
- new Metadata(), new ParseContext()),
resource[0]);
- assertTrue(exception.getMessage().contains(resource[1]),
- exception.getMessage());
+ Metadata metadata = new Metadata();
+ new OneNoteParser().parse(tis, new ToTextContentHandler(),
metadata,
+ new ParseContext());
+ assertTrue(Arrays.stream(
+
metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains(resource[1])
+ && warning.contains("falling back to legacy
text dump")),
+ () -> resource[0] + ": " +
Arrays.toString(metadata.getValues(
+
TikaCoreProperties.TIKA_META_EXCEPTION_WARNING)));
}
}
}
+ @Test
+ public void testNoContentFsshttpbWalkFallsBackToLegacyDump(@TempDir Path
tempDir)
+ throws Exception {
+ Path file = tempDir.resolve("empty-walk.one");
+ // trailing newline: the dump excludes the file's final byte from
scanning
+ Files.write(file,
+ "recoverable legacy dump
text\n".getBytes(StandardCharsets.US_ASCII));
+ Metadata metadata = new Metadata();
+ StringWriter writer = new StringWriter();
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(new
ToTextContentHandler(writer),
+ metadata, new ParseContext());
+ xhtml.startDocument();
+ try (OneNoteDirectFileResource resource =
+ new OneNoteDirectFileResource(file.toFile())) {
+ // fresh package = the walk completed without emitting anything
+ OneNoteParser.legacyFallbackIfNoContent(new MSOneStorePackage(),
metadata, xhtml,
+ resource);
+ assertTrue(writer.toString().contains("recoverable legacy dump
text"));
+
assertTrue(metadata.get(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING)
+ .contains("produced no content"));
+
+ // pkg == null means the exception path already dumped - no second
dump
+ StringWriter exceptionPath = new StringWriter();
+ XHTMLContentHandler xhtml2 = new XHTMLContentHandler(
+ new ToTextContentHandler(exceptionPath), metadata, new
ParseContext());
+ xhtml2.startDocument();
+ OneNoteParser.legacyFallbackIfNoContent(null, metadata, xhtml2,
resource);
+ xhtml2.endDocument();
+ assertTrue(exceptionPath.toString().isBlank());
+ }
+ xhtml.endDocument();
+ }
+
+ @Test
+ public void testTruncatedFileFallsBackToLegacyDump(@TempDir Path tempDir)
throws Exception {
+ byte[] full;
+ try (InputStream is = getClass()
+ .getResourceAsStream("/test-documents/testOneNote1.one")) {
+ full = is.readAllBytes();
+ }
+ // keep the 1024-byte header plus a sliver of content so the root file
node list
+ // is unreachable
+ Path truncated = tempDir.resolve("truncated.one");
+ Files.write(truncated, Arrays.copyOf(full, 2048));
+
+ Metadata metadata = new Metadata();
+ try (TikaInputStream tis = TikaInputStream.get(truncated)) {
+ new OneNoteParser().parse(tis, new ToTextContentHandler(),
metadata,
+ new ParseContext());
+ }
+
assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains("falling back to legacy
text dump")));
+ }
+
/**
* This is the sample document that is automatically created from onenote
2013.
*/
@@ -293,6 +356,16 @@ public class OneNoteParserTest extends TikaTest {
assertTrue(embeddedTypes.contains("INLINE"));
}
+ @Test
+ public void testOneNoteEmbeddedImageRecursiveMetadata() throws Exception {
+ List<Metadata> metadataList =
getRecursiveMetadata("testOneNoteEmbeddedImage.one");
+
+ assertEquals(2, metadataList.size());
+ Metadata embedded = metadataList.get(1);
+ assertEquals("INLINE",
embedded.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE));
+
assertNotNull(embedded.get(TikaCoreProperties.EMBEDDED_RELATIONSHIP_ID));
+ }
+
@Test
public void testPropertyValueBudgetIsSharedAcrossCopiesAndResetsPerList(
@TempDir Path tempDir) throws Exception {
@@ -338,6 +411,135 @@ public class OneNoteParserTest extends TikaTest {
.count());
}
+ @Test
+ public void testDeepFileNodeChainIsDepthCapped(@TempDir Path tempDir)
throws Exception {
+ // a long acyclic chain slips past the cycle guard - the depth cap
must stop it
+ // before the recursion can overflow the stack
+ FileNode root = new FileNode().setGosid(ExtendedGUID.nil());
+ FileNode current = root;
+ for (int i = 0; i < 600; i++) {
+ FileNode child = new FileNode().setGosid(ExtendedGUID.nil());
+ current.childFileNodeList.setFileNodeListHeader(new
FileNodeListHeader(0,
+ FileNodeListHeader.UNIT_MAGIC_CONSTANT, 0x10, 0));
+ current.childFileNodeList.children.add(child);
+ current = child;
+ }
+ Metadata metadata = new Metadata();
+ ParseContext parseContext = new ParseContext();
+ Path emptyFile = Files.createFile(tempDir.resolve("empty"));
+ try (OneNoteDirectFileResource dif = new
OneNoteDirectFileResource(emptyFile.toFile())) {
+ OneNoteTreeWalker walker = new OneNoteTreeWalker(new
OneNoteTreeWalkerOptions(),
+ new OneNoteDocument(), dif,
+ new XHTMLContentHandler(new ToTextContentHandler(),
metadata, parseContext),
+ metadata, parseContext, null);
+ walker.walkFileNode(root, null);
+ }
+ assertEquals(1, Arrays.stream(
+
metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .filter(warning -> warning.contains("exceeded depth limit"))
+ .count());
+ }
+
+ /**
+ * Writes one minimal 56-byte file-node-list fragment: header, a single
baseType-2 node
+ * whose child list is at {childStp, childCb}, a terminator, a nil
next-fragment
+ * reference and the footer.
+ */
+ private static void writeFileNodeListBlock(ByteBuffer buf, long childStp,
int childCb) {
+ // id=0x10, size=16 (node header + 8-byte stp + 4-byte cb),
stpFormat=0, cbFormat=0,
+ // baseType=2, reserved=1
+ int fileNodeHeader = 0x10 | (16 << 10) | (2 << 27) | (1 << 31);
+
buf.putLong(FileNodeListHeader.UNIT_MAGIC_CONSTANT).putInt(0x10).putInt(0)
+ .putInt(fileNodeHeader).putLong(childStp).putInt(childCb)
+ .putInt(0)
+ .putLong(-1).putInt(0)
+ .putLong(OneNotePtr.FOOTER_CONST);
+ }
+
+ @Test
+ public void testFileNodeListCycleFailsCleanly(@TempDir Path tempDir)
throws Exception {
+ // one fragment holding a baseType-2 node whose child list points back
at itself
+ ByteBuffer buf =
ByteBuffer.allocate(56).order(ByteOrder.LITTLE_ENDIAN);
+ writeFileNodeListBlock(buf, 0, 56);
+ Path cyclic = tempDir.resolve("cyclic");
+ Files.write(cyclic, buf.array());
+ try (OneNoteDirectFileResource dif = new
OneNoteDirectFileResource(cyclic.toFile())) {
+ OneNotePtr ptr = new OneNotePtr(new OneNoteDocument(), dif);
+ TikaException e = assertThrows(TikaException.class,
+ () -> ptr.deserializeFileNodeList(new FileNodeList(), new
FileNodePtr()));
+ assertTrue(e.getMessage().contains("cycle"), e.getMessage());
+ }
+ }
+
+ @Test
+ public void testFileNodeListNestingIsDepthCapped(@TempDir Path tempDir)
throws Exception {
+ // 120 lists, each holding one baseType-2 node pointing at the next
list - acyclic,
+ // so only the depth cap can stop the recursion
+ int lists = 120;
+ ByteBuffer buf = ByteBuffer.allocate(56 *
lists).order(ByteOrder.LITTLE_ENDIAN);
+ for (int i = 0; i < lists; i++) {
+ long childStp = (i + 1 < lists ? i + 1 : i) * 56L;
+ writeFileNodeListBlock(buf, childStp, 56);
+ }
+ Path deep = tempDir.resolve("deep");
+ Files.write(deep, buf.array());
+ try (OneNoteDirectFileResource dif = new
OneNoteDirectFileResource(deep.toFile())) {
+ OneNotePtr ptr = new OneNotePtr(new OneNoteDocument(), dif);
+ assertThrows(TikaMemoryLimitException.class,
+ () -> ptr.deserializeFileNodeList(new FileNodeList(), new
FileNodePtr()));
+ }
+ }
+
+ @Test
+ public void testFragmentChainCycleFailsCleanly(@TempDir Path tempDir)
throws Exception {
+ // an empty fragment whose next-fragment reference points back at
itself would
+ // previously loop forever
+ ByteBuffer buf = ByteBuffer.allocate(36).order(ByteOrder.LITTLE_ENDIAN)
+
.putLong(FileNodeListHeader.UNIT_MAGIC_CONSTANT).putInt(0x10).putInt(0)
+ .putLong(0).putInt(36)
+ .putLong(OneNotePtr.FOOTER_CONST);
+ Path cyclic = tempDir.resolve("cyclic-fragment");
+ Files.write(cyclic, buf.array());
+ try (OneNoteDirectFileResource dif = new
OneNoteDirectFileResource(cyclic.toFile())) {
+ OneNotePtr ptr = new OneNotePtr(new OneNoteDocument(), dif);
+ TikaException e = assertThrows(TikaException.class,
+ () -> ptr.deserializeFileNodeList(new FileNodeList(), new
FileNodePtr()));
+ assertTrue(e.getMessage().contains("fragment cycle"),
e.getMessage());
+ }
+ }
+
+ @Test
+ public void testLegacyEmbeddedExtractionHonorsShouldParseEmbedded() throws
Exception {
+ List<Metadata> offered = new ArrayList<>();
+ ParseContext context = new ParseContext();
+ context.set(EmbeddedDocumentExtractor.class, new
EmbeddedDocumentExtractor() {
+ @Override
+ public boolean shouldParseEmbedded(Metadata metadata, ParseContext
parseContext) {
+ offered.add(metadata);
+ return false;
+ }
+
+ @Override
+ public void parseEmbedded(TikaInputStream stream, ContentHandler
handler,
+ Metadata metadata, ParseContext context,
+ boolean outputHtml) {
+ throw new AssertionError("must not parse embedded when
shouldParseEmbedded" +
+ " returns false");
+ }
+ });
+ try (TikaInputStream tis =
+
getResourceAsStream("/test-documents/testOneNoteEmbeddedWordDoc.one")) {
+ new OneNoteParser().parse(tis, new ToTextContentHandler(), new
Metadata(), context);
+ }
+ assertTrue(offered.size() > 0);
+ // the legacy path offers every embedded object as an ATTACHMENT; the
relationship
+ // id is set only when the object carries a non-nil gosid
+ for (Metadata offeredMetadata : offered) {
+
assertEquals(TikaCoreProperties.EmbeddedResourceType.ATTACHMENT.toString(),
+
offeredMetadata.get(TikaCoreProperties.EMBEDDED_RESOURCE_TYPE));
+ }
+ }
+
/**
* Test a document pulled from Office 365 which stores the MS-ONESTORE
document using the MS-FSSHTTPB
* protocol.
@@ -351,7 +553,10 @@ public class OneNoteParserTest extends TikaTest {
// older page version snapshots are not reported
assertEquals(Arrays.asList("Chang Du", "Du Chang"),
Arrays.asList(metadata.getValues(TikaCoreProperties.CREATOR)));
- assertEquals(1, metadata.getValues(ONE_NOTE_PREFIX +
"mostRecentAuthors").length);
+ // both authors are referenced as AuthorMostRecent by current content;
one of them
+ // is first visited under another role, so its most-recent role must
still register
+ assertEquals(Arrays.asList("Chang Du", "Du Chang"),
+ Arrays.asList(metadata.getValues(ONE_NOTE_PREFIX +
"mostRecentAuthors")));
assertEquals(Instant.ofEpochSecond(1636621406),
Instant.ofEpochSecond(Long.parseLong(metadata.get(ONE_NOTE_PREFIX +
"creationTimestamp"))));
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreBlobTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreBlobTest.java
index 38b5fc9b1b..b7d7508f8b 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreBlobTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStoreBlobTest.java
@@ -18,6 +18,7 @@ package org.apache.tika.parser.microsoft.onenote.fsshttpb;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -213,8 +214,7 @@ public class MSOneStoreBlobTest {
assertNull(blob.getData());
blob.data = new BinaryItem();
blob.data.content.add((byte) 1);
- // Exercise serialization of a non-null BLOB; the round-trip assertion
is above.
- blob.serializeToByteList();
+ assertFalse(blob.serializeToByteList().isEmpty());
ObjectDataBLOB validBlob = new ObjectDataBLOB();
validBlob.data.content.add((byte) 1);
@@ -263,6 +263,19 @@ public class MSOneStoreBlobTest {
});
}
+ @Test
+ public void testDeclarationWithoutObjectDataFailsCleanly() {
+ ObjectGroupDataElementData mismatched = new
ObjectGroupDataElementData();
+ ObjectGroupObjectDeclare declaration = new ObjectGroupObjectDeclare();
+ declaration.objectPartitionID.setDecodedValue(1);
+
mismatched.objectGroupDeclarations.objectDeclarationList.add(declaration);
+
+ IOException e = assertThrows(IOException.class,
+ () -> RevisionStoreObjectGroup.createInstance(new ExGuid(9,
UUID.randomUUID()),
+ mismatched, false, Collections.emptyMap()));
+ assertTrue(e.getMessage().contains("Missing object data"));
+ }
+
@Test
public void testEncryptedObjectGroupAndMissingBlobReference() throws
Exception {
ObjectGroupDataElementData encryptedData = new
ObjectGroupDataElementData();
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
index 4de6e92c12..bbeadea253 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
@@ -285,6 +285,91 @@ public class MSOneStorePackageTest {
assertEquals("Иван Петров", metadata.get(OneNote.ORIGINAL_AUTHORS));
}
+ @Test
+ public void testDualRoleAuthorRecordedForBothRoles() throws Exception {
+ ExGuid authorId = id(710);
+ // the same author object referenced as both AuthorOriginal and
AuthorMostRecent
+ RevisionStoreObject root = object(id(711), propertySet(
+ new PropertySpec(PropertyType.ObjectID, 0x20001D78,
new NoData()),
+ new PropertySpec(PropertyType.ObjectID, 0x20001D79,
new NoData())),
+ Arrays.asList(authorId, authorId), Collections.emptyList());
+ RevisionStoreObject author = object(authorId, propertySet(
+ new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001D75, utf16Text("Single Author"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(root, author));
+ RevisionManifestRootDeclare rootDeclare = new
RevisionManifestRootDeclare();
+ rootDeclare.objectExGuid = root.objectID;
+ cell.rootDeclares.add(rootDeclare);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ Metadata metadata = new Metadata();
+ walk(pkg, metadata);
+
+ assertEquals("Single Author",
metadata.get(TikaCoreProperties.CREATOR));
+ assertEquals("Single Author", metadata.get(OneNote.ORIGINAL_AUTHORS));
+ assertEquals("Single Author",
metadata.get(OneNote.MOST_RECENT_AUTHORS));
+ }
+
+ @Test
+ public void testBlobOnlyRootWithDanglingContentRootWalksAllObjects()
throws Exception {
+ RevisionStoreObject blobRoot = object(id(620), propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ blobRoot.propertySet = null;
+ blobRoot.fileDataObject = fileData("blob root");
+ RevisionStoreObject textObject = object(id(621), propertySet(
+ new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("page body text"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(blobRoot, textObject));
+ RevisionManifestRootDeclare blobDeclare = new
RevisionManifestRootDeclare();
+ blobDeclare.objectExGuid = blobRoot.objectID;
+ cell.rootDeclares.add(blobDeclare);
+ RevisionManifestRootDeclare danglingContentRoot = new
RevisionManifestRootDeclare();
+ danglingContentRoot.objectExGuid = id(622);
+ cell.rootDeclares.add(danglingContentRoot);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ Metadata metadata = new Metadata();
+
+ // the blob root alone cannot reach the page body; the dangling
content root must
+ // trigger the walk-everything fallback so the body is not lost
+ assertTrue(walk(pkg, metadata).contains("page body text"));
+
assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(warning -> warning.contains("walking all objects")));
+ }
+
+ @Test
+ public void testHasEmittedContentTracksTextAndEmptyWalks() throws
Exception {
+ MSOneStorePackage withText = new MSOneStorePackage();
+ withText.cells.add(cellWithText(cell(70, 71), "some text"));
+ walk(withText);
+ assertTrue(withText.hasEmittedContent());
+
+ MSOneStorePackage empty = new MSOneStorePackage();
+ RevisionStoreCell danglingCell = new RevisionStoreCell();
+ RevisionManifestRootDeclare missingRoot = new
RevisionManifestRootDeclare();
+ missingRoot.objectExGuid = id(72);
+ danglingCell.rootDeclares.add(missingRoot);
+ empty.cells.add(danglingCell);
+ walk(empty);
+ assertFalse(empty.hasEmittedContent());
+ }
+
+ @Test
+ public void testSanitizeResourceNameKeepsBasenameOfPathShapedNames() {
+ assertEquals("pic 1.png",
+ MSOneStorePackage.sanitizeResourceName("D:\\images\\pic
1.png"));
+ assertEquals("pic.png",
MSOneStorePackage.sanitizeResourceName("C:pic.png"));
+ assertEquals("a.png",
MSOneStorePackage.sanitizeResourceName("/tmp/a.png"));
+ assertEquals("plain.png",
MSOneStorePackage.sanitizeResourceName("plain.png"));
+ assertEquals("", MSOneStorePackage.sanitizeResourceName(".."));
+ assertEquals("",
MSOneStorePackage.sanitizeResourceName("D:\\images\\.."));
+ }
+
@Test
public void testBlobOnlyRootDoesNotFallBackToOtherObjects() throws
Exception {
RevisionStoreObject blobRoot = object(id(610), propertySet(),
@@ -417,19 +502,17 @@ public class MSOneStorePackageTest {
new NoData())),
Collections.emptyList(), Collections.emptyList());
Method collectActions =
MSOneStorePackage.class.getDeclaredMethod("collectActions",
- PropertySet.class, List.class, int[].class, List.class,
int[].class, List.class);
+ PropertySet.class, List.class, int[].class, List.class,
int[].class, List.class,
+ int.class);
collectActions.setAccessible(true);
List<Object> actions = new ArrayList<>();
collectActions.invoke(new MSOneStorePackage(),
hugeArrayRoot.propertySet.objectSpaceObjectPropSet.body,
Collections.emptyList(), new int[]{0},
Collections.emptyList(), new int[]{0},
- actions);
+ actions, 0);
assertTrue(actions.isEmpty());
- Method collectActionsWithDepth =
MSOneStorePackage.class.getDeclaredMethod(
- "collectActions", PropertySet.class, List.class, int[].class,
List.class,
- int[].class, List.class, int.class);
- collectActionsWithDepth.setAccessible(true);
- collectActionsWithDepth.invoke(new MSOneStorePackage(),
+ actions.clear();
+ collectActions.invoke(new MSOneStorePackage(),
hugeArrayRoot.propertySet.objectSpaceObjectPropSet.body,
Collections.singletonList(id(601)), new int[]{0},
Collections.emptyList(),
new int[]{0}, actions, 1000);
@@ -454,7 +537,7 @@ public class MSOneStorePackageTest {
actions.clear();
collectActions.invoke(new MSOneStorePackage(), alignedSet,
Collections.nCopies(100001, id(601)), new int[]{0},
Collections.emptyList(),
- new int[]{0}, actions);
+ new int[]{0}, actions, 0);
assertEquals(100001, actions.size());
java.lang.reflect.Field childReference = actions.get(actions.size() -
1).getClass()
.getDeclaredField("childReference");
@@ -467,15 +550,26 @@ public class MSOneStorePackageTest {
OneNoteTreeWalkerOptions.class, Metadata.class,
XHTMLContentHandler.class,
int.class);
walkObject.setAccessible(true);
+ // an object that emits text when walked, so the depth-capped walk's
blank
+ // output proves the cap fired rather than the setup having nothing to
emit
+ RevisionStoreObject textRoot = object(id(605), propertySet(
+ new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("depth capped text"))),
+ Collections.emptyList(), Collections.emptyList());
Metadata metadata = new Metadata();
StringWriter writer = new StringWriter();
XHTMLContentHandler xhtml = new XHTMLContentHandler(
new ToTextContentHandler(writer), metadata, new
ParseContext());
xhtml.startDocument();
- walkObject.invoke(new MSOneStorePackage(), hugeArrayRoot,
+ walkObject.invoke(new MSOneStorePackage(), textRoot,
new java.util.HashMap<>(), new java.util.HashSet<>(), null,
new OneNoteTreeWalkerOptions(), metadata, xhtml, 1000);
+ assertTrue(writer.toString().isBlank());
+ walkObject.invoke(new MSOneStorePackage(), textRoot,
+ new java.util.HashMap<>(), new java.util.HashSet<>(), null,
+ new OneNoteTreeWalkerOptions(), metadata, xhtml, 0);
xhtml.endDocument();
+ assertTrue(writer.toString().contains("depth capped text"));
}
@Test
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/PropertyDeserializationBoundsTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/PropertyDeserializationBoundsTest.java
new file mode 100644
index 0000000000..73d5ecd395
--- /dev/null
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/PropertyDeserializationBoundsTest.java
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.tika.parser.microsoft.onenote.fsshttpb;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.List;
+
+import org.junit.jupiter.api.Test;
+
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtArrayOfPropertyValues;
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtFourBytesOfLengthFollowedByData;
+import org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.PropertySet;
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyID;
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.basic.PropertyType;
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectStreamHeader;
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectStreamOfContextIDs;
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectStreamOfOIDs;
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.streamobj.space.ObjectSpaceObjectStreamOfOSIDs;
+
+/**
+ * File-derived counts and nesting in the fsshttpb property deserializers must
be bounded -
+ * a malformed count would otherwise allocate up to gigabytes and a deeply
nested property
+ * set would overflow the stack, both Errors that escape the parser's
Exception fallback.
+ */
+public class PropertyDeserializationBoundsTest {
+
+ @Test
+ public void testFourBytesOfLengthRejectsLengthBeyondRemainingData() {
+ byte[] bytes = concat(int32(1000), new byte[2]);
+ IOException e = assertThrows(IOException.class,
+ () -> new PrtFourBytesOfLengthFollowedByData()
+ .doDeserializeFromByteArray(bytes, 0));
+ assertTrue(e.getMessage().contains("exceeds remaining data"),
e.getMessage());
+ }
+
+ @Test
+ public void testFourBytesOfLengthRejectsNegativeLength() {
+ byte[] bytes = concat(int32(0xF0000000), new byte[8]);
+ assertThrows(IOException.class,
+ () -> new PrtFourBytesOfLengthFollowedByData()
+ .doDeserializeFromByteArray(bytes, 0));
+ }
+
+ @Test
+ public void testFourBytesOfLengthAcceptsExactFit() throws IOException {
+ byte[] bytes = concat(int32(3), new byte[]{1, 2, 3});
+ PrtFourBytesOfLengthFollowedByData property = new
PrtFourBytesOfLengthFollowedByData();
+ assertEquals(7, property.doDeserializeFromByteArray(bytes, 0));
+ assertEquals(3, property.data.length);
+ }
+
+ @Test
+ public void testArrayOfPropertyValuesRejectsCountBeyondRemainingData()
throws IOException {
+ byte[] bytes = concat(int32(Integer.MAX_VALUE),
+ propertyId(PropertyType.PropertySet.getIntVal()));
+ IOException e = assertThrows(IOException.class,
+ () -> new
PrtArrayOfPropertyValues().doDeserializeFromByteArray(bytes, 0));
+ assertTrue(e.getMessage().contains("exceeds remaining data"),
e.getMessage());
+ }
+
+ @Test
+ public void testArrayOfPropertyValuesAcceptsSmallCount() throws
IOException {
+ // one element: an empty PropertySet (int16 count of 0)
+ byte[] bytes = concat(int32(1),
propertyId(PropertyType.PropertySet.getIntVal()),
+ new byte[]{0, 0});
+ PrtArrayOfPropertyValues array = new PrtArrayOfPropertyValues();
+ array.doDeserializeFromByteArray(bytes, 0);
+ assertEquals(1, array.data.length);
+ }
+
+ @Test
+ public void testStreamOfIdsRejectCountBeyondRemainingData() throws
IOException {
+ byte[] bytes = streamHeader(0xFFFFFF);
+ assertThrows(IOException.class,
+ () -> new
ObjectSpaceObjectStreamOfOIDs().doDeserializeFromByteArray(bytes, 0));
+ assertThrows(IOException.class,
+ () -> new
ObjectSpaceObjectStreamOfOSIDs().doDeserializeFromByteArray(bytes, 0));
+ assertThrows(IOException.class,
+ () -> new ObjectSpaceObjectStreamOfContextIDs()
+ .doDeserializeFromByteArray(bytes, 0));
+ }
+
+ @Test
+ public void testStreamOfOidsAcceptsSmallCount() throws IOException {
+ // one CompactID, 4 bytes
+ byte[] bytes = concat(streamHeader(1), new byte[4]);
+ ObjectSpaceObjectStreamOfOIDs stream = new
ObjectSpaceObjectStreamOfOIDs();
+ stream.doDeserializeFromByteArray(bytes, 0);
+ assertEquals(1, stream.body.length);
+ }
+
+ @Test
+ public void testPropertySetNestingIsDepthCapped() throws IOException {
+ byte[] bytes = nestedPropertySet(PropertySet.MAX_PROPERTY_NESTING +
50);
+ IOException e = assertThrows(IOException.class,
+ () -> new PropertySet().doDeserializeFromByteArray(bytes, 0));
+ assertTrue(e.getMessage().contains("nesting exceeds"), e.getMessage());
+ }
+
+ @Test
+ public void testPropertySetShallowNestingParses() throws IOException {
+ PropertySet propertySet = new PropertySet();
+ propertySet.doDeserializeFromByteArray(nestedPropertySet(3), 0);
+ assertEquals(1, propertySet.rgData.size());
+ }
+
+ @Test
+ public void testNestingThroughArrayOfPropertyValuesIsDepthCapped() throws
IOException {
+ // alternate PropertySet -> ArrayOfPropertyValues -> PropertySet -> ...
+ byte[] bytes = new byte[]{0, 0};
+ for (int i = 0; i < PropertySet.MAX_PROPERTY_NESTING + 50; i++) {
+ bytes = concat(new byte[]{1, 0},
+
propertyId(PropertyType.ArrayOfPropertyValues.getIntVal()), int32(1),
+ propertyId(PropertyType.PropertySet.getIntVal()), bytes);
+ }
+ byte[] finalBytes = bytes;
+ IOException e = assertThrows(IOException.class,
+ () -> new PropertySet().doDeserializeFromByteArray(finalBytes,
0));
+ assertTrue(e.getMessage().contains("nesting exceeds"), e.getMessage());
+ }
+
+ private static byte[] nestedPropertySet(int depth) throws IOException {
+ // innermost: empty PropertySet; each wrapper declares one
PropertySet-typed property
+ byte[] bytes = new byte[]{0, 0};
+ for (int i = 0; i < depth; i++) {
+ bytes = concat(new byte[]{1, 0},
+ propertyId(PropertyType.PropertySet.getIntVal()), bytes);
+ }
+ return bytes;
+ }
+
+ private static byte[] propertyId(int type) throws IOException {
+ PropertyID propertyID = new PropertyID();
+ propertyID.id = 1;
+ propertyID.type = type;
+ return toArray(propertyID.serializeToByteList());
+ }
+
+ private static byte[] streamHeader(int count) throws IOException {
+ ObjectSpaceObjectStreamHeader header = new
ObjectSpaceObjectStreamHeader();
+ header.count = count;
+ return toArray(header.serializeToByteList());
+ }
+
+ private static byte[] int32(int value) {
+ return new byte[]{(byte) value, (byte) (value >> 8), (byte) (value >>
16),
+ (byte) (value >> 24)};
+ }
+
+ private static byte[] toArray(List<Byte> bytes) {
+ byte[] result = new byte[bytes.size()];
+ for (int i = 0; i < result.length; i++) {
+ result[i] = bytes.get(i);
+ }
+ return result;
+ }
+
+ private static byte[] concat(byte[]... parts) {
+ ByteArrayOutputStream bos = new ByteArrayOutputStream();
+ for (byte[] part : parts) {
+ bos.writeBytes(part);
+ }
+ return bos.toByteArray();
+ }
+}
