This is an automated email from the ASF dual-hosted git repository.
tballison pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/tika.git
The following commit(s) were added to refs/heads/main by this push:
new 49e40e6dda [TIKA-4943] add tika table and list handling (#3277)
49e40e6dda is described below
commit 49e40e6ddaf44910111f6b8c44db1f768d39098b
Author: Henry Lindeman <[email protected]>
AuthorDate: Fri Oct 9 12:17:56 2026 -0700
[TIKA-4943] add tika table and list handling (#3277)
* add tika table and list handling
Generated-by: Glean Tau <various models>
Signed-off-by: Henry Lindeman <[email protected]>
* AI review fixes + jacoco
Generated-by: Glean Tau <various models>
Signed-off-by: Henry Lindeman <[email protected]>
* AI reviews
Generated-by: Glean Tau <various models>
Signed-off-by: Henry Lindeman <[email protected]>
* Bot nits
Generated-by: Glean Tau <various models>
Signed-off-by: Henry Lindeman <[email protected]>
* Copilot comments
Generated-by: Glean Tau <various models>
Signed-off-by: Henry Lindeman <[email protected]>
* TIKA-4943: lower the fsshttpb walk depth cap to 256
---------
Signed-off-by: Henry Lindeman <[email protected]>
Co-authored-by: tallison <[email protected]>
---
CHANGES.txt | 18 +-
.../tika/parser/microsoft/onenote/OneNoteJcid.java | 6 +
.../onenote/fsshttpb/MSOneStorePackage.java | 520 +++++++++++-
.../microsoft/onenote/OneNoteParserTest.java | 25 +
.../onenote/fsshttpb/MSOneStorePackageTest.java | 933 ++++++++++++++++++++-
.../test-documents/testOneNoteTablesLists.one | Bin 0 -> 86561 bytes
6 files changed, 1463 insertions(+), 39 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index a0e04e1857..48dc600091 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -31,10 +31,10 @@ Release 4.2.0 - unreleased
streaming parser; tika-parser-miscoffice-module and
tika-parser-cad-module no
longer depend on jackson-databind, and OSGi deployments of
tika-bundle-standard
need only jackson-core (TIKA-4956).
-
+
* Forked pipes servers get their config and auth token over stdin; nothing
is written
to the temp dir. Server-manager and PipesServer signatures change
accordingly (TIKA-4952).
-
+
* GeoGebraParser switched to stream parsing to drop jackson-databind as
a dependency (TIKA-4956).
@@ -49,7 +49,7 @@ Release 4.2.0 - unreleased
charset detection in OSGi, which had fallen back to BOM and declared
charsets
only. The bundle now declares its real Java requirement (JavaSE 17, not
1.8)
and drops 67 imports of packages it never referenced (TIKA-4955).
-
+
* PipesParser.start() and PipesForkParser.start() bring the forks up before
the
first parse and fail fast if one can't start (TIKA-4953).
@@ -69,12 +69,12 @@ Release 4.2.0 - unreleased
DefaultDetector no longer nests another dynamic one (TIKA-4942).
* XmlSecurityContractTest pins XMLReaderUtils' XXE and entity-expansion
defenses
- forbidden-apis now rejects building JAXP or Commons Secure XML parsers
anywhere but
+ forbidden-apis now rejects building JAXP or Commons Secure XML parsers
anywhere but
XMLReaderUtils (TIKA-4939).
-
+
* Retire Tika's entity expansion limit (default 20) in SAX, DOM and StAX
parsing in favor of standard Java configuration methods (TIKA-4940).
-
+
* Report the external DTDs and entities a document's XML referenced, which
Tika never
resolves: tk:xml-external-reference (up to 20 system ids) and
tk:xml-external-reference-count on that document's metadata, and
@@ -82,6 +82,8 @@ Release 4.2.0 - unreleased
Every resolver Tika installs answers with an empty stream, never null
(TIKA-4941).
* Deprecate XMLReaderUtils.getXMLInputFactory() and use SAX for XFA
(TIKA-4938).
+ * OneNote preserves table and list structure in XHTML for FSSHTTPB files
(TIKA-4943).
+
* OneNote extracts a section GUID and page, page-series, page-node, and
conflict-page
GUID metadata from classic and FSSHTTPB files; the GUID bags are
deduplicated and
sorted, not page-ordered. Tree-collected GUID metadata is capped at
100,000 values
@@ -90,10 +92,10 @@ Release 4.2.0 - unreleased
FSSHTTPB root resolution is bounded at 10,000 roots per cell; when the
bound
truncates the declared roots before a content root, the walk falls back
to every
object in the cell so page content is not silently lost (TIKA-4944).
-
+
* Raise the default maxFilesProcessedPerProcess from 10,000 to 100,000;
forked pipes JVMs restart a tenth as often (TIKA-4950).
-
+
* Quiet default pipes logging: per-parse timing lines (WORKER/CLIENT/
SERVER/RESOURCE_TIMING) move from INFO to TRACE, and per-fork-start,
per-connection and plugin lifecycle lines move to DEBUG (TIKA-4949).
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteJcid.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteJcid.java
index d4f87508e0..0cb6d47571 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteJcid.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/OneNoteJcid.java
@@ -22,6 +22,12 @@ public final class OneNoteJcid {
public static final int SECTION_NODE = 0x07;
public static final int PAGE_SERIES_NODE = 0x08;
public static final int PAGE_NODE = 0x0B;
+ public static final int OUTLINE_NODE = 0x0C;
+ public static final int OUTLINE_ELEMENT_NODE = 0x0D;
+ public static final int NUMBER_LIST_NODE = 0x12;
+ public static final int TABLE_NODE = 0x22;
+ public static final int TABLE_ROW_NODE = 0x23;
+ public static final int TABLE_CELL_NODE = 0x24;
public static final int PAGE_METADATA = 0x30;
public static final int CONFLICT_PAGE_METADATA = 0x38;
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
index a5cad9d413..0a02f61e93 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackage.java
@@ -26,6 +26,7 @@ import java.time.ZoneOffset;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
+import java.util.EnumMap;
import java.util.HashMap;
import java.util.HashSet;
import java.util.IdentityHashMap;
@@ -37,6 +38,7 @@ import java.util.function.Consumer;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.apache.commons.codec.digest.DigestUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.xml.sax.SAXException;
@@ -99,9 +101,11 @@ public class MSOneStorePackage {
Pattern.compile("\uFDDFHYPERLINK\\s+\"([^\"]+)\"([^\"]+)$");
private static final Logger LOG =
LoggerFactory.getLogger(MSOneStorePackage.class);
private static final String P = "p";
- private static final int MAX_OBJECT_WALK_DEPTH = 1000;
+ // 3 frames/level; 1000 overflowed a 1 MB stack. Real nesting is tens of
levels.
+ static final int MAX_OBJECT_WALK_DEPTH = 256;
private static final int MAX_REFERENCE_COUNT = 100000;
private static final int MAX_PARSE_WARNINGS = 100;
+ private static final int MIN_CANONICAL_LIST_FORMAT_LENGTH = 128;
/** Defensive bound on root objects resolved per cell; real cells declare
a handful. */
static final int MAX_CELL_ROOTS = 10_000;
@@ -153,10 +157,15 @@ public class MSOneStorePackage {
private boolean parseWarningsSuppressed;
private boolean storageMappingsIndexed;
private boolean contentEmitted;
+ private final Map<RevisionStoreObject, Map<AuthorRole,
EmbeddedResourceInfo>> depthSkippedObjects =
+ new LinkedHashMap<>();
// flattened property actions per object; objects can be re-flattened many
times during
// picture/resource-name resolution, which is quadratic without this cache
private final Map<RevisionStoreObject, List<PropertyAction>>
objectActionsCache =
new IdentityHashMap<>();
+ private final Map<RevisionStoreObject, NumberListInfo> numberListInfoCache
=
+ new IdentityHashMap<>();
+ private final Map<String, List<byte[]>> listFormatKeysByDigest = new
HashMap<>();
private final Map<CellID, StorageIndexCellMapping>
storageIndexCellMappingsById =
new HashMap<>();
private final Map<ExGuid, StorageIndexRevisionMapping>
storageIndexRevisionMappingsById =
@@ -236,6 +245,7 @@ public class MSOneStorePackage {
public void walkTree(OneNoteTreeWalkerOptions options, Metadata metadata,
XHTMLContentHandler xhtml, ParseContext parseContext)
throws SAXException, TikaException, IOException {
+ depthSkippedObjects.clear();
this.parseContext = parseContext;
this.parentMetadata = metadata;
this.embeddedDocumentExtractor =
@@ -265,15 +275,8 @@ public class MSOneStorePackage {
// no cell information available - walk the object groups in
revision order
Map<ExGuid, RevisionStoreObject> objectsById =
indexObjectsById(OtherFileNodeList);
Set<ExGuid> visited = new HashSet<>();
- for (RevisionStoreObjectGroup objectGroup : OtherFileNodeList)
{
- if (objectGroup == null || objectGroup.objects == null) {
- continue;
- }
- for (RevisionStoreObject object : objectGroup.objects) {
- walkObject(object, objectsById, visited,
AuthorRole.NONE, options, metadata,
- xhtml, 0);
- }
- }
+ walkObjectGroupRoots(OtherFileNodeList, objectsById, visited,
options, metadata,
+ xhtml);
}
if (!authors.isEmpty()) {
metadata.set(TikaCoreProperties.CREATOR,
sortedValues(authors));
@@ -285,6 +288,8 @@ public class MSOneStorePackage {
metadata.set(OneNote.ORIGINAL_AUTHORS,
sortedValues(originalAuthors));
}
} finally {
+ numberListInfoCache.clear();
+ listFormatKeysByDigest.clear();
guids.publish(metadata);
}
}
@@ -293,6 +298,10 @@ public class MSOneStorePackage {
return guids;
}
+ boolean hasCachedListStyles() {
+ return !numberListInfoCache.isEmpty() ||
!listFormatKeysByDigest.isEmpty();
+ }
+
void emitPage(RevisionStoreCell cell, OneNoteTreeWalkerOptions options,
Metadata metadata, XHTMLContentHandler xhtml)
throws SAXException, TikaException, IOException {
@@ -411,14 +420,14 @@ public class MSOneStorePackage {
OneNoteTreeWalkerOptions options, Metadata metadata,
XHTMLContentHandler xhtml)
throws SAXException, TikaException, IOException {
+ depthSkippedObjects.clear();
Set<ExGuid> visited = new HashSet<>();
// Only objects reachable from the root objects of the current
revision are part of
// the current content. The object groups may also contain older,
superseded versions
// of objects (under a different object ID); those are intentionally
not walked.
CellRoots roots = cellRoots(cell, objectsById,
this::recordParseWarning);
for (RevisionStoreObject rootObject : roots.declared) {
- walkObject(rootObject, objectsById, visited, AuthorRole.NONE,
options, metadata,
- xhtml, 0);
+ walkRootObject(rootObject, objectsById, visited, options,
metadata, xhtml);
}
if (roots.walkAll) {
if (cell.rootDeclares != null && cell.rootDeclares.isEmpty()) {
@@ -431,12 +440,8 @@ public class MSOneStorePackage {
} else {
recordParseWarning("OneNote cell root objects could not be
resolved; walking all objects");
}
- for (RevisionStoreObjectGroup objectGroup : cell.objectGroups) {
- for (RevisionStoreObject object : objectGroup.objects) {
- walkObject(object, objectsById, visited, AuthorRole.NONE,
options, metadata,
- xhtml, 0);
- }
- }
+ walkObjectGroupRoots(cell.objectGroups, objectsById, visited,
options, metadata,
+ xhtml);
}
}
@@ -580,6 +585,14 @@ public class MSOneStorePackage {
if (depth >= MAX_OBJECT_WALK_DEPTH) {
recordParseWarning("OneNote object traversal exceeded depth limit
" +
MAX_OBJECT_WALK_DEPTH);
+ if (object.objectID != null) {
+ Map<AuthorRole, EmbeddedResourceInfo> references =
depthSkippedObjects.computeIfAbsent(
+ object, key -> new EnumMap<>(AuthorRole.class));
+ AuthorRole role = authorRole == null ? AuthorRole.NONE :
authorRole;
+ if (!references.containsKey(role)) {
+ references.put(role, inheritedResourceInfo);
+ }
+ }
return;
}
if (object.objectID != null && !visited.add(object.objectID)) {
@@ -622,24 +635,473 @@ public class MSOneStorePackage {
}
int objectType = jcidIndex(object);
recordEntityGuid(object, objectType);
- // The title structure of a page (StructureElementChildNodes) appears
above the page
- // body on screen, but is declared after the body child nodes. Emit it
first so the
- // text comes out in visual order.
- for (PropertyAction action : actions) {
- if (action.oneNotePropertyEnum ==
OneNotePropertyEnum.StructureElementChildNodes) {
- processAction(action, objectsById, visited, authorRole,
options, metadata, xhtml,
- depth, resourceInfo);
+ ListStyle listStyle = listStyleForObject(object, actions, objectsById);
+ startObjectStructure(objectType, listStyle, xhtml);
+ try {
+ // Emit the title structure before page body nodes to preserve
visual order.
+ List<PropertyAction> structureActions = new ArrayList<>();
+ for (PropertyAction action : actions) {
+ if (action.oneNotePropertyEnum ==
OneNotePropertyEnum.StructureElementChildNodes) {
+ structureActions.add(action);
+ }
}
+ processStructureChildActions(structureActions, objectsById,
visited, authorRole,
+ options, metadata, xhtml, depth, resourceInfo);
+ List<PropertyAction> remainingActions = new ArrayList<>();
+ for (PropertyAction action : actions) {
+ if (hasPrimaryPicture &&
+ action.oneNotePropertyEnum ==
OneNotePropertyEnum.WebPictureContainer14) {
+ continue;
+ }
+ if (action.oneNotePropertyEnum !=
OneNotePropertyEnum.StructureElementChildNodes) {
+ remainingActions.add(action);
+ }
+ }
+ processStructureChildActions(remainingActions, objectsById,
visited, authorRole,
+ options, metadata, xhtml, depth, resourceInfo);
+ } finally {
+ endObjectStructure(objectType, listStyle, xhtml);
}
- for (PropertyAction action : actions) {
- if (hasPrimaryPicture &&
- action.oneNotePropertyEnum ==
OneNotePropertyEnum.WebPictureContainer14) {
+ }
+
+ private void startObjectStructure(int objectType, ListStyle listStyle,
+ XHTMLContentHandler xhtml) throws
SAXException {
+ switch (objectType) {
+ case OneNoteJcid.TABLE_NODE:
+ xhtml.startElement("table");
+ break;
+ case OneNoteJcid.TABLE_ROW_NODE:
+ xhtml.startElement("tr");
+ break;
+ case OneNoteJcid.TABLE_CELL_NODE:
+ xhtml.startElement("td");
+ break;
+ case OneNoteJcid.OUTLINE_ELEMENT_NODE:
+ if (listStyle != null) {
+ if (listStyle.restartValue == null) {
+ xhtml.startElement("li");
+ } else {
+ xhtml.startElement("li", "value",
listStyle.restartValue.toString());
+ }
+ }
+ break;
+ default:
+ break;
+ }
+ }
+
+ private void endObjectStructure(int objectType, ListStyle listStyle,
+ XHTMLContentHandler xhtml) throws
SAXException {
+ switch (objectType) {
+ case OneNoteJcid.TABLE_NODE:
+ xhtml.endElement("table");
+ break;
+ case OneNoteJcid.TABLE_ROW_NODE:
+ xhtml.endElement("tr");
+ break;
+ case OneNoteJcid.TABLE_CELL_NODE:
+ xhtml.endElement("td");
+ break;
+ case OneNoteJcid.OUTLINE_ELEMENT_NODE:
+ if (listStyle != null) {
+ xhtml.endElement("li");
+ }
+ break;
+ default:
+ break;
+ }
+ }
+
+ private void walkObjectGroupRoots(List<RevisionStoreObjectGroup>
objectGroups,
+ Map<ExGuid, RevisionStoreObject>
objectsById,
+ Set<ExGuid> visited,
OneNoteTreeWalkerOptions options,
+ Metadata metadata, XHTMLContentHandler
xhtml)
+ throws SAXException, TikaException, IOException {
+ Set<ExGuid> structureDescendantIds =
collectStructureDescendantIds(objectsById);
+ List<RevisionStoreObject> deferredObjects = new ArrayList<>();
+ for (RevisionStoreObjectGroup objectGroup : objectGroups) {
+ if (objectGroup == null || objectGroup.objects == null) {
continue;
}
- if (action.oneNotePropertyEnum !=
OneNotePropertyEnum.StructureElementChildNodes) {
+ for (RevisionStoreObject object : objectGroup.objects) {
+ if (object != null && object.objectID != null &&
+ structureDescendantIds.contains(object.objectID)) {
+ deferredObjects.add(object);
+ continue;
+ }
+ walkRootObject(object, objectsById, visited, options,
metadata, xhtml);
+ }
+ }
+ recoverDepthSkippedObjects(objectsById, visited, options, metadata,
xhtml);
+ // Residual owners belong to cycles; keep their storage order, not
type priority.
+ for (RevisionStoreObject object : deferredObjects) {
+ if (!visited.contains(object.objectID) &&
+ (jcidIndex(object) == OneNoteJcid.TABLE_NODE ||
+ isListContainer(object, objectsById))) {
+ walkRootObject(object, objectsById, visited, options,
metadata, xhtml);
+ }
+ }
+ recoverDepthSkippedObjects(objectsById, visited, options, metadata,
xhtml);
+ }
+
+ private void recoverDepthSkippedObjects(Map<ExGuid, RevisionStoreObject>
objectsById,
+ Set<ExGuid> visited,
OneNoteTreeWalkerOptions options,
+ Metadata metadata,
XHTMLContentHandler xhtml)
+ throws SAXException, TikaException, IOException {
+ // Recover only depth-skipped objects, not intentionally suppressed
renditions.
+ while (!depthSkippedObjects.isEmpty()) {
+ Map.Entry<RevisionStoreObject, Map<AuthorRole,
EmbeddedResourceInfo>> skipped =
+ depthSkippedObjects.entrySet().iterator().next();
+ depthSkippedObjects.remove(skipped.getKey());
+ for (Map.Entry<AuthorRole, EmbeddedResourceInfo> reference :
skipped.getValue().entrySet()) {
+ walkRootObject(skipped.getKey(), objectsById, visited,
reference.getKey(), options,
+ metadata, xhtml, reference.getValue());
+ }
+ }
+ }
+
+ private boolean isListContainer(RevisionStoreObject object,
+ Map<ExGuid, RevisionStoreObject>
objectsById) {
+ int type = jcidIndex(object);
+ if (type == OneNoteJcid.OUTLINE_NODE ||
+ type == OneNoteJcid.OUTLINE_ELEMENT_NODE) {
+ return true;
+ }
+ if (object == null || object.propertySet == null ||
+ object.propertySet.objectSpaceObjectPropSet == null) {
+ return false;
+ }
+ for (PropertyAction action : collectObjectActions(object)) {
+ if (action.isChildReference &&
+ jcidIndex(objectsById.get(action.childReference)) ==
+ OneNoteJcid.OUTLINE_ELEMENT_NODE) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ private Set<ExGuid> collectStructureDescendantIds(
+ Map<ExGuid, RevisionStoreObject> objectsById) {
+ Set<ExGuid> descendants = new HashSet<>();
+ Set<ExGuid> traversed = new HashSet<>();
+ List<RevisionStoreObject> pending = new ArrayList<>();
+ for (RevisionStoreObject object : objectsById.values()) {
+ if (jcidIndex(object) == OneNoteJcid.TABLE_NODE ||
+ isListContainer(object, objectsById)) {
+ traversed.add(object.objectID);
+ pending.add(object);
+ }
+ }
+ for (int index = 0; index < pending.size(); index++) {
+ RevisionStoreObject object = pending.get(index);
+ if (object.propertySet == null ||
+ object.propertySet.objectSpaceObjectPropSet == null) {
+ continue;
+ }
+ for (PropertyAction action : collectObjectActions(object)) {
+ if (!action.isChildReference || action.childReference == null)
{
+ continue;
+ }
+ ExGuid childId = action.childReference;
+ if (childId.equals(object.objectID)) {
+ continue;
+ }
+ descendants.add(childId);
+ RevisionStoreObject child = objectsById.get(childId);
+ if (child != null && traversed.add(childId)) {
+ pending.add(child);
+ }
+ }
+ }
+ return descendants;
+ }
+
+ private void walkRootObject(RevisionStoreObject object,
+ Map<ExGuid, RevisionStoreObject> objectsById,
Set<ExGuid> visited,
+ OneNoteTreeWalkerOptions options, Metadata
metadata,
+ XHTMLContentHandler xhtml)
+ throws SAXException, TikaException, IOException {
+ walkRootObject(object, objectsById, visited, AuthorRole.NONE, options,
metadata, xhtml, null);
+ }
+
+ private void walkRootObject(RevisionStoreObject object,
+ Map<ExGuid, RevisionStoreObject> objectsById,
Set<ExGuid> visited,
+ AuthorRole authorRole,
OneNoteTreeWalkerOptions options, Metadata metadata,
+ XHTMLContentHandler xhtml,
EmbeddedResourceInfo resourceInfo)
+ throws SAXException, TikaException, IOException {
+ if (!canWalkObject(object, visited, 0)) {
+ walkObject(object, objectsById, visited, authorRole, options,
metadata, xhtml, 0, resourceInfo);
+ return;
+ }
+ List<PropertyAction> actions = object.propertySet != null
+ && object.propertySet.objectSpaceObjectPropSet != null ?
+ collectObjectActions(object) : Collections.emptyList();
+ ListStyle listStyle = listStyleForObject(object, actions, objectsById);
+ if (listStyle != null) {
+ startList(listStyle, xhtml);
+ }
+ try {
+ walkObject(object, objectsById, visited, authorRole, options,
metadata, xhtml,
+ 0, resourceInfo);
+ } finally {
+ if (listStyle != null) {
+ xhtml.endElement(listStyle.elementName);
+ }
+ }
+ }
+
+ private void processStructureChildActions(List<PropertyAction> actions,
+ Map<ExGuid, RevisionStoreObject>
objectsById,
+ Set<ExGuid> visited, AuthorRole
authorRole,
+ OneNoteTreeWalkerOptions
options, Metadata metadata,
+ XHTMLContentHandler xhtml, int
depth,
+ EmbeddedResourceInfo
resourceInfo)
+ throws SAXException, TikaException, IOException {
+ int index = 0;
+ while (index < actions.size()) {
+ PropertyAction action = actions.get(index);
+ ListStyle style = isSkippedChild(action, objectsById, visited,
depth) ? null :
+ listStyleForChild(action, objectsById);
+ if (style == null) {
processAction(action, objectsById, visited, authorRole,
options, metadata, xhtml,
depth, resourceInfo);
+ index++;
+ continue;
}
+ startList(style, xhtml);
+ int next = index;
+ try {
+ while (next < actions.size()) {
+ PropertyAction sibling = actions.get(next);
+ if (isSkippedChild(sibling, objectsById, visited, depth)) {
+ processAction(sibling, objectsById, visited,
authorRole, options, metadata,
+ xhtml, depth, resourceInfo);
+ next++;
+ continue;
+ }
+ ListStyle siblingStyle = listStyleForChild(sibling,
objectsById);
+ if (siblingStyle == null || !style.matches(siblingStyle)) {
+ break;
+ }
+ processAction(sibling, objectsById, visited, authorRole,
options, metadata,
+ xhtml, depth, resourceInfo);
+ next++;
+ }
+ } finally {
+ xhtml.endElement(style.elementName);
+ }
+ index = next;
+ }
+ }
+
+ private boolean canWalkObject(RevisionStoreObject object, Set<ExGuid>
visited, int depth) {
+ return object != null && depth < MAX_OBJECT_WALK_DEPTH &&
+ (object.objectID == null ||
!visited.contains(object.objectID));
+ }
+
+ private boolean isSkippedChild(PropertyAction action,
+ Map<ExGuid, RevisionStoreObject>
objectsById,
+ Set<ExGuid> visited, int depth) {
+ return action.isChildReference &&
+ !canWalkObject(objectsById.get(action.childReference),
visited, depth + 1);
+ }
+
+ private ListStyle listStyleForChild(PropertyAction action,
+ Map<ExGuid, RevisionStoreObject>
objectsById) {
+ if (!action.isChildReference || action.childReference == null) {
+ return null;
+ }
+ RevisionStoreObject child = objectsById.get(action.childReference);
+ if (child == null || child.propertySet == null
+ || child.propertySet.objectSpaceObjectPropSet == null
+ || jcidIndex(child) != OneNoteJcid.OUTLINE_ELEMENT_NODE) {
+ return null;
+ }
+ return listStyleForObject(child, collectObjectActions(child),
objectsById);
+ }
+
+ private ListStyle listStyleForObject(RevisionStoreObject object,
List<PropertyAction> actions,
+ Map<ExGuid, RevisionStoreObject>
objectsById) {
+ if (object == null || jcidIndex(object) !=
OneNoteJcid.OUTLINE_ELEMENT_NODE) {
+ return null;
+ }
+ boolean isListItem = false;
+ NumberListInfo numberListInfo = null;
+ NumberListInfo malformedNumberListInfo = null;
+ for (PropertyAction action : actions) {
+ if (action.oneNotePropertyEnum == OneNotePropertyEnum.ListNodes) {
+ isListItem = true;
+ RevisionStoreObject listNode = action.childReference == null ?
null :
+ objectsById.get(action.childReference);
+ NumberListInfo candidate = numberListInfo(listNode);
+ if (candidate != null && candidate.formatValid) {
+ numberListInfo = candidate;
+ } else if (candidate != null) {
+ malformedNumberListInfo = candidate;
+ }
+ }
+ }
+ if (!isListItem) {
+ return null;
+ }
+ if (numberListInfo == null) {
+ numberListInfo = malformedNumberListInfo;
+ }
+ boolean numbered = numberListInfo != null && numberListInfo.numbered;
+ ListStyle style = new ListStyle(numbered ? "ol" : "ul",
+ numberListInfo == null ? null : numberListInfo.htmlType,
+ numberListInfo == null ? null : numberListInfo.format,
+ numbered ? numberListInfo.restartValue : null);
+ return style;
+ }
+
+ private NumberListInfo numberListInfo(RevisionStoreObject listNode) {
+ if (listNode == null || listNode.propertySet == null
+ || listNode.propertySet.objectSpaceObjectPropSet == null
+ || jcidIndex(listNode) != OneNoteJcid.NUMBER_LIST_NODE) {
+ return null;
+ }
+ if (numberListInfoCache.containsKey(listNode)) {
+ return numberListInfoCache.get(listNode);
+ }
+ byte[] format = null;
+ Long restartValue = null;
+ for (PropertyAction action : collectObjectActions(listNode)) {
+ if (action.oneNotePropertyEnum ==
OneNotePropertyEnum.NumberListFormat
+ && action.property instanceof
PrtFourBytesOfLengthFollowedByData) {
+ format = ((PrtFourBytesOfLengthFollowedByData)
action.property).data;
+ } else if (action.oneNotePropertyEnum ==
OneNotePropertyEnum.ListRestart
+ && action.property instanceof FourBytesOfData) {
+ restartValue = unsignedInt32(((FourBytesOfData)
action.property).data);
+ }
+ }
+ ParsedListFormat parsedFormat = parseListFormat(format);
+ NumberListInfo info = new NumberListInfo(canonicalListFormat(format),
parsedFormat != null,
+ parsedFormat != null && parsedFormat.numbered,
+ parsedFormat == null ? null : parsedFormat.htmlType,
+ parsedFormat != null && parsedFormat.numbered ? restartValue :
null);
+ numberListInfoCache.put(listNode, info);
+ return info;
+ }
+
+ private static ParsedListFormat parseListFormat(byte[] format) {
+ if (format == null || format.length < 2 || (format.length & 1) != 0) {
+ return null;
+ }
+ int characterCount = unsignedInt16(format, 0);
+ if (characterCount != format.length / 2 - 1) {
+ return null;
+ }
+ for (int i = 2; i + 1 < format.length; i += 2) {
+ if (unsignedInt16(format, i) == 0xfffd) {
+ if (i + 3 >= format.length) {
+ return null;
+ }
+ return new ParsedListFormat(true,
+ htmlListType(unsignedInt16(format, i + 2)));
+ }
+ }
+ return new ParsedListFormat(false, null);
+ }
+
+ private static int unsignedInt16(byte[] data, int index) {
+ return (data[index] & 0xff) | ((data[index + 1] & 0xff) << 8);
+ }
+
+ private static Long unsignedInt32(byte[] data) {
+ if (data == null || data.length != 4) {
+ return null;
+ }
+ return (data[0] & 0xffL) | ((data[1] & 0xffL) << 8) |
+ ((data[2] & 0xffL) << 16) | ((data[3] & 0xffL) << 24);
+ }
+
+ private static String htmlListType(int formatCode) {
+ switch (formatCode) {
+ case 0x00:
+ return "1";
+ case 0x01:
+ return "I";
+ case 0x02:
+ return "i";
+ case 0x03:
+ return "A";
+ case 0x04:
+ return "a";
+ default:
+ return null;
+ }
+ }
+
+ private void startList(ListStyle style, XHTMLContentHandler xhtml) throws
SAXException {
+ if (style.htmlType == null) {
+ xhtml.startElement(style.elementName);
+ } else {
+ xhtml.startElement(style.elementName, "type", style.htmlType);
+ }
+ }
+
+ private static final class ParsedListFormat {
+ private final boolean numbered;
+ private final String htmlType;
+
+ private ParsedListFormat(boolean numbered, String htmlType) {
+ this.numbered = numbered;
+ this.htmlType = htmlType;
+ }
+ }
+
+ private static final class NumberListInfo {
+ private final byte[] format;
+ private final boolean formatValid;
+ private final boolean numbered;
+ private final String htmlType;
+ private final Long restartValue;
+
+ private NumberListInfo(byte[] format, boolean formatValid, boolean
numbered,
+ String htmlType, Long restartValue) {
+ this.format = format;
+ this.formatValid = formatValid;
+ this.numbered = numbered;
+ this.htmlType = htmlType;
+ this.restartValue = restartValue;
+ }
+ }
+
+ private byte[] canonicalListFormat(byte[] format) {
+ if (format == null || format.length <
MIN_CANONICAL_LIST_FORMAT_LENGTH) {
+ return format;
+ }
+ List<byte[]> candidates = listFormatKeysByDigest.computeIfAbsent(
+ DigestUtils.sha256Hex(format), key -> new ArrayList<>());
+ for (byte[] candidate : candidates) {
+ // Keep grouping exact even if digests collide.
+ if (Arrays.equals(candidate, format)) {
+ return candidate;
+ }
+ }
+ candidates.add(format);
+ return format;
+ }
+
+ private static final class ListStyle {
+ private final String elementName;
+ private final String htmlType;
+ private final byte[] format;
+ private final Long restartValue;
+
+ private ListStyle(String elementName, String htmlType, byte[] format,
Long restartValue) {
+ this.elementName = elementName;
+ this.htmlType = htmlType;
+ this.format = format;
+ this.restartValue = restartValue;
+ }
+
+ private boolean matches(ListStyle other) {
+ return elementName.equals(other.elementName) &&
+ (htmlType == null ? other.htmlType == null :
htmlType.equals(other.htmlType)) &&
+ Arrays.equals(format, other.format);
}
}
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
index f1dceec2cc..561f7664b9 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/OneNoteParserTest.java
@@ -432,6 +432,31 @@ public class OneNoteParserTest extends TikaTest {
assertContains("Correlation between Outlook and OneNote", txt);
}
+ @Test
+ public void testFsshttpbTablesAndListsRetainXhtmlStructure() throws
Exception {
+ String xml = getXML("testOneNoteTablesLists.one", new
OneNoteParser()).xml;
+ assertEquals(1, StringUtils.countMatches(xml, "<table>"), xml);
+ assertEquals(2, StringUtils.countMatches(xml, "<tr>"), xml);
+ assertEquals(8, StringUtils.countMatches(xml, "<td"), xml);
+ assertTrue(xml.matches("(?s).*<table>.*<td><p>Table cell 1a</p>.*" +
+ "<td><p>Table cell 4b</p>.*</table>.*"), xml);
+ assertEquals(3, StringUtils.countMatches(xml, "<ul>"), xml);
+ assertTrue(xml.matches("(?s).*<li><p>Bullet 2</p>\\s*<ul>\\s*<li>" +
+ "<p>Indented bullet</p>\\s*<ul>\\s*<li><p>Indented
square</p>.*"), xml);
+ assertEquals(2, StringUtils.countMatches(xml, "<ol type=\"1\">"), xml);
+ assertTrue(xml.matches("(?s).*<li><p>Number 2</p>\\s*<ol
type=\"a\">\\s*<li>" +
+ "<p>Letter a</p>\\s*<ol type=\"i\">\\s*<li><p>Numeral
I</p>.*"), xml);
+ assertTrue(xml.matches("(?s).*<ol type=\"1\">\\s*<li><p>Outdented
number</p>.*"), xml);
+ assertEquals(9, StringUtils.countMatches(xml, "<li>"), xml);
+ }
+
+ @Test
+ public void testOneNote2007ListStructure() throws Exception {
+ String xml = getXML("testOneNote2007OrEarlier.one", new
OneNoteParser()).xml;
+ assertEquals(2, StringUtils.countMatches(xml, "<ul>"), xml);
+ assertEquals(6, StringUtils.countMatches(xml, "<li>"), xml);
+ }
+
@Test
public void testOneNoteEmbeddedWordDoc() throws Exception {
List<Metadata> metadataList =
getRecursiveMetadata("testOneNoteEmbeddedWordDoc.one");
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
index 9ff8a0bc14..b8095abd45 100644
---
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
+++
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/java/org/apache/tika/parser/microsoft/onenote/fsshttpb/MSOneStorePackageTest.java
@@ -38,9 +38,12 @@ import java.util.Set;
import java.util.UUID;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.CsvSource;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.DefaultHandler;
+import org.apache.tika.extractor.EmbeddedDocumentExtractor;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.metadata.OneNote;
import org.apache.tika.metadata.TikaCoreProperties;
@@ -49,6 +52,7 @@ import
org.apache.tika.parser.microsoft.onenote.OneNoteGuidCollector;
import org.apache.tika.parser.microsoft.onenote.OneNoteJcid;
import org.apache.tika.parser.microsoft.onenote.OneNoteTreeWalkerOptions;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.ArrayNumber;
+import
org.apache.tika.parser.microsoft.onenote.fsshttpb.property.FourBytesOfData;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.IProperty;
import org.apache.tika.parser.microsoft.onenote.fsshttpb.property.NoData;
import
org.apache.tika.parser.microsoft.onenote.fsshttpb.property.PrtArrayOfPropertyValues;
@@ -79,6 +83,770 @@ import org.apache.tika.sax.XHTMLContentHandler;
public class MSOneStorePackageTest {
+ @Test
+ public void testTableAndListObjectsRetainXhtmlStructure() throws Exception
{
+ ExGuid tableId = id(1100);
+ ExGuid rowId = id(1101);
+ ExGuid cellId = id(1102);
+ ExGuid listId = id(1103);
+ RevisionStoreObject tableCell = object(cellId,
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("table text"))),
Collections.emptyList(),
+ Collections.emptyList());
+ RevisionStoreObject row = object(rowId,
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(1))), Collections.singletonList(cellId),
Collections.emptyList());
+ RevisionStoreObject table = object(tableId,
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(1))), Collections.singletonList(rowId),
Collections.emptyList());
+ ExGuid bulletListId = id(1104);
+ RevisionStoreObject bulletList = numberListNode(bulletListId,
+ new byte[] {0x01, 0x00, 0x22, 0x20}, null);
+ RevisionStoreObject listItem = listItem(listId, bulletListId, "list
text");
+ setJcid(table, OneNoteJcid.TABLE_NODE);
+ setJcid(row, OneNoteJcid.TABLE_ROW_NODE);
+ setJcid(tableCell, OneNoteJcid.TABLE_CELL_NODE);
+
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(table, row, tableCell, listItem,
bulletList));
+ cell.rootDeclares.add(rootDeclare(tableId));
+ cell.rootDeclares.add(rootDeclare(listId));
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ String xml = walkXml(pkg);
+ assertTrue(xml.contains("<table>"));
+ assertTrue(xml.contains("<tr>"));
+ assertTrue(xml.contains("<td>"));
+ assertTrue(xml.contains("table text"));
+ assertTrue(xml.contains("<ul>"));
+ assertTrue(xml.contains("<li>"));
+ assertTrue(xml.contains("list text"));
+ }
+
+ @Test
+ public void testNumberedListItemsShareOrderedListMarkup() throws Exception
{
+ ExGuid outlineId = id(1150);
+ ExGuid firstItemId = id(1151);
+ ExGuid secondItemId = id(1152);
+ ExGuid firstNumberListId = id(1153);
+ ExGuid secondNumberListId = id(1154);
+ byte[] numberFormat = new byte[] {0x02, 0x00, (byte) 0xfd, (byte)
0xff, 0x01, 0x00};
+
+ RevisionStoreObject firstNumberList =
numberListNode(firstNumberListId, numberFormat, null);
+ RevisionStoreObject secondNumberList =
numberListNode(secondNumberListId, numberFormat, 5);
+ RevisionStoreObject firstItem = listItem(firstItemId,
firstNumberListId,
+ "first numbered item");
+ RevisionStoreObject secondItem = listItem(secondItemId,
secondNumberListId,
+ "second numbered item");
+ RevisionStoreObject outline = object(outlineId,
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(2))), Arrays.asList(firstItemId,
secondItemId),
+ Collections.emptyList());
+ setJcid(outline, OneNoteJcid.OUTLINE_NODE);
+
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(outline, firstItem, secondItem,
firstNumberList,
+ secondNumberList));
+ cell.rootDeclares.add(rootDeclare(outlineId));
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ String xml = walkXml(pkg);
+ assertEquals(1, countOccurrences(xml, "<ol type=\"I\">"), xml);
+ assertEquals(1, countOccurrences(xml, "<li value=\"5\">"), xml);
+ assertEquals(2, countOccurrences(xml, "<li"));
+ assertTrue(xml.contains("first numbered item"));
+ assertTrue(xml.contains("second numbered item"));
+ assertFalse(xml.contains("<ul>"));
+ }
+
+ @Test
+ public void testListFallbackPreservesGroupingRegardlessOfRootOrder()
throws Exception {
+ for (boolean useCellFallback : new boolean[] {false, true}) {
+ for (boolean parentFirst : new boolean[] {false, true}) {
+ for (boolean typedParent : new boolean[] {false, true}) {
+ RevisionStoreObject numberList = numberListNode(id(1703),
+ numberListFormat(0), null);
+ RevisionStoreObject first = listItem(id(1701),
numberList.objectID, "first item");
+ RevisionStoreObject second = listItem(id(1702),
numberList.objectID, "second item");
+ RevisionStoreObject outline = object(id(1700),
+ propertySet(new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C20,
+ arrayNumber(2))),
Arrays.asList(first.objectID, second.objectID),
+ Collections.emptyList());
+ if (typedParent) {
+ setJcid(outline, OneNoteJcid.OUTLINE_NODE);
+ }
+ RevisionStoreObjectGroup objects = parentFirst ?
+ group(outline, first, second, numberList) :
+ group(first, second, numberList, outline);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ if (useCellFallback) {
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(objects);
+ pkg.cells.add(cell);
+ } else {
+ pkg.OtherFileNodeList.add(objects);
+ }
+ String xml = walkXml(pkg);
+ assertEquals(1, countOccurrences(xml, "<ol"), xml);
+ assertEquals(2, countOccurrences(xml, "<li"), xml);
+ assertTrue(xml.matches("(?s).*<ol type=\"1\">.*first
item.*second item.*</ol>.*"), xml);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testVisitedListReferencesDoNotCreateOrSplitContainers() throws
Exception {
+ RevisionStoreObject roman = numberListNode(id(1724),
numberListFormat(1), null);
+ RevisionStoreObject decimal = numberListNode(id(1725),
numberListFormat(0), null);
+ RevisionStoreObject first = listItem(id(1721), roman.objectID,
"visited item");
+ RevisionStoreObject second = listItem(id(1722), decimal.objectID,
"second item");
+ RevisionStoreObject third = listItem(id(1723), decimal.objectID,
"third item");
+ RevisionStoreObject outline = object(id(1720),
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(5))), Arrays.asList(first.objectID,
second.objectID,
+ first.objectID, second.objectID, third.objectID),
Collections.emptyList());
+ setJcid(outline, OneNoteJcid.OUTLINE_NODE);
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(outline, first, second, third, roman,
decimal));
+ cell.rootDeclares.add(rootDeclare(first.objectID));
+ cell.rootDeclares.add(rootDeclare(outline.objectID));
+ cell.rootDeclares.add(rootDeclare(second.objectID));
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ String xml = walkXml(pkg);
+ assertEquals(2, countOccurrences(xml, "<ol"), xml);
+ assertEquals(3, countOccurrences(xml, "<li"), xml);
+ assertEquals(1, countOccurrences(xml, "visited item"), xml);
+ assertTrue(xml.matches("(?s).*<ol type=\"1\">.*second item.*third
item.*</ol>.*"), xml);
+ }
+
+ @Test
+ public void testRepeatedLargeListFormatsPreserveExactGrouping() throws
Exception {
+ byte[] format = new byte[1024];
+ byte[] equalFormat = format.clone();
+ byte[] anotherEqualFormat = format.clone();
+ byte[] differentFormat = format.clone();
+ differentFormat[differentFormat.length - 1] = 1;
+
+ RevisionStoreObject[] nodes = {
+ numberListNode(id(1750), format, null),
+ numberListNode(id(1751), equalFormat, null),
+ numberListNode(id(1752), anotherEqualFormat, null),
+ numberListNode(id(1753), differentFormat, null)
+ };
+ List<RevisionStoreObject> objects = new
ArrayList<>(Arrays.asList(nodes));
+ List<ExGuid> itemIds = new ArrayList<>();
+ for (int i = 0; i <= 2000; i++) {
+ ExGuid itemId = id(1800 + i);
+ RevisionStoreObject node = nodes[i == 2000 ? 3 : i % 3];
+ objects.add(listItem(itemId, node.objectID, "large item " + i));
+ itemIds.add(itemId);
+ }
+ RevisionStoreObject outline = object(id(1740),
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(itemIds.size()))), itemIds,
Collections.emptyList());
+ setJcid(outline, OneNoteJcid.OUTLINE_NODE);
+ objects.add(outline);
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(objects.toArray(new
RevisionStoreObject[0])));
+ cell.rootDeclares.add(rootDeclare(outline.objectID));
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ String xml = walkXml(pkg);
+ assertEquals(2, countOccurrences(xml, "<ul"), xml);
+ assertEquals(2001, countOccurrences(xml, "<li"), xml);
+ assertTrue(xml.contains("large item 2000"), xml);
+ assertTrue(xml.matches("(?s).*<ul>.*large item 0.*large item
1999.*</ul>\\s*" +
+ "<ul>\\s*<li><p>large item 2000</p>.*</ul>.*"), xml);
+ }
+
+ @ParameterizedTest(name = "cellFallback={0}, failWalk={1}")
+ @CsvSource({"false, false", "false, true", "true, false", "true, true"})
+ public void testListCachesAreReleasedAfterWalk(boolean useCellFallback,
boolean failWalk)
+ throws Exception {
+ RevisionStoreObject node = numberListNode(id(1740), new byte[1024],
null);
+ RevisionStoreObject item = listItem(id(1741), node.objectID, "cached
item");
+ RevisionStoreObject outline = object(id(1742),
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(1))),
Collections.singletonList(item.objectID), Collections.emptyList());
+ setJcid(outline, OneNoteJcid.OUTLINE_NODE);
+ MSOneStorePackage pkg = fallbackPackage(group(outline, item, node),
useCellFallback);
+ RevisionStoreObject next = object(id(1743),
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("next cell"))),
Collections.emptyList(), Collections.emptyList());
+ if (useCellFallback) {
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(next));
+ pkg.cells.add(cell);
+ } else {
+ pkg.OtherFileNodeList.add(group(next));
+ }
+ assertFalse(pkg.hasCachedListStyles());
+ List<String> observedText = new ArrayList<>();
+ DefaultHandler handler = new DefaultHandler() {
+ @Override
+ public void characters(char[] ch, int start, int length) throws
SAXException {
+ String value = new String(ch, start, length);
+ if ("cached item".equals(value) || "next cell".equals(value)) {
+ assertTrue(pkg.hasCachedListStyles());
+ observedText.add(value);
+ }
+ if (failWalk && "next cell".equals(value)) {
+ throw new SAXException("stop walk");
+ }
+ }
+ };
+ Metadata metadata = new Metadata();
+ ParseContext context = new ParseContext();
+ XHTMLContentHandler xhtml = new XHTMLContentHandler(handler, metadata,
context);
+ xhtml.startDocument();
+ if (failWalk) {
+ SAXException exception = assertThrows(SAXException.class,
+ () -> pkg.walkTree(new OneNoteTreeWalkerOptions(),
metadata, xhtml, context));
+ assertEquals("stop walk", exception.getMessage());
+ } else {
+ pkg.walkTree(new OneNoteTreeWalkerOptions(), metadata, xhtml,
context);
+ xhtml.endDocument();
+ }
+ assertEquals(Arrays.asList("cached item", "next cell"), observedText);
+ assertFalse(pkg.hasCachedListStyles());
+ String xml = walkXml(pkg);
+ assertEquals(1, countOccurrences(xml, "cached item"), xml);
+ assertEquals(1, countOccurrences(xml, "next cell"), xml);
+ assertFalse(pkg.hasCachedListStyles());
+ }
+
+ @Test
+ public void testValidListMetadataWinsInEitherReferenceOrder() throws
Exception {
+ for (boolean validFirst : new boolean[] {false, true}) {
+ RevisionStoreObject valid = numberListNode(id(1731),
numberListFormat(1), 5);
+ RevisionStoreObject malformed = numberListNode(id(1732),
+ new byte[] {3, 0, (byte) 0xfd, (byte) 0xff, 1, 0}, 7);
+ RevisionStoreObject item = object(id(1730),
+ propertySet(new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C26,
+ arrayNumber(2)),
+ new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("multi-node item"))),
+ validFirst ? Arrays.asList(valid.objectID,
malformed.objectID) :
+ Arrays.asList(malformed.objectID, valid.objectID),
Collections.emptyList());
+ setJcid(item, OneNoteJcid.OUTLINE_ELEMENT_NODE);
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(item, valid, malformed));
+ cell.rootDeclares.add(rootDeclare(item.objectID));
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ String xml = walkXml(pkg);
+ assertTrue(xml.contains("<ol type=\"I\">"), xml);
+ assertTrue(xml.contains("<li value=\"5\">"), xml);
+ assertTrue(xml.contains("multi-node item"), xml);
+ assertFalse(xml.contains("<ul"), xml);
+ }
+ }
+
+ @Test
+ public void testNumberListLengthPrefixIsNotNumberingMarker() throws
Exception {
+ ExGuid listId = id(1160);
+ ExGuid numberListId = id(1161);
+ byte[] numberFormat = new byte[(0xfffd + 1) * 2];
+ numberFormat[0] = (byte) 0xfd;
+ numberFormat[1] = (byte) 0xff;
+ for (int i = 2; i < numberFormat.length; i += 2) {
+ numberFormat[i] = 0x2e;
+ }
+ RevisionStoreObject numberList = numberListNode(numberListId,
numberFormat, null);
+ RevisionStoreObject listItem = listItem(listId, numberListId, "bullet
item");
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(listItem, numberList));
+ cell.rootDeclares.add(rootDeclare(listId));
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+
+ String xml = walkXml(pkg);
+
+ assertTrue(xml.contains("<ul>"), xml);
+ assertFalse(xml.contains("<ol"), xml);
+ }
+
+ @Test
+ public void testNumberListFormatsMapHtmlTypesAndRejectMalformedValues()
throws Exception {
+ String[] types = {"1", "I", "i", "A", "a"};
+ for (int code = 0; code < types.length; code++) {
+ String xml = singleListXml(1200 + code * 10,
numberListFormat(code), null);
+ assertTrue(xml.contains("<ol type=\"" + types[code] + "\">"), xml);
+ }
+
+ String unknown = singleListXml(1260, numberListFormat(5), null);
+ assertTrue(unknown.contains("<ol>"), unknown);
+ assertFalse(unknown.contains("<ol type="), unknown);
+
+ byte[][] malformedFormats = {null, {0}, {1, 0, 0}, {3, 0, (byte) 0xfd,
(byte) 0xff,
+ 1, 0}, {1, 0, (byte) 0xfd, (byte) 0xff}, {1, 0, 0x2e, 0}};
+ for (int i = 0; i < malformedFormats.length; i++) {
+ String xml = singleListXml(1270 + i * 10, malformedFormats[i],
null);
+ assertTrue(xml.contains("<ul>"), xml);
+ assertFalse(xml.contains("<ol"), xml);
+ }
+ }
+
+ @Test
+ public void testMalformedListFormatsDoNotShareUnorderedList() throws
Exception {
+ ExGuid outlineId = id(1320);
+ ExGuid firstItemId = id(1321);
+ ExGuid secondItemId = id(1322);
+ byte[] firstMalformedFormat = {3, 0, (byte) 0xfd, (byte) 0xff, 1, 0};
+ byte[] secondMalformedFormat = {4, 0, (byte) 0xfd, (byte) 0xff, 1, 0};
+ RevisionStoreObject firstNumberList = numberListNode(id(1323),
firstMalformedFormat,
+ null);
+ RevisionStoreObject secondNumberList = numberListNode(id(1324),
secondMalformedFormat,
+ null);
+ List<ExGuid> itemIds = Arrays.asList(firstItemId, secondItemId);
+ List<RevisionStoreObject> items = Arrays.asList(
+ listItem(firstItemId, firstNumberList.objectID, "first
malformed item"),
+ listItem(secondItemId, secondNumberList.objectID, "second
malformed item"));
+
+ String xml = walkListItems(outlineId, itemIds, items,
+ Arrays.asList(firstNumberList, secondNumberList), "");
+
+ assertEquals(2, countOccurrences(xml, "<ul>"), xml);
+ assertEquals(2, countOccurrences(xml, "<li>"), xml);
+ assertFalse(xml.contains("<ol"), xml);
+ }
+
+ @Test
+ public void testListStylesSplitOnFormatAndShareWhenEqual() throws
Exception {
+ int seed = 1350;
+ ExGuid outlineId = id(seed);
+ RevisionStoreObject upperRoman = numberListNode(id(seed + 30),
numberListFormat(1), null);
+ RevisionStoreObject lowerRoman = numberListNode(id(seed + 31),
numberListFormat(2), null);
+ byte[] lowerRomanWithSuffix = {3, 0, (byte) 0xfd, (byte) 0xff, 2, 0,
0x78, 0};
+ RevisionStoreObject lowerRomanSuffix = numberListNode(id(seed + 32),
+ lowerRomanWithSuffix, null);
+ RevisionStoreObject bullet = numberListNode(id(seed + 33),
+ new byte[] {1, 0, 0x22, 0x20}, null);
+ RevisionStoreObject unknown = numberListNode(id(seed + 34),
numberListFormat(5), null);
+ RevisionStoreObject decimal = numberListNode(id(seed + 35),
numberListFormat(0), null);
+
+ List<RevisionStoreObject> items = new ArrayList<>();
+ List<ExGuid> itemIds = new ArrayList<>();
+ RevisionStoreObject[] listNodes = {upperRoman, lowerRoman,
lowerRomanSuffix,
+ lowerRomanSuffix, lowerRomanSuffix, bullet, bullet, unknown,
decimal, unknown,
+ unknown};
+ for (int i = 0; i < listNodes.length; i++) {
+ ExGuid itemId = id(seed + 1 + i);
+ RevisionStoreObject item = listItem(itemId, listNodes[i].objectID,
"item " + i);
+ items.add(item);
+ itemIds.add(itemId);
+ }
+ ExGuid plainOutlineId = id(seed + 20);
+ RevisionStoreObject plainOutline = object(plainOutlineId,
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("plain outline text"))),
Collections.emptyList(),
+ Collections.emptyList());
+ setJcid(plainOutline, OneNoteJcid.OUTLINE_ELEMENT_NODE);
+ items.add(plainOutline);
+ itemIds.add(plainOutlineId);
+
+ List<RevisionStoreObject> nodes = Arrays.asList(upperRoman, lowerRoman,
+ lowerRomanSuffix, bullet, unknown, decimal);
+ String xml = walkListItems(outlineId, itemIds, items, nodes, "outline
caption");
+
+ assertEquals(6, countOccurrences(xml, "<ol"), xml);
+ assertEquals(1, countOccurrences(xml, "<ul>"), xml);
+ assertEquals(11, countOccurrences(xml, "<li"), xml);
+ assertTrue(xml.contains("plain outline text"), xml);
+ assertTrue(xml.contains("outline caption"), xml);
+ }
+
+ @Test
+ public void testOutlineIndentMetadataOnElementDoesNotSplitLists() throws
Exception {
+ RevisionStoreObject node = numberListNode(id(1380),
numberListFormat(0), null);
+ RevisionStoreObject first = listItem(id(1381), node.objectID, "first
item",
+ new byte[] {1, 0, 0, 0});
+ RevisionStoreObject second = listItem(id(1382), node.objectID, "second
item",
+ new byte[] {2, 0, 0, 0});
+ String xml = walkListItems(id(1383), Arrays.asList(first.objectID,
second.objectID),
+ Arrays.asList(first, second), Collections.singletonList(node),
"");
+ assertEquals(1, countOccurrences(xml, "<ol"), xml);
+ assertEquals(2, countOccurrences(xml, "<li"), xml);
+ }
+
+ @Test
+ public void testFallbackKeepsNestedTableInsideUnreachedListOwner() throws
Exception {
+ for (boolean useCellFallback : new boolean[] {false, true}) {
+ for (boolean cyclic : new boolean[] {false, true}) {
+ for (boolean tableFirst : new boolean[] {false, true}) {
+ RevisionStoreObject node = numberListNode(id(4202),
numberListFormat(0), null);
+ RevisionStoreObject item = listItem(id(4200),
node.objectID, "owner text");
+ RevisionStoreObject table = object(id(4201),
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("nested table text"))),
+ Collections.emptyList(), Collections.emptyList());
+ setJcid(table, OneNoteJcid.TABLE_NODE);
+ addChildReferences(item, cyclic ?
Arrays.asList(item.objectID, table.objectID) :
+ Collections.singletonList(table.objectID));
+ List<RevisionStoreObject> objects = new
ArrayList<>(tableFirst ?
+ Arrays.asList(table, item, node) :
Arrays.asList(item, table, node));
+ if (!cyclic) {
+ addPrefixChain(objects, item,
MSOneStorePackage.MAX_OBJECT_WALK_DEPTH, 7000);
+ }
+ MSOneStorePackage pkg = fallbackPackage(
+ group(objects.toArray(new
RevisionStoreObject[0])), useCellFallback);
+ Metadata metadata = new Metadata();
+ String xml = walkXml(pkg, metadata);
+ assertTrue(xml.matches("(?s).*<li><p>owner
text</p>\\s*<table>" +
+ ".*nested table text.*</table>\\s*</li>.*"), xml);
+ assertEquals(1, countOccurrences(xml, "<table>"), xml);
+ assertEquals(1, countOccurrences(xml, "owner text"), xml);
+ if (!cyclic) {
+
assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(w -> w.contains("depth limit " +
MSOneStorePackage.MAX_OBJECT_WALK_DEPTH)));
+ }
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testInvalidNumberListNodesAndRestartDataFallBackSafely()
throws Exception {
+ int seed = 1400;
+ ExGuid outlineId = id(seed);
+ ExGuid missingNodeId = id(seed + 50);
+ RevisionStoreObject noProperties = numberListNode(id(seed + 30), null,
null);
+ noProperties.propertySet = null;
+ RevisionStoreObject noObjectSpace = numberListNode(id(seed + 31),
numberListFormat(0), null);
+ noObjectSpace.propertySet.objectSpaceObjectPropSet = null;
+ RevisionStoreObject wrongJcid = numberListNode(id(seed + 32),
numberListFormat(0), null);
+ setJcid(wrongJcid, OneNoteJcid.OUTLINE_ELEMENT_NODE);
+ RevisionStoreObject missingFormat = numberListNode(id(seed + 33),
null, null);
+ RevisionStoreObject wrongFormatType =
numberListNodeWithProperties(id(seed + 34),
+ new PropertySpec(PropertyType.FourBytesOfData, 0x1C001C1A,
fourBytes(1)));
+ RevisionStoreObject wrongRestartType =
numberListNodeWithProperties(id(seed + 35),
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
0x1C001C1A,
+ bytes(numberListFormat(0))),
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
0x14001CB7,
+ bytes(new byte[] {1, 0, 0, 0})));
+ RevisionStoreObject nullRestartData =
numberListNodeWithProperties(id(seed + 36),
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
0x1C001C1A,
+ bytes(numberListFormat(0))),
+ new PropertySpec(PropertyType.FourBytesOfData, 0x14001CB7,
+ fourBytes((byte[]) null)));
+ RevisionStoreObject shortRestartData =
numberListNodeWithProperties(id(seed + 37),
+ new PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
0x1C001C1A,
+ bytes(numberListFormat(0))),
+ new PropertySpec(PropertyType.FourBytesOfData, 0x14001CB7,
+ fourBytes(new byte[] {1, 0, 0})));
+
+ RevisionStoreObject[] listNodes = {null, null, noProperties,
noObjectSpace, wrongJcid,
+ missingFormat, missingFormat, wrongFormatType,
wrongRestartType, nullRestartData,
+ shortRestartData};
+ List<RevisionStoreObject> items = new ArrayList<>();
+ List<ExGuid> itemIds = new ArrayList<>();
+ for (int i = 0; i < listNodes.length; i++) {
+ ExGuid itemId = id(seed + 1 + i);
+ ExGuid numberListId = listNodes[i] == null ?
+ (i == 0 ? missingNodeId : null) : listNodes[i].objectID;
+ items.add(listItem(itemId, numberListId, "invalid item " + i));
+ itemIds.add(itemId);
+ }
+ ExGuid plainItemId = id(seed + 20);
+ RevisionStoreObject plainItem = object(plainItemId,
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("unstyled outline item"))),
Collections.emptyList(),
+ Collections.emptyList());
+ setJcid(plainItem, OneNoteJcid.OUTLINE_ELEMENT_NODE);
+ items.add(plainItem);
+ itemIds.add(plainItemId);
+ itemIds.add(id(seed + 21));
+
+ List<RevisionStoreObject> nodes = new ArrayList<>();
+ for (RevisionStoreObject node : listNodes) {
+ if (node != null && !nodes.contains(node)) {
+ nodes.add(node);
+ }
+ }
+ String xml = walkListItems(outlineId, itemIds, items, nodes, "outline
text");
+
+ assertEquals(1, countOccurrences(xml, "<ul>"), xml);
+ assertEquals(1, countOccurrences(xml, "<ol type=\"1\">"), xml);
+ assertEquals(0, countOccurrences(xml, "<li value="), xml);
+ assertTrue(xml.contains("unstyled outline item"), xml);
+ for (int i = 0; i < listNodes.length; i++) {
+ assertTrue(xml.contains("<p>invalid item " + i + "</p>"), xml);
+ }
+ }
+
+ @Test
+ public void testFallbackRecoversDepthSkippedTableDescendants() throws
Exception {
+ RevisionStoreObject leaf = object(id(4003),
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("recovered leaf"))),
Collections.emptyList(), Collections.emptyList());
+ RevisionStoreObject tableCell = object(id(4002),
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(1))),
Collections.singletonList(leaf.objectID), Collections.emptyList());
+ RevisionStoreObject author = object(id(4005),
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001D75, utf16Text("Depth Author"))),
Collections.emptyList(), Collections.emptyList());
+ RevisionStoreObject picture = object(id(4006), propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ picture.fileDataObject = fileData("depth image");
+ RevisionStoreObject row = object(id(4001),
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(1)),
+ new PropertySpec(PropertyType.ObjectID, 0x20001D79,
new NoData()),
+ new PropertySpec(PropertyType.ObjectID, 0x20001D78,
new NoData()),
+ new PropertySpec(PropertyType.ObjectID, 0x20001C3F,
new NoData()),
+ new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001DD7, utf16Text("depth-image.png"))),
+ Arrays.asList(tableCell.objectID, author.objectID,
author.objectID, picture.objectID),
+ Collections.emptyList());
+ RevisionStoreObject table = object(id(4000),
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(1))),
Collections.singletonList(row.objectID), Collections.emptyList());
+ RevisionStoreObject shallow = object(id(4004),
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("shallow text"))),
Collections.emptyList(), Collections.emptyList());
+ setJcid(table, OneNoteJcid.TABLE_NODE);
+ setJcid(row, OneNoteJcid.TABLE_ROW_NODE);
+ setJcid(tableCell, OneNoteJcid.TABLE_CELL_NODE);
+ List<RevisionStoreObject> objects = new
ArrayList<>(Arrays.asList(shallow, table, row,
+ tableCell, leaf, author, picture));
+ addPrefixChain(objects, table, MSOneStorePackage.MAX_OBJECT_WALK_DEPTH
- 2, 4300);
+ for (boolean useCellFallback : new boolean[] {false, true}) {
+ MSOneStorePackage pkg = fallbackPackage(group(objects.toArray(new
RevisionStoreObject[0])),
+ useCellFallback);
+ Metadata metadata = new Metadata();
+ ParseContext context = new ParseContext();
+ List<String> recoveredResourceNames = new ArrayList<>();
+ context.set(EmbeddedDocumentExtractor.class, new
EmbeddedDocumentExtractor() {
+ @Override
+ public boolean shouldParseEmbedded(Metadata embeddedMetadata,
ParseContext parseContext) {
+ return true;
+ }
+
+ @Override
+ public void parseEmbedded(org.apache.tika.io.TikaInputStream
stream,
+ org.xml.sax.ContentHandler handler,
Metadata embeddedMetadata,
+ ParseContext parseContext, boolean
outputHtml) {
+
recoveredResourceNames.add(embeddedMetadata.get(TikaCoreProperties.RESOURCE_NAME_KEY));
+ }
+ });
+ String xml = walkXml(pkg, metadata, context);
+ assertTrue(xml.contains("shallow text"));
+ assertEquals(1, countOccurrences(xml, "recovered leaf"));
+ assertTrue(pkg.hasEmittedContent());
+ assertEquals(Collections.singletonList("depth-image.png"),
recoveredResourceNames);
+ assertEquals("Depth Author",
metadata.get(OneNote.MOST_RECENT_AUTHORS));
+ assertEquals("Depth Author",
metadata.get(OneNote.ORIGINAL_AUTHORS));
+ assertEquals("Depth Author",
metadata.get(TikaCoreProperties.CREATOR));
+
assertTrue(Arrays.stream(metadata.getValues(TikaCoreProperties.TIKA_META_EXCEPTION_WARNING))
+ .anyMatch(w -> w.contains("depth limit " +
MSOneStorePackage.MAX_OBJECT_WALK_DEPTH)));
+ }
+ }
+
+ @Test
+ public void testFallbackRecoveryKeepsDerivedPicturesSuppressed() throws
Exception {
+ for (boolean useCellFallback : new boolean[] {false, true}) {
+ RevisionStoreObject primary = object(id(4101), propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ primary.fileDataObject = fileData("primary image");
+ RevisionStoreObject derived = object(id(4102),
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("suppressed derived image"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreObject table = object(id(4100),
+ propertySet(new PropertySpec(PropertyType.ObjectID,
0x20001C3F, new NoData()),
+ new PropertySpec(PropertyType.ObjectID,
0x200034C8, new NoData())),
+ Arrays.asList(primary.objectID, derived.objectID),
Collections.emptyList());
+ setJcid(table, OneNoteJcid.TABLE_NODE);
+ RevisionStoreObjectGroup objects = group(derived, primary, table);
+ MSOneStorePackage pkg = fallbackPackage(objects, useCellFallback);
+
+ assertFalse(walkXml(pkg).contains("suppressed derived image"));
+ }
+ }
+
+ @Test
+ public void testFallbackRecoveryDoesNotPromoteRecentAuthors() throws
Exception {
+ for (boolean useCellFallback : new boolean[] {false, true}) {
+ RevisionStoreObject author = object(id(4121),
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001D75, utf16Text("Recent Author"))),
+ Collections.emptyList(), Collections.emptyList());
+ RevisionStoreObject table = object(id(4120),
+ propertySet(new PropertySpec(PropertyType.ObjectID,
0x20001D79, new NoData())),
+ Collections.singletonList(author.objectID),
Collections.emptyList());
+ setJcid(table, OneNoteJcid.TABLE_NODE);
+ MSOneStorePackage pkg = fallbackPackage(group(author, table),
useCellFallback);
+ Metadata metadata = new Metadata();
+ walkXml(pkg, metadata);
+
+ assertEquals("Recent Author",
metadata.get(OneNote.MOST_RECENT_AUTHORS));
+ assertNull(metadata.get(TikaCoreProperties.CREATOR));
+ }
+ }
+
+ @Test
+ public void testTableFallbackHandlesCyclesAndBrokenObjects() throws
Exception {
+ ExGuid outerTableId = id(1500);
+ ExGuid nestedTableId = id(1501);
+ ExGuid rowId = id(1502);
+ ExGuid cellId = id(1503);
+ ExGuid noPropertiesId = id(1504);
+ ExGuid noObjectSpaceId = id(1505);
+ ExGuid missingId = id(1506);
+
+ RevisionStoreObject outerTable = object(outerTableId,
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(5)),
+ new PropertySpec(PropertyType.ObjectID, 0x20001D00,
new NoData()),
+ new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("outer table text"))),
+ Arrays.asList(nestedTableId, rowId, noPropertiesId,
noObjectSpaceId, missingId),
+ Collections.emptyList());
+ RevisionStoreObject nestedTable = object(nestedTableId,
+ propertySet(new PropertySpec(PropertyType.ObjectID,
0x20001D00, new NoData())),
+ Collections.singletonList(outerTableId),
Collections.emptyList());
+ RevisionStoreObject row = object(rowId,
+ propertySet(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C20,
+ arrayNumber(2))), Arrays.asList(cellId, outerTableId),
Collections.emptyList());
+ RevisionStoreObject cell = object(cellId,
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("cycle cell text"))),
Collections.emptyList(),
+ Collections.emptyList());
+ RevisionStoreObject noProperties = new RevisionStoreObject();
+ noProperties.objectID = noPropertiesId;
+ RevisionStoreObject noObjectSpace = object(noObjectSpaceId,
propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ noObjectSpace.propertySet.objectSpaceObjectPropSet = null;
+ RevisionStoreObject withoutId = new RevisionStoreObject();
+ RevisionStoreObject rootNoObjectSpace = object(id(1507), propertySet(),
+ Collections.emptyList(), Collections.emptyList());
+ rootNoObjectSpace.propertySet.objectSpaceObjectPropSet = null;
+ setJcid(outerTable, OneNoteJcid.TABLE_NODE);
+ setJcid(nestedTable, OneNoteJcid.TABLE_NODE);
+ setJcid(row, OneNoteJcid.TABLE_ROW_NODE);
+ setJcid(cell, OneNoteJcid.TABLE_CELL_NODE);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.OtherFileNodeList.add(null);
+ RevisionStoreObjectGroup missingObjects = group();
+ missingObjects.objects = null;
+ pkg.OtherFileNodeList.add(missingObjects);
+ pkg.OtherFileNodeList.add(group(null, withoutId, rootNoObjectSpace,
row, cell,
+ nestedTable, noProperties, noObjectSpace, outerTable));
+ String xml = walkXml(pkg);
+
+ assertEquals(2, countOccurrences(xml, "<table>"), xml);
+ assertEquals(1, countOccurrences(xml, "<tr>"), xml);
+ assertEquals(1, countOccurrences(xml, "<td>"), xml);
+ assertTrue(xml.contains("outer table text"), xml);
+ assertTrue(xml.contains("cycle cell text"), xml);
+ }
+
+ @Test
+ public void testTableStructureSurvivesFallbackRootOrder() throws Exception
{
+ for (boolean useCellFallback : new boolean[] {false, true}) {
+ ExGuid tableId = id(1170);
+ ExGuid rowId = id(1171);
+ ExGuid cellId = id(1172);
+ RevisionStoreObject tableCell = object(cellId,
+ propertySet(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text("table text"))),
Collections.emptyList(),
+ Collections.emptyList());
+ RevisionStoreObject row = object(rowId,
+ propertySet(new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C20,
+ arrayNumber(1))),
Collections.singletonList(cellId),
+ Collections.emptyList());
+ RevisionStoreObject table = object(tableId,
+ propertySet(new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C20,
+ arrayNumber(1))), Collections.singletonList(rowId),
+ Collections.emptyList());
+ setJcid(tableCell, OneNoteJcid.TABLE_CELL_NODE);
+ setJcid(row, OneNoteJcid.TABLE_ROW_NODE);
+ setJcid(table, OneNoteJcid.TABLE_NODE);
+
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ RevisionStoreObjectGroup outOfOrder = group(row, tableCell, table);
+ if (useCellFallback) {
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(outOfOrder);
+ pkg.cells.add(cell);
+ } else {
+ pkg.OtherFileNodeList.add(outOfOrder);
+ }
+
+ String xml = walkXml(pkg);
+ int tableStart = xml.indexOf("<table>");
+ int rowStart = xml.indexOf("<tr>", tableStart);
+ int cellStart = xml.indexOf("<td>", rowStart);
+ int textStart = xml.indexOf("table text", cellStart);
+ int cellEnd = xml.indexOf("</td>", textStart);
+ int rowEnd = xml.indexOf("</tr>", cellEnd);
+ int tableEnd = xml.indexOf("</table>", rowEnd);
+ assertTrue(tableStart >= 0 && rowStart > tableStart && cellStart >
rowStart &&
+ textStart > cellStart && cellEnd > textStart && rowEnd >
cellEnd &&
+ tableEnd > rowEnd, xml);
+ }
+ }
+
+ @Test
+ public void testTableListStructureAndGuidsSurviveCombinedWalk() throws
Exception {
+ for (int mode = 0; mode < 3; mode++) {
+ ExGuid tableId = id(1600);
+ ExGuid rowId = id(1601);
+ ExGuid cellId = id(1602);
+ ExGuid itemId = id(1603);
+ RevisionStoreObject numberList = numberListNode(id(1604),
numberListFormat(1), 5);
+ RevisionStoreObject item = listItem(itemId, numberList.objectID,
"combined cell text");
+ RevisionStoreObject pageMetadata = notebookGuidObject(1605,
OneNoteJcid.PAGE_METADATA,
+ sectionGuidBytes());
+ RevisionStoreObject tableCell = object(cellId,
+ propertySet(new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C20,
+ arrayNumber(2))), Arrays.asList(itemId,
pageMetadata.objectID),
+ Collections.emptyList());
+ RevisionStoreObject row = object(rowId,
+ propertySet(new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C20,
+ arrayNumber(1))),
Collections.singletonList(cellId),
+ Collections.emptyList());
+ RevisionStoreObject table = object(tableId,
+ propertySet(new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C20,
+ arrayNumber(1))), Collections.singletonList(rowId),
+ Collections.emptyList());
+ setJcid(table, OneNoteJcid.TABLE_NODE);
+ setJcid(row, OneNoteJcid.TABLE_ROW_NODE);
+ setJcid(tableCell, OneNoteJcid.TABLE_CELL_NODE);
+ RevisionStoreObjectGroup objects = group(item, tableCell, row,
numberList,
+ pageMetadata, table);
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ if (mode == 0) {
+ pkg.OtherFileNodeList.add(objects);
+ } else {
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(objects);
+ if (mode == 2) {
+ cell.rootDeclares.add(rootDeclare(tableId));
+ }
+ pkg.cells.add(cell);
+ }
+ Metadata metadata = new Metadata();
+ String xml = walkXml(pkg, metadata);
+
+ assertTrue(xml.matches("(?s).*<table>\\s*<tr>\\s*<td>\\s*<ol
type=\"I\">\\s*" +
+ "<li value=\"5\">.*combined cell
text.*</li>\\s*</ol>\\s*</td>\\s*" +
+ "</tr>\\s*</table>.*"), xml);
+ assertEquals(1, countOccurrences(xml, "<table>"), xml);
+ assertEquals(1, countOccurrences(xml, "<li"), xml);
+ assertArrayEquals(new String[]
{"{00112233-4455-6677-8899-AABBCCDDEEFF}"},
+ metadata.getValues(OneNote.PAGE_GUIDS));
+ }
+ }
+
@Test
public void testSectionAndPageGuidsAreExtracted() throws Exception {
String sectionGuid = "{00112233-4455-6677-8899-AABBCCDDEEFF}";
@@ -1677,7 +2445,11 @@ public class MSOneStorePackageTest {
}
private static String walkXml(MSOneStorePackage pkg, Metadata metadata)
throws Exception {
- ParseContext context = new ParseContext();
+ return walkXml(pkg, metadata, new ParseContext());
+ }
+
+ private static String walkXml(MSOneStorePackage pkg, Metadata metadata,
ParseContext context)
+ throws Exception {
ToXMLContentHandler xml = new ToXMLContentHandler();
XHTMLContentHandler xhtml = new XHTMLContentHandler(xml, metadata,
context);
xhtml.startDocument();
@@ -1686,8 +2458,95 @@ public class MSOneStorePackageTest {
return xml.toString();
}
+ private static MSOneStorePackage fallbackPackage(RevisionStoreObjectGroup
objects,
+ boolean
useCellFallback) {
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ if (useCellFallback) {
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(objects);
+ pkg.cells.add(cell);
+ } else {
+ pkg.OtherFileNodeList.add(objects);
+ }
+ return pkg;
+ }
+
+ private static void addChildReferences(RevisionStoreObject object,
List<ExGuid> children) {
+ PropertySet body = object.propertySet.objectSpaceObjectPropSet.body;
+ body.rgPrids = Arrays.copyOf(body.rgPrids, body.rgPrids.length + 1);
+ body.rgPrids[body.rgPrids.length - 1] =
propertyID(PropertyType.ArrayOfObjectIDs, 0x24001C20);
+ body.rgData.add(arrayNumber(children.size()));
+ body.cProperties++;
+ List<ExGuid> references = new
ArrayList<>(object.referencedObjectID.content);
+ references.addAll(children);
+ object.referencedObjectID.content = references;
+ }
+
+ private static void addPrefixChain(List<RevisionStoreObject> objects,
RevisionStoreObject child,
+ int depth, int seed) throws Exception {
+ RevisionStoreObject next = child;
+ for (int i = depth - 1; i >= 0; i--) {
+ next = object(id(seed + i),
+ propertySet(new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C20,
+ arrayNumber(1))),
Collections.singletonList(next.objectID), Collections.emptyList());
+ if (i == 0) {
+ setJcid(next, OneNoteJcid.OUTLINE_NODE);
+ }
+ objects.add(next);
+ }
+ }
+
+ private static byte[] numberListFormat(int code) {
+ return new byte[] {2, 0, (byte) 0xfd, (byte) 0xff, (byte) code, 0};
+ }
+
+ private static String singleListXml(int seed, byte[] format, Integer
restart)
+ throws Exception {
+ return singleListXml(seed,
+ numberListNode(id(seed + 2), format, restart));
+ }
+
+ private static String singleListXml(int seed, RevisionStoreObject
numberList)
+ throws Exception {
+ ExGuid outlineId = id(seed);
+ ExGuid itemId = id(seed + 1);
+ List<ExGuid> itemIds = Collections.singletonList(itemId);
+ List<RevisionStoreObject> items = Collections.singletonList(
+ listItem(itemId, numberList.objectID, "single item"));
+ return walkListItems(outlineId, itemIds, items,
Collections.singletonList(numberList),
+ "");
+ }
+
+ private static String walkListItems(ExGuid outlineId, List<ExGuid> itemIds,
+ List<RevisionStoreObject> items,
+ List<RevisionStoreObject> numberLists,
String caption)
+ throws Exception {
+ PropertySpec childArray = new
PropertySpec(PropertyType.ArrayOfObjectIDs, 0x24001C20,
+ arrayNumber(itemIds.size()));
+ List<PropertySpec> properties = new ArrayList<>();
+ properties.add(childArray);
+ if (!caption.isEmpty()) {
+ properties.add(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text(caption)));
+ }
+ RevisionStoreObject outline = object(outlineId,
+ propertySet(properties.toArray(new PropertySpec[0])), itemIds,
+ Collections.emptyList());
+ setJcid(outline, OneNoteJcid.OUTLINE_NODE);
+ List<RevisionStoreObject> objects = new ArrayList<>();
+ objects.add(outline);
+ objects.addAll(items);
+ objects.addAll(numberLists);
+ RevisionStoreCell cell = new RevisionStoreCell();
+ cell.objectGroups.add(group(objects.toArray(new
RevisionStoreObject[0])));
+ cell.rootDeclares.add(rootDeclare(outlineId));
+ MSOneStorePackage pkg = new MSOneStorePackage();
+ pkg.cells.add(cell);
+ return walkXml(pkg);
+ }
+
private static int count(String value, String needle) {
- return value.split(java.util.regex.Pattern.quote(needle), -1).length -
1;
+ return countOccurrences(value, needle);
}
private static RevisionManifestRootDeclare rootDeclare(ExGuid objectID) {
@@ -1702,6 +2561,76 @@ public class MSOneStorePackageTest {
object.jcid = jcidObject;
}
+ private static RevisionStoreObject numberListNode(ExGuid objectId, byte[]
format,
+ Integer restart) throws
Exception {
+ List<PropertySpec> properties = new ArrayList<>();
+ if (format != null) {
+ properties.add(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001C1A, bytes(format)));
+ }
+ if (restart != null) {
+ properties.add(new PropertySpec(PropertyType.FourBytesOfData,
0x14001CB7,
+ fourBytes(restart)));
+ }
+ return numberListNodeWithProperties(objectId,
+ properties.toArray(new PropertySpec[0]));
+ }
+
+ private static RevisionStoreObject numberListNodeWithProperties(ExGuid
objectId,
+
PropertySpec... properties)
+ throws Exception {
+ RevisionStoreObject numberList = object(objectId,
propertySet(properties),
+ Collections.emptyList(), Collections.emptyList());
+ setJcid(numberList, OneNoteJcid.NUMBER_LIST_NODE);
+ return numberList;
+ }
+
+ private static RevisionStoreObject listItem(ExGuid objectId, ExGuid
numberListId,
+ String text) throws Exception {
+ return listItem(objectId, numberListId, text, null);
+ }
+
+ private static RevisionStoreObject listItem(ExGuid objectId, ExGuid
numberListId,
+ String text, byte[] indent)
throws Exception {
+ List<PropertySpec> properties = new ArrayList<>();
+ properties.add(new PropertySpec(PropertyType.ArrayOfObjectIDs,
0x24001C26,
+ arrayNumber(1)));
+ properties.add(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C003498, text(text)));
+ if (indent != null) {
+ properties.add(new
PropertySpec(PropertyType.FourBytesOfLengthFollowedByData,
+ 0x1C001C12, bytes(indent)));
+ }
+ RevisionStoreObject listItem = object(objectId,
+ propertySet(properties.toArray(new PropertySpec[0])),
+ Collections.singletonList(numberListId),
Collections.emptyList());
+ setJcid(listItem, OneNoteJcid.OUTLINE_ELEMENT_NODE);
+ return listItem;
+ }
+
+ private static FourBytesOfData fourBytes(int value) {
+ FourBytesOfData data = new FourBytesOfData();
+ data.data = new byte[] {(byte) value, (byte) (value >>> 8), (byte)
(value >>> 16),
+ (byte) (value >>> 24)};
+ return data;
+ }
+
+ private static FourBytesOfData fourBytes(byte[] value) {
+ FourBytesOfData data = new FourBytesOfData();
+ data.data = value;
+ return data;
+ }
+
+ private static int countOccurrences(String text, String fragment) {
+ int count = 0;
+ int index = 0;
+ while ((index = text.indexOf(fragment, index)) >= 0) {
+ count++;
+ index += fragment.length();
+ }
+ return count;
+ }
+
/** Adds a PAGE_METADATA guid object as a second root so the page div
carries its id. */
private static void attachPageGuid(RevisionStoreCell cell, int objectId,
byte firstGroupByte)
throws Exception {
diff --git
a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNoteTablesLists.one
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNoteTablesLists.one
new file mode 100644
index 0000000000..034029648d
Binary files /dev/null and
b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/test/resources/test-documents/testOneNoteTablesLists.one
differ
