[
https://issues.apache.org/jira/browse/TIKA-5057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18142945#comment-18142945
]
Tim Allison commented on TIKA-5057:
-----------------------------------
:robot: status update.
h3. Status
PR #3564 is the review and refactor of the OneNote parser, for .one sections
(OneNote 2007–2016) and FSSHTTPB sections (OneNote for the web, Windows 10 and
later). The code is complete and tested; what is left is a rebase onto main and
a fresh review.
* *Current content by default.* Output describes the section as it is now, like
the final state of an incrementally updated PDF. Text comes from each page's
current revision, and metadata (authors, timestamps, page GUIDs) follows the
text. New \{{one-note-parser}} config, both default false:
** \{{includePageHistory}}: earlier revisions and saved page versions, with
their metadata. Exception: .one attachments sit in one store shared by all
versions, and Tika cannot yet link them to page versions. Every stored
attachment is extracted either way, including files only earlier versions used.
** \{{includeDeletedContent}}: pages moved to the recycle bin (FSSHTTPB only).
Before, they came out as live text, once per stored version.
* *Each stored file is extracted once.* FSSHTTPB sections used to emit a file
once per reference to it. One 161 MB corpus section goes from 19 attachments
(245 MB) to 12. .one sections already did this.
* *Correctness:*
** The current revision is the one that last took the current-revision role.
Objects resolve within their own revision, which stops old and new text being
mixed.
** File node lists end at the transaction log's committed count, which recovers
text that 4.2.0 misses.
** Text: windows-1252 extended ASCII; NULs and U+FFFC placeholders removed;
text with no NUL terminator kept.
** Handwriting words stay aligned with their placeholders when a word has no
recognition.
** FSSHTTPB GUIDs are read in Microsoft byte order. They were MD5-hashed, so
password-protected FSSHTTPB sections were never detected.
** \{{dc:creator}} includes original authors. The creation timestamp no longer
reports \{{Long.MAX_VALUE}} when a section has none.
** A failure partway through a walk keeps the structured output and adds a
warning, instead of appending the legacy text dump.
* *New output:* pictures as \{{<img>}}, OneNote's OCR text, handwriting
recognition, hyperlinks, OneNote 2007 paragraph text, and table-of-contents
sections. Encrypted sections in both formats throw
\{{EncryptedDocumentException}}.
* *Resource limits,* all scaled to file size:
** budgets on property values, stream entries, file nodes, global ID copies,
transaction-log entries, authors and emitted text;
** bounded walk depth, walk visits (charged per property) and dependency-chain
lookups;
** embedded bytes capped at the file size;
** FSSHTTPB cells that share one manifest walked once.
h3. Performance
Wall time is the mean per parse; allocation is per parse; GC count and time are
per run. All are summed over the files in each set.
*17 OneNote test files in tika-parser-microsoft-module, 1,000 parses each* (the
TIKA-5055 benchmark):
||Build||Wall||Allocated per parse||GC count||GC time||
|4.2.0|304 ms|1,029 MiB|6,533|12.7 s|
|main before TIKA-5055|298 ms|1,029 MiB|6,337|11.9 s|
|main with TIKA-5055|253 ms|1,003 MiB|4,791|7.1 s|
|this PR|24 ms|31 MiB|251|0.2 s|
*57 FSSHTTPB corpus files, 200 parses each:*
||Build||Wall||Allocated per parse||GC count||GC time||
|4.2.0|2,794 ms|2,313 MiB|6,351|121 s|
|main before TIKA-5055|2,678 ms|2,314 MiB|6,502|114 s|
|main with TIKA-5055|190 ms|519 MiB|734|0.7 s|
|this PR|189 ms|501 MiB|649|0.7 s|
*4 largest .one corpus files (18–41 MB), 3 parses each* (rerun 2026-11-11 at
load under 1.2, two runs each; main matched its earlier row):
||Build||Wall||Allocated per parse||GC time||
|4.2.0|1,677 ms|2,280 MiB|85 ms|
|main with TIKA-5055|1,514 ms|2,279 MiB|96 ms|
|this PR, earlier|3,195 ms|1,284 MiB|619 ms|
|this PR, now|2,430 ms|785 MiB|435 ms|
* TIKA-5055 is what fixed FSSHTTPB. This PR's small-file gains come from
reading .one sections through a 64 KB window instead of a new 8 KB buffer and a
system call per read, and from dropping a node map nothing read.
* Large multi-revision .one sections still cost more than on main: 1.6 times
the wall time and 4.5 times the GC time, though the PR allocates a third as
much. Main stops reading their file node lists early. On the 41 MB section it
sees 31 revisions and 1,590 objects, where this PR reads all 185 revisions and
56,333 objects.
* Since the last update: metadata no longer takes a second walk over every
revision, property values are read on first use, and file nodes allocate only
their own type's fields. The 41 MB section now parses in ~225 MB of heap, down
from ~350 MB (main: 96 MB). What remains is each revision's file node tree and
object ID streams, built up front; that is a follow-up.
h3. Testing
* *Module tests:* 778 run, 0 failures, 14 skipped.
* *Corpus:* 913 unique files: 804 sections and 109 table-of-contents files,
from Common Crawl, open-source repositories and test fixtures, and unpacked
.onepkg packages.
** Sections by format: 709 OneNote 2010–2016, 57 FSSHTTPB, 31 OneNote 2007, 6
OneNote 2003 and 1 truncated.
** Output was diffed before and after every change, and every difference was
intended.
* *Replay:* every corpus file through the tika-fuzzing harness (TIKA-5060) in
bytes, stream and path modes, on the final code. 2,739 parses, no hangs, OOMs
or unexpected exceptions, in normal and strict mode.
* *Fuzzing:* coverage-guided Jazzer runs of \{{AutoDetectParserFuzzTest}},
seeded with OneNote files: one hour before the resource-limit fixes (388,410
executions) and 30 minutes after them (212,925 executions). No crashes, hangs
or OOMs.
h3. Follow-ups
* TIKA-XXXX: output gaps between .one and FSSHTTPB sections (linking .one
pictures and attachments to their file data, marking .one page history,
FSSHTTPB handwriting, OneNote 2007 pictures)
* TIKA-XXXX: deleted-page detection in both formats, with a real test file
* TIKA-XXXX: .one object resolution per global ID table (lazy object ID
streams, narrowing the last-declaration fallback)
> General improvements to onenote parser
> --------------------------------------
>
> Key: TIKA-5057
> URL: https://issues.apache.org/jira/browse/TIKA-5057
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Trivial
>
> When I have extra tokens, I'll have an agent focus on a parser at a time for
> robustness and simplicity. Let's start with onenote. CC [~henrylindemanglean]
> :robot: swarm recommends:
> {noformat}
> Native .one path (the common format, unrelated to TIKA-5055):
> 1. Exponential parse on shared child lists (OneNotePtr:714). A file-node
> list referenced twice is parsed twice. Only the current path is checked for
> cycles, so a crafted file of a few KB blows up as 2^depth. Three reviewers
> found this independently.
> 2. Exponential tree walk (OneNoteTreeWalker:648). The same shape: shared
> objects are walked once per reference path. Each visit also builds and keeps
> a HashMap of decoded properties, which OneNoteParser:144 throws away.
> 3. Wrong references, lost text (OneNotePtr:1185). The OID/OSID/ContextID
> stream position is copied to a local variable and never written back. Every
> reference property starts again at stream[0], so later references are never
> walked. It also removes the
> cap on reference counts.
> 4. Duplicate text in real files (OneNoteTreeWalker:569-592). Reproduced:
> testOneNote2.one produces 176 paragraphs, 105 of them "Untitled picture.png".
> Image filenames and TextExtendedAscii text are never deduplicated and ignore
> the revision filter.
> 5. Unchecked counts and budgets (OneNotePtr:838, :518, :809):
> - ObjectInfoDependencyOverrideData counts are unbounded.
> - The global-ID cap applies per node, with no total.
> - A negative FileDataStoreObject length gets through.
> 6. Smaller issues:
> - Embedded file data is read in full before shouldParseEmbedded is asked
> (OneNoteTreeWalker:435).
> - The UTF-16 NUL terminator stays in author names and image filenames.
> Legacy string dump, used whenever a parse falls back
> (OneNoteLegacyDumpStrings:95):
> - Never resets its buffer at 1 MB boundaries. A long run is emitted again
> at every boundary, so work grows quadratically and normal files over 1 MB get
> duplicated text.
> - Small bugs: it drops the last byte of each buffer,
> Character.isWhitespace(i) tests the loop index instead of the character, and
> split(" ") allocates one String per word.
> Fallback after a partial walk (OneNoteParser:172, :197): on both paths, the
> full legacy dump is added after whatever the failed walk already wrote, so
> content appears twice.
> FSSHTTPB:
> - GUIDs are MD5 hashes, not decoded (BitReader:140, AdapterHelper:37). They
> come from UUID.nameUUIDFromBytes. As a result:
> - Encrypted sections are never detected; the check at
> MSOneStoreParser:328 can't match.
> - Embedded relationship IDs contain hashed GUIDs.
> - Every cell mapping copies its whole revision chain
> (MSOneStoreParser:362), so the cost grows with mappings × objects.
> - Slow spots:
> - removeSupersededObjects is O(n²) (:393).
> - resourceInfoFromReferencingObject scans every object for each unnamed
> blob (MSOneStorePackage:1140).
> - hasUsablePicture copies a whole blob just to check it's non-empty
> (:1118).
> - Small port bugs:
> - cProperties is read as signed, so values ≥ 0x8000 abort the parse.
> - An undefined property type throws an NPE.
> - StorageManifestSchemaGUID reads from offset 0 instead of the current
> position.
> - BitConverter's bounds check ignores the index.
> Dead code:
> - The serialize/write path: 65 files, about 970 lines.
> - The FSSHTTPD chunking/build side.
> - getContent() and its builders.
> - Unreferenced classes: Error, PropertyIDType, RequestTypes, UMath.
> - Debug print() methods and option branches that are never set.
> {noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
