[
https://issues.apache.org/jira/browse/TIKA-5057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141661#comment-18141661
]
Tim Allison edited comment on TIKA-5057 at 11/10/26, 12:49 AM:
---------------------------------------------------------------
Bot comments on current state:
h3. Status
The review and refactor of the OneNote parser is in PR #3564. Summary of where
it stands:
* *Current text only.* .one sections give each page's current revision, not
every earlier revision and version-history snapshot. Metadata and attachments
still come from all revisions. New {{one-note-parser}} config, both default
false:
** {{includePageHistory}}: earlier revisions and saved page versions.
** {{includeDeletedContent}}: pages moved to the recycle bin; FSSHTTPB sections
only.
* *Correctness:*
** Objects resolve within their own revision, which stops old and new text
being mixed.
** File node lists end at the transaction log's committed count, which recovers
text that 4.2.0 misses.
** Text cleanup: windows-1252 extended ASCII; no NULs or U+FFFC placeholders.
* *New output:*
** Pictures as {{<img>}}, OneNote's OCR text, handwriting recognition and
hyperlinks.
** OneNote 2007 paragraph text, and table-of-contents sections.
** Encrypted sections fail with EncryptedDocumentException.
* *Limits:* bounded file node lists, walk visits and depth, and a document-wide
budget on property values that scales with file size.
h3. Performance and GC
Four builds: 4.2.0, main just before TIKA-5055, main with TIKA-5055, and this
PR. Wall time is the mean per parse; allocation, GC count and GC time are per
run. All are summed over the files in each set. Machine load was 0.7–3.4 during
the runs, so wall times are approximate; allocation and GC counts are not
affected.
*17 OneNote test files in tika-parser-microsoft-module (15 .one, 2 FSSHTTPB),
1,000 parses each* (the TIKA-5055 benchmark):
||Build||Wall||Allocated per parse||GC count||GC time||
|4.2.0|304 ms|1,029 MiB|6,533|12.7 s|
|main before TIKA-5055|298 ms|1,029 MiB|6,337|11.9 s|
|main with TIKA-5055|253 ms|1,003 MiB|4,791|7.1 s|
|this PR|28 ms|44 MiB|325|0.3 s|
*57 FSSHTTPB corpus files, 200 parses each:*
||Build||Wall||Allocated per parse||GC count||GC time||
|4.2.0|2,794 ms|2,313 MiB|6,351|121 s|
|main before TIKA-5055|2,678 ms|2,314 MiB|6,502|114 s|
|main with TIKA-5055|190 ms|519 MiB|734|0.7 s|
|this PR|182 ms|503 MiB|627|0.6 s|
* TIKA-5055 is what fixed FSSHTTPB. One 161 MB file dominates these totals;
TIKA-5055 takes it from 1,951 MiB and 2.4 s per parse to 390 MiB and 142 ms.
Without that file, the other 56 go from 363 MiB and 5.7 s of GC to 130 MiB and
0.25 s; this PR brings them to 114 MiB and 0.21 s.
* For .one sections, the PR's gains come from {{OneNoteDirectFileResource}}.
Every read of a few bytes allocated a new 8 KB transfer buffer and made at
least one system call. Reads now come from a 64 KB window of the file. The tree
walker also no longer builds a nested map of every node and property that
nothing read, and {{ExtendedGUID.hashCode()}} no longer allocates.
*4 largest .one corpus files (18–41 MB), 3 parses each:*
||Build||Wall||Allocated per parse||GC time||
|4.2.0|1,677 ms|2,280 MiB|85 ms|
|main with TIKA-5055|1,514 ms|2,279 MiB|96 ms|
|this PR|3,854 ms|2,930 MiB|835 ms|
Large sections with many revisions cost more in this PR: about 2.5 times the
wall time and 9 times the GC time of main. 4.2.0 stops reading their file node
lists early: on the 41 MB section it sees 31 revisions and 1,590 objects, while
this PR reads all 185 revisions and 56,333 objects. Lowering that means
deserializing only the revisions a parse walks; about half the work on these
files.
h3. Testing
* *Corpus:* 913 unique files: 804 sections and 109 table-of-contents files.
** Sources: Common Crawl, open-source repositories and test fixtures, and
sections unpacked from .onepkg packages.
** Sections by format: 709 OneNote 2010–2016, 57 FSSHTTPB, 31 OneNote 2007, 6
OneNote 2003, and 1 truncated.
** 877 parse with content. The rest are 13 encrypted, 12 with no content, 6
OneNote 2003 files that fall back to the legacy text dump, and a few broken or
synthetic files.
* *Replay:* every corpus file through the tika-fuzzing harness (TIKA-5060) in
bytes, stream and path modes: 2,739 parses, no hangs, OOMs, crashes or
unexpected exceptions, including in strict mode.
* *Fuzzing:* a one-hour coverage-guided Jazzer run of
{{AutoDetectParserFuzzTest}}, seeded with 56 OneNote files: 417,088 executions,
and no crashes, hangs or OOMs.
h3. Follow-ups
* OneNote 2007 inline pictures
* Handwriting recognition in FSSHTTPB sections
* Linking .one pictures to their attachment data
* Marking page-history text in .one output
* Checking whether .one sections keep deleted pages
* A small test file with a deleted page
* Lower memory for large multi-revision sections
was (Author: [email protected]):
Bot comments on current state:
h3. Status
The review and refactor of the OneNote parser is in PR #3564. Summary of where
it stands:
* *Current text only.* .one sections give each page's current revision, not
every earlier revision and version-history snapshot. Metadata and attachments
still come from all revisions. New {{one-note-parser}} config, both default
false:
** {{includePageHistory}}: earlier revisions and saved page versions.
** {{includeDeletedContent}}: pages moved to the recycle bin; FSSHTTPB sections
only.
* *Correctness:*
** Objects resolve within their own revision, which stops old and new text
being mixed.
** File node lists end at the transaction log's committed count, which recovers
text that 4.2.0 misses.
** Text cleanup: windows-1252 extended ASCII; no NULs or U+FFFC placeholders.
* *New output:*
** Pictures as {{<img>}}, OneNote's OCR text, handwriting recognition and
hyperlinks.
** OneNote 2007 paragraph text, and table-of-contents sections.
** Encrypted sections fail with EncryptedDocumentException.
* *Limits:* bounded file node lists, walk visits and depth, and a document-wide
budget on property values that scales with file size.
h3. Performance and GC
Four builds: 4.2.0, main just before TIKA-5055, main with TIKA-5055, and this
PR. Wall time is the mean per parse; allocation, GC count and GC time are per
run. All are summed over the files in each set. Machine load was 0.7–3.4 during
the runs, so wall times are approximate; allocation and GC counts are not
affected.
*17 OneNote test files in tika-parser-microsoft-module (15 .one, 2 FSSHTTPB),
1,000 parses each* (the TIKA-5055 benchmark):
||Build||Wall||Allocated per parse||GC count||GC time||
|4.2.0|304 ms|1,029 MiB|6,533|12.7 s|
|main before TIKA-5055|298 ms|1,029 MiB|6,337|11.9 s|
|main with TIKA-5055|253 ms|1,003 MiB|4,791|7.1 s|
|this PR|28 ms|44 MiB|325|0.3 s|
*57 FSSHTTPB corpus files, 200 parses each:*
||Build||Wall||Allocated per parse||GC count||GC time||
|4.2.0|2,794 ms|2,313 MiB|6,351|121 s|
|main before TIKA-5055|2,678 ms|2,314 MiB|6,502|114 s|
|main with TIKA-5055|190 ms|519 MiB|734|0.7 s|
|this PR|182 ms|503 MiB|627|0.6 s|
* TIKA-5055 is what fixed FSSHTTPB. One 161 MB file dominates these totals;
TIKA-5055 takes it from 1,951 MiB and 2.4 s per parse to 390 MiB and 142 ms.
Without that file, the other 56 go from 363 MiB and 5.7 s of GC to 130 MiB and
0.25 s; this PR brings them to 114 MiB and 0.21 s.
* For .one sections, the PR's gains come from {{OneNoteDirectFileResource}}.
Every read of a few bytes allocated a new 8 KB transfer buffer and made at
least one system call. Reads now come from a 64 KB window of the file. The tree
walker also no longer builds a nested map of every node and property that
nothing read, and {{ExtendedGUID.hashCode()}} no longer allocates.
*4 largest .one corpus files (18–41 MB), 3 parses each:*
||Build||Wall||Allocated per parse||GC time||
|4.2.0|1,677 ms|2,280 MiB|85 ms|
|main with TIKA-5055|1,514 ms|2,279 MiB|96 ms|
|this PR|3,854 ms|2,930 MiB|835 ms|
Large sections with many revisions cost more in this PR: about 2.5 times the
wall time and 9 times the GC time of main. 4.2.0 stops reading their file node
lists early: on the 41 MB section it sees 31 revisions and 1,590 objects, while
this PR reads all 185 revisions and 56,333 objects. Lowering that means
releasing deserialized objects during the parse, a structural change better
done as a follow-up.
h3. Testing
* *Corpus:* 913 unique files: 804 sections and 109 table-of-contents files.
** Sources: Common Crawl, open-source repositories and test fixtures, and
sections unpacked from .onepkg packages.
** Sections by format: 709 OneNote 2010–2016, 57 FSSHTTPB, 31 OneNote 2007, 6
OneNote 2003, and 1 truncated.
** 877 parse with content. The rest are 13 encrypted, 12 with no content, 6
OneNote 2003 files that fall back to the legacy text dump, and a few broken or
synthetic files.
* *Replay:* every corpus file through the tika-fuzzing harness (TIKA-5060) in
bytes, stream and path modes: 2,739 parses, no hangs, OOMs, crashes or
unexpected exceptions, including in strict mode.
* *Fuzzing:* a one-hour coverage-guided Jazzer run of
{{AutoDetectParserFuzzTest}}, seeded with 56 OneNote files: 417,088 executions,
and no crashes, hangs or OOMs.
h3. Follow-ups
* OneNote 2007 inline pictures
* Handwriting recognition in FSSHTTPB sections
* Linking .one pictures to their attachment data
* Marking page-history text in .one output
* Checking whether .one sections keep deleted pages
* A small test file with a deleted page
* Lower memory for large multi-revision sections
> General improvements to onenote parser
> --------------------------------------
>
> Key: TIKA-5057
> URL: https://issues.apache.org/jira/browse/TIKA-5057
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Trivial
>
> When I have extra tokens, I'll have an agent focus on a parser at a time for
> robustness and simplicity. Let's start with onenote. CC [~henrylindemanglean]
> :robot: swarm recommends:
> {noformat}
> Native .one path (the common format, unrelated to TIKA-5055):
> 1. Exponential parse on shared child lists (OneNotePtr:714). A file-node
> list referenced twice is parsed twice. Only the current path is checked for
> cycles, so a crafted file of a few KB blows up as 2^depth. Three reviewers
> found this independently.
> 2. Exponential tree walk (OneNoteTreeWalker:648). The same shape: shared
> objects are walked once per reference path. Each visit also builds and keeps
> a HashMap of decoded properties, which OneNoteParser:144 throws away.
> 3. Wrong references, lost text (OneNotePtr:1185). The OID/OSID/ContextID
> stream position is copied to a local variable and never written back. Every
> reference property starts again at stream[0], so later references are never
> walked. It also removes the
> cap on reference counts.
> 4. Duplicate text in real files (OneNoteTreeWalker:569-592). Reproduced:
> testOneNote2.one produces 176 paragraphs, 105 of them "Untitled picture.png".
> Image filenames and TextExtendedAscii text are never deduplicated and ignore
> the revision filter.
> 5. Unchecked counts and budgets (OneNotePtr:838, :518, :809):
> - ObjectInfoDependencyOverrideData counts are unbounded.
> - The global-ID cap applies per node, with no total.
> - A negative FileDataStoreObject length gets through.
> 6. Smaller issues:
> - Embedded file data is read in full before shouldParseEmbedded is asked
> (OneNoteTreeWalker:435).
> - The UTF-16 NUL terminator stays in author names and image filenames.
> Legacy string dump, used whenever a parse falls back
> (OneNoteLegacyDumpStrings:95):
> - Never resets its buffer at 1 MB boundaries. A long run is emitted again
> at every boundary, so work grows quadratically and normal files over 1 MB get
> duplicated text.
> - Small bugs: it drops the last byte of each buffer,
> Character.isWhitespace(i) tests the loop index instead of the character, and
> split(" ") allocates one String per word.
> Fallback after a partial walk (OneNoteParser:172, :197): on both paths, the
> full legacy dump is added after whatever the failed walk already wrote, so
> content appears twice.
> FSSHTTPB:
> - GUIDs are MD5 hashes, not decoded (BitReader:140, AdapterHelper:37). They
> come from UUID.nameUUIDFromBytes. As a result:
> - Encrypted sections are never detected; the check at
> MSOneStoreParser:328 can't match.
> - Embedded relationship IDs contain hashed GUIDs.
> - Every cell mapping copies its whole revision chain
> (MSOneStoreParser:362), so the cost grows with mappings × objects.
> - Slow spots:
> - removeSupersededObjects is O(n²) (:393).
> - resourceInfoFromReferencingObject scans every object for each unnamed
> blob (MSOneStorePackage:1140).
> - hasUsablePicture copies a whole blob just to check it's non-empty
> (:1118).
> - Small port bugs:
> - cProperties is read as signed, so values ≥ 0x8000 abort the parse.
> - An undefined property type throws an NPE.
> - StorageManifestSchemaGUID reads from offset 0 instead of the current
> position.
> - BitConverter's bounds check ignores the index.
> Dead code:
> - The serialize/write path: 65 files, about 970 lines.
> - The FSSHTTPD chunking/build side.
> - getContent() and its builders.
> - Unreferenced classes: Error, PropertyIDType, RequestTypes, UMath.
> - Debug print() methods and option branches that are never set.
> {noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
