[ https://issues.apache.org/jira/browse/TIKA-5057 ]
Tim Allison deleted comment on TIKA-5057:
-----------------------------------
was (Author: [email protected]):
Notes on work to date from the bot:
h3. Status
The review and refactor of the OneNote parser is in PR #3564. Summary of where
it stands:
* *Current text only.* .one sections give each page's current revision, not
every earlier revision and version-history snapshot. Metadata and attachments
still come from all revisions. New {{one-note-parser}} config, both default
false:
** {{includePageHistory}}: earlier revisions and saved page versions.
** {{includeDeletedContent}}: pages moved to the recycle bin; FSSHTTPB sections
only.
* *Correctness:*
** Objects resolve within their own revision, which stops old and new text
being mixed.
** File node lists end at the transaction log's committed count, which recovers
text that 4.2.0 misses.
** Text cleanup: windows-1252 extended ASCII; no NULs or U+FFFC placeholders.
* *New output:*
** Pictures as {{<img>}}, OneNote's OCR text, handwriting recognition and
hyperlinks.
** OneNote 2007 paragraph text, and table-of-contents sections.
** Encrypted sections fail with EncryptedDocumentException.
* *Limits:* bounded file node lists, walk visits and depth, and a document-wide
budget on property values that scales with file size.
h3. Performance and GC, compared with 4.2.0
This is the benchmark from TIKA-5055: every OneNote test file in
tika-parser-microsoft-module (17 files), parsed 1,000 times each. Allocation
and GC counts are per run and summed over the files.
||Build||Allocated per parse (MiB)||GC count||GC time||
|4.2.0|1,029|6,314|12.7 s|
|main (with TIKA-5055)|1,029|6,639|12.3 s|
|this PR|44|336|0.3 s|
* Most of the allocation came from {{OneNoteDirectFileResource}}. Every read of
a few bytes allocated a new 8 KB transfer buffer and made at least one system
call. Reads now come from a 64 KB window of the file.
* The tree walker no longer builds a nested map of every node and property that
nothing read, and {{ExtendedGUID.hashCode()}} no longer allocates.
Large sections with many revisions still cost more than in 4.2.0. 4.2.0 stops
reading their file node lists early: on a 41 MB section it sees 31 revisions
and 1,590 objects, while this PR reads all 185 revisions and 56,333 objects. On
the four largest corpus files (18–41 MB), the PR allocates 0.9–2.9 times what
4.2.0 does (two of the four allocate less), and GC time is 7–16 times higher,
at 150–310 ms per parse. Lowering that would mean releasing deserialized
objects during the parse. That is a structural change, better done as a
follow-up.
h3. Testing
* *Corpus:* 913 unique files: 804 sections and 109 table-of-contents files.
** Sources: Common Crawl, open-source repositories and test fixtures, and
sections unpacked from .onepkg packages.
** Sections by format: 709 OneNote 2010–2016, 57 FSSHTTPB, 31 OneNote 2007, 6
OneNote 2003, and 1 truncated.
** 877 parse with content. The rest are 13 encrypted, 12 with no content, 6
OneNote 2003 files that fall back to the legacy text dump, and a few broken or
synthetic files.
* *Replay:* every corpus file through the tika-fuzzing harness (TIKA-5060) in
bytes, stream and path modes: 2,739 parses, no hangs, OOMs, crashes or
unexpected exceptions, including in strict mode.
* *Fuzzing:* a coverage-guided Jazzer session seeded with OneNote files is
running. Results to follow.
h3. Follow-ups
* OneNote 2007 inline pictures
* Handwriting recognition in FSSHTTPB sections
* Linking .one pictures to their attachment data
* Marking page-history text in .one output
* Checking whether .one sections keep deleted pages
* A small test file with a deleted page
* Lower memory for large multi-revision sections
> General improvements to onenote parser
> --------------------------------------
>
> Key: TIKA-5057
> URL: https://issues.apache.org/jira/browse/TIKA-5057
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Trivial
>
> When I have extra tokens, I'll have an agent focus on a parser at a time for
> robustness and simplicity. Let's start with onenote. CC [~henrylindemanglean]
> :robot: swarm recommends:
> {noformat}
> Native .one path (the common format, unrelated to TIKA-5055):
> 1. Exponential parse on shared child lists (OneNotePtr:714). A file-node
> list referenced twice is parsed twice. Only the current path is checked for
> cycles, so a crafted file of a few KB blows up as 2^depth. Three reviewers
> found this independently.
> 2. Exponential tree walk (OneNoteTreeWalker:648). The same shape: shared
> objects are walked once per reference path. Each visit also builds and keeps
> a HashMap of decoded properties, which OneNoteParser:144 throws away.
> 3. Wrong references, lost text (OneNotePtr:1185). The OID/OSID/ContextID
> stream position is copied to a local variable and never written back. Every
> reference property starts again at stream[0], so later references are never
> walked. It also removes the
> cap on reference counts.
> 4. Duplicate text in real files (OneNoteTreeWalker:569-592). Reproduced:
> testOneNote2.one produces 176 paragraphs, 105 of them "Untitled picture.png".
> Image filenames and TextExtendedAscii text are never deduplicated and ignore
> the revision filter.
> 5. Unchecked counts and budgets (OneNotePtr:838, :518, :809):
> - ObjectInfoDependencyOverrideData counts are unbounded.
> - The global-ID cap applies per node, with no total.
> - A negative FileDataStoreObject length gets through.
> 6. Smaller issues:
> - Embedded file data is read in full before shouldParseEmbedded is asked
> (OneNoteTreeWalker:435).
> - The UTF-16 NUL terminator stays in author names and image filenames.
> Legacy string dump, used whenever a parse falls back
> (OneNoteLegacyDumpStrings:95):
> - Never resets its buffer at 1 MB boundaries. A long run is emitted again
> at every boundary, so work grows quadratically and normal files over 1 MB get
> duplicated text.
> - Small bugs: it drops the last byte of each buffer,
> Character.isWhitespace(i) tests the loop index instead of the character, and
> split(" ") allocates one String per word.
> Fallback after a partial walk (OneNoteParser:172, :197): on both paths, the
> full legacy dump is added after whatever the failed walk already wrote, so
> content appears twice.
> FSSHTTPB:
> - GUIDs are MD5 hashes, not decoded (BitReader:140, AdapterHelper:37). They
> come from UUID.nameUUIDFromBytes. As a result:
> - Encrypted sections are never detected; the check at
> MSOneStoreParser:328 can't match.
> - Embedded relationship IDs contain hashed GUIDs.
> - Every cell mapping copies its whole revision chain
> (MSOneStoreParser:362), so the cost grows with mappings × objects.
> - Slow spots:
> - removeSupersededObjects is O(n²) (:393).
> - resourceInfoFromReferencingObject scans every object for each unnamed
> blob (MSOneStorePackage:1140).
> - hasUsablePicture copies a whole blob just to check it's non-empty
> (:1118).
> - Small port bugs:
> - cProperties is read as signed, so values ≥ 0x8000 abort the parse.
> - An undefined property type throws an NPE.
> - StorageManifestSchemaGUID reads from offset 0 instead of the current
> position.
> - BitConverter's bounds check ignores the index.
> Dead code:
> - The serialize/write path: 65 files, about 970 lines.
> - The FSSHTTPD chunking/build side.
> - getContent() and its builders.
> - Unreferenced classes: Error, PropertyIDType, RequestTypes, UMath.
> - Debug print() methods and option branches that are never set.
> {noformat}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
