[
https://issues.apache.org/jira/browse/TIKA-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138153#comment-18138153
]
ASF GitHub Bot commented on TIKA-5026:
--------------------------------------
tballison commented on PR #3520:
URL: https://github.com/apache/tika/pull/3520#issuecomment-6639552072
A couple of important finds and some that are meh? Let me know what you
think.
```
Contract (fix first)
1. A hybrid file's book text comes out twice. Three reviewers found this
independently.
- Most kindlegen .mobi files are hybrids, carrying the same book in
both halves. The second (KF8) half is emitted
as an embedded document with full text, so /tika and text output
contain the book twice, /rmeta has two
near-duplicate documents, and the work is done twice.
- It's marked ALTERNATE_FORMAT_CHUNK, which in Tika means OOXML
altChunk: content that is part of the document,
not a second copy.
- The test fixture hides this because its two halves have different
text.
- Suggested shape: parse the first half by default and make the KF8
half opt-in, the way
RFC822Parser.extractAllAlternatives works.
2. The hybrid rebuild gets the record numbering backwards. I checked this
against KindleUnpack's source.
- In a real hybrid, the second header's record-index fields count from
the KF8 record 0; KindleUnpack adds that
record's index to each of them. The PR assumes they count from the
start of the file and subtracts.
- Most fields therefore become "none". A hybrid whose second half uses
HUFF/CDIC compression would lose its text.
- The generator makes the same assumption, so the test confirms the bug.
- The PR says it matched a real 19-file sample. PalmDOC hybrids hide
the bug because nothing they need reads
those fields yet.
3. KF8 text includes the CSS and SVG. The parser ignores the FDST record,
which marks where each content flow starts
and ends. Raw stylesheet rules after </html> end up as book text. The
fix depends on #2.
4. Encrypted books never set tika:encrypted. Recording a warning instead
of throwing (the settled choice) skips the
code that normally sets the flag. Setting it explicitly fixes this.
5. TikaCLITest.testSupportedFormatsPartialIsCurrent will fail. I confirmed
the generated
docs/.../partials/supported-formats.adoc has no mobi entry; it needs
regenerating.
Edge cases
6. Small files can force huge HTML parses. With HUFF/CDIC, a file of about
45 KB can decode to the 64 MB text cap,
and jsoup then builds roughly a 2 GB tree in memory. Two things make it
worse:
- The HTML pass writes metadata into new Metadata() instead of
Metadata.newInstance(context), so repeated <meta>
tags skip the write limiter and hit the quadratic Metadata.add path.
The estimate is about an hour of CPU,
extrapolated from component timings, not run end to end.
- Fix: use newInstance(context) and lower the default cap, or tie it to
the expansion ratio.
7. The number of EXTH metadata records is unbounded. Header metadata
entries are limited only by record 0's size,
and each value is added to metadata (quadratic). Roughly 10 minutes of
CPU for a 10 MB file, and out-of-memory
near the 256 MB cap. Cap the record count and the number of values per
key.
8. A hybrid holds about 4–5 copies of the book in memory at once, and
builds them before asking whether embedded
documents should be parsed at all. Ask first, then build the rebuilt
file straight from slices.
9. One bad HUFF code loses all the text decoded so far, even from records
that decoded fine. Four reviewers flagged
it. It contradicts the PR's own "read what's there" policy for
truncated and otherwise broken books.
10. Some .prc/.azw files go from empty success to an exception. A Palm app
or PalmDOC .prc, or a Topaz or KFX .azw
(format details unverified), is routed here by file extension alone
and now throws. Return metadata only, or
document it. Plain PalmDOC books would be a cheap follow-up, since the
decompressor is already there.
11. .azw files now detect as vnd.amazon.ebook instead of
x-mobipocket-ebook. This affects anyone routing on the
type. It needs a CHANGES line and a detection test (BOOKMOBI bytes
plus a .azw/.azw3 name).
12. The 256 MB cap is hard-coded and undocumented, and a book over it
returns no metadata at all. Comic and manga
books can plausibly be that large (unverified).
13. Dates: EXTH 106 dates are parsed by hand instead of with TikaDates.
Date-only values gain an invented midnight
UTC time, and non-ISO dates are silently dropped.
14. Unknown or zero codepage falls back to UTF-8. KindleUnpack uses
windows-1252, so old .prc files would get
garbled accented characters.
15. Test gaps:
- The 256 MB cap has no test; a sparse file would cover it without a
256 MB fixture.
- The fuzz test never feeds mutated input into the hybrid's KF8 half
and has no @Timeout.
- Two tests suppress exceptions without asserting there wasn't one.
- Untested: EXTH 524 taking precedence over the header locale, stored
(uncompressed) text, the title fallback,
and a positive cde-type.
```
> Parse Mobipocket and Kindle KF8 ebooks (metadata, text, cover)
> --------------------------------------------------------------
>
> Key: TIKA-5026
> URL: https://issues.apache.org/jira/browse/TIKA-5026
> Project: Tika
> Issue Type: New Feature
> Reporter: Dominik Schmidt
> Priority: Major
> Attachments: mobi-5026.zip
>
>
> Tika detects application/x-mobipocket-ebook (BOOKMOBI magic at offset 60,
> globs *.prc and *.mobi) and application/vnd.amazon.ebook (glob *.azw only),
> but no parser module handles either: /rmeta on a 9.6 MB .azw3 returns the
> Content-Type and nothing else, /unpack/preset/thumbnail returns 204, while
> the EPUB of the same book yields title, author, text and the cover marked
> THUMBNAIL. The KF8 type application/vnd.amazon.mobi8-ebook (IANA, glob *.azw3
> in shared-mime-info) is missing from tika-mimetypes.xml altogether.
> The container is the same for all three: a Palm Database (PDB) whose record 0
> holds the PalmDOC header, the MOBI header and the EXTH metadata records. A
> parser next to EpubParser could do, in this order:
> 1. Metadata from EXTH: 100 creator, 101 publisher, 103 description, 104 ISBN,
> 105 subject, 106 publishing date, 108 contributor, 503 title (falls back to
> the full name in the MOBI header), 524 language; plus the MOBI header's
> locale and encoding (CP1252 or UTF-8). Hybrid files carry EXTH 121 (KF8
> boundary) and a second record 0 for the KF8 part with its own EXTH.
> 2. Cover: EXTH 201 (CoverOffset) and 202 (ThumbOffset) index into the image
> records starting at the MOBI header's first image index; the records are
> plain JPEG/GIF/PNG. Emit the cover as an embedded resource with
> tk:embedded-resource-type THUMBNAIL (TIKA-4856 marker), the thumbnail record
> as a second one, so /unpack/preset/thumbnail serves the cover like it does
> for EPUB.
> 3. Text: MOBI7 records with PalmDOC LZ77 (compression 2) or none (1),
> trailing entries stripped per the MOBI header's extra data flags, HTML handed
> to the HTML parser. HUFF/CDIC (17480) and the KF8 skeleton/fragment text
> format are more work and can be a follow-up; metadata and cover are the same
> for both generations and cover the common search and preview use cases.
> DRM (MOBI header encryption type != 0, EXTH 208/209): record 0, EXTH and the
> image records are not encrypted, so metadata and cover still come out; the
> text does not and should raise EncryptedDocumentException rather than emit
> garbage.
> Detection: add application/vnd.amazon.mobi8-ebook as a sub-type of
> application/x-mobipocket-ebook with glob *.azw3 (the magic is identical, the
> generation is only visible from the MOBI header version and EXTH 121, so
> content detection stays at the parent), and give application/vnd.amazon.ebook
> the same parent so one parser claims all three.
> Test files: Project Gutenberg offers every title as .mobi/.kf8 (public
> domain), Calibre converts EPUB to both .mobi and .azw3 for fixtures with
> known metadata and cover.
> Related: TIKA-1892 (the magic), TIKA-4856 (thumbnail marker), TIKA-4988
> (thumbnail preset).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
