[
https://issues.apache.org/jira/browse/TIKA-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137376#comment-18137376
]
Dominik Schmidt commented on TIKA-5026:
---------------------------------------
Thanks for the sample. I ran all 19 files: cover presence, text and metadata
match your index for every one, including the HUFF/CDIC books, the KF8-only and
hybrid ones, versions 4 to 8, and the two DRM books (title and cover, the
locked text as a warning). The 1 MiB truncation found a gap: the record table
lists records past the end of the file and the parser refused the whole book.
As of 5510d0443c it reads the records that are there (title, 430k characters of
text) and warns "529 of 3670 records are there". The cover of that file sits
behind the cut, so there is none to extract.
> Parse Mobipocket and Kindle KF8 ebooks (metadata, text, cover)
> --------------------------------------------------------------
>
> Key: TIKA-5026
> URL: https://issues.apache.org/jira/browse/TIKA-5026
> Project: Tika
> Issue Type: New Feature
> Reporter: Dominik Schmidt
> Priority: Major
> Attachments: mobi-5026.zip
>
>
> Tika detects application/x-mobipocket-ebook (BOOKMOBI magic at offset 60,
> globs *.prc and *.mobi) and application/vnd.amazon.ebook (glob *.azw only),
> but no parser module handles either: /rmeta on a 9.6 MB .azw3 returns the
> Content-Type and nothing else, /unpack/preset/thumbnail returns 204, while
> the EPUB of the same book yields title, author, text and the cover marked
> THUMBNAIL. The KF8 type application/vnd.amazon.mobi8-ebook (IANA, glob *.azw3
> in shared-mime-info) is missing from tika-mimetypes.xml altogether.
> The container is the same for all three: a Palm Database (PDB) whose record 0
> holds the PalmDOC header, the MOBI header and the EXTH metadata records. A
> parser next to EpubParser could do, in this order:
> 1. Metadata from EXTH: 100 creator, 101 publisher, 103 description, 104 ISBN,
> 105 subject, 106 publishing date, 108 contributor, 503 title (falls back to
> the full name in the MOBI header), 524 language; plus the MOBI header's
> locale and encoding (CP1252 or UTF-8). Hybrid files carry EXTH 121 (KF8
> boundary) and a second record 0 for the KF8 part with its own EXTH.
> 2. Cover: EXTH 201 (CoverOffset) and 202 (ThumbOffset) index into the image
> records starting at the MOBI header's first image index; the records are
> plain JPEG/GIF/PNG. Emit the cover as an embedded resource with
> tk:embedded-resource-type THUMBNAIL (TIKA-4856 marker), the thumbnail record
> as a second one, so /unpack/preset/thumbnail serves the cover like it does
> for EPUB.
> 3. Text: MOBI7 records with PalmDOC LZ77 (compression 2) or none (1),
> trailing entries stripped per the MOBI header's extra data flags, HTML handed
> to the HTML parser. HUFF/CDIC (17480) and the KF8 skeleton/fragment text
> format are more work and can be a follow-up; metadata and cover are the same
> for both generations and cover the common search and preview use cases.
> DRM (MOBI header encryption type != 0, EXTH 208/209): record 0, EXTH and the
> image records are not encrypted, so metadata and cover still come out; the
> text does not and should raise EncryptedDocumentException rather than emit
> garbage.
> Detection: add application/vnd.amazon.mobi8-ebook as a sub-type of
> application/x-mobipocket-ebook with glob *.azw3 (the magic is identical, the
> generation is only visible from the MOBI header version and EXTH 121, so
> content detection stays at the parent), and give application/vnd.amazon.ebook
> the same parent so one parser claims all three.
> Test files: Project Gutenberg offers every title as .mobi/.kf8 (public
> domain), Calibre converts EPUB to both .mobi and .azw3 for fixtures with
> known metadata and cover.
> Related: TIKA-1892 (the magic), TIKA-4856 (thumbnail marker), TIKA-4988
> (thumbnail preset).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
