Dominik Schmidt created TIKA-5026:
-------------------------------------
Summary: Parse Mobipocket and Kindle KF8 ebooks (metadata, text,
cover)
Key: TIKA-5026
URL: https://issues.apache.org/jira/browse/TIKA-5026
Project: Tika
Issue Type: New Feature
Reporter: Dominik Schmidt
Tika detects application/x-mobipocket-ebook (BOOKMOBI magic at offset 60, globs
*.prc and *.mobi) and application/vnd.amazon.ebook (glob *.azw only), but no
parser module handles either: /rmeta on a 9.6 MB .azw3 returns the Content-Type
and nothing else, /unpack/preset/thumbnail returns 204, while the EPUB of the
same book yields title, author, text and the cover marked THUMBNAIL. The KF8
type application/vnd.amazon.mobi8-ebook (IANA, glob *.azw3 in shared-mime-info)
is missing from tika-mimetypes.xml altogether.
The container is the same for all three: a Palm Database (PDB) whose record 0
holds the PalmDOC header, the MOBI header and the EXTH metadata records. A
parser next to EpubParser could do, in this order:
1. Metadata from EXTH: 100 creator, 101 publisher, 103 description, 104 ISBN,
105 subject, 106 publishing date, 108 contributor, 503 title (falls back to the
full name in the MOBI header), 524 language; plus the MOBI header's locale and
encoding (CP1252 or UTF-8). Hybrid files carry EXTH 121 (KF8 boundary) and a
second record 0 for the KF8 part with its own EXTH.
2. Cover: EXTH 201 (CoverOffset) and 202 (ThumbOffset) index into the image
records starting at the MOBI header's first image index; the records are plain
JPEG/GIF/PNG. Emit the cover as an embedded resource with
tk:embedded-resource-type THUMBNAIL (TIKA-4856 marker), the thumbnail record as
a second one, so /unpack/preset/thumbnail serves the cover like it does for
EPUB.
3. Text: MOBI7 records with PalmDOC LZ77 (compression 2) or none (1), trailing
entries stripped per the MOBI header's extra data flags, HTML handed to the
HTML parser. HUFF/CDIC (17480) and the KF8 skeleton/fragment text format are
more work and can be a follow-up; metadata and cover are the same for both
generations and cover the common search and preview use cases.
DRM (MOBI header encryption type != 0, EXTH 208/209): record 0, EXTH and the
image records are not encrypted, so metadata and cover still come out; the text
does not and should raise EncryptedDocumentException rather than emit garbage.
Detection: add application/vnd.amazon.mobi8-ebook as a sub-type of
application/x-mobipocket-ebook with glob *.azw3 (the magic is identical, the
generation is only visible from the MOBI header version and EXTH 121, so
content detection stays at the parent), and give application/vnd.amazon.ebook
the same parent so one parser claims all three.
Test files: Project Gutenberg offers every title as .mobi/.kf8 (public domain),
Calibre converts EPUB to both .mobi and .azw3 for fixtures with known metadata
and cover.
Related: TIKA-1892 (the magic), TIKA-4856 (thumbnail marker), TIKA-4988
(thumbnail preset).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
