[
https://issues.apache.org/jira/browse/TIKA-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18137189#comment-18137189
]
ASF GitHub Bot commented on TIKA-5026:
--------------------------------------
dschmidt opened a new pull request, #3520:
URL: https://github.com/apache/tika/pull/3520
Tika detects Mobipocket books (`application/x-mobipocket-ebook`, by the
`BOOKMOBI` magic) and Kindle `.azw` files, but no parser handles them: `/rmeta`
returns the content type and nothing else, while the EPUB of the same book
yields title, author, text and the cover. The KF8 generation (`.azw3`) is not
even in `tika-mimetypes.xml`.
This adds `MobiParser` next to `EpubParser` in the miscoffice module. All
three formats share one container, a Palm Database whose record 0 holds the
PalmDOC header, the MOBI header and the EXTH metadata records, followed by the
text records and the images; KF8 only has a longer MOBI header and a different
text layout, so one parser serves them all:
- **Metadata** from EXTH: title (503, else the full name), creators (100),
publisher (101), description (103), identifiers (104 ISBN, 113/504 ASIN,
deduplicated), subjects (105), publishing date (106, ISO date with or without
time), contributor (108), rights (109), language (524). The content type is set
from the header's file version: 8 and up is
`application/vnd.amazon.mobi8-ebook`.
- **Images**: every image record from the header's first image index on,
recognised by its JPEG/PNG/GIF/BMP magic (index, font and boundary records sit
among them and are skipped). The record EXTH 201 points at is emitted as the
`THUMBNAIL` embedded document named `cover.<ext>`, the EXTH 202 thumbnail and
the other images as `INLINE`. In a hybrid file (EXTH 121) the images are read
up to the KF8 boundary; the two halves share them.
- **Text**: the text records (PalmDOC-compressed, type 2, or stored, type 1)
with their trailing entries stripped per the header's flags at 0xF2,
concatenated and decoded once (a multibyte character may straddle two records;
codepage 1252 or 65001), then handed to `JSoupParser.parseString`. KF8 text is
taken as the records hold it, skeletons and fragments in order, without
reassembly; the text comes out in reading order, the markup is not what a
reader would render. HUFF/CDIC (17480) is not decoded; the parser records a
warning and extracts metadata and images only.
- **Encrypted books** (encryption type != 0): record 0, EXTH and the image
records are not encrypted, so metadata and cover still come out; then the
parser throws `EncryptedDocumentException` instead of emitting the encrypted
bytes.
Detection: `application/vnd.amazon.mobi8-ebook` with glob `*.azw3` and
`application/vnd.amazon.ebook` (`*.azw`) both become sub-types of
`application/x-mobipocket-ebook`, so a file whose name says `.azw3` and whose
content says `BOOKMOBI` resolves to the specific type and reaches the parser
either way. The generation is only visible from the header version, which the
parser reports.
The miscoffice module gains a dependency on `tika-parser-html-module` (as
the mail and microsoft modules have) for the text.
Tests: four fixtures of about 3 KB each, built by a script rather than a
reader application and described in the test's javadoc: a version 6 book in
CP1252 with two PalmDOC-compressed text records, trailing entries, a JPEG
cover, a PNG thumbnail, an inline GIF and FLIS/FCIS/EOF records; the same as a
version 8 book in UTF-8 with KF8-style markup; an encrypted and a HUFF/CDIC
variant. `MobiParserTest` covers metadata, text, the embedded image order and
types, KF8, the encrypted and HUFF/CDIC behaviour, date parsing and the
trailing-entry arithmetic; `TikaDetectionTest` gets `x.azw3`. Also run against
Project Gutenberg's Frankenstein as `.mobi` (version 6) and `.azw3` (version
8): metadata, cover and 450k characters of text each, and against an encrypted
retail KF8 book: metadata and cover, no text.
Docs: formats page, CHANGES.
Follow-ups, out of scope here: HUFF/CDIC decoding, KF8 skeleton/fragment
reassembly, the KF8 half of a hybrid file, and MOBI-specific metadata keys
(header version, encryption type) once there is a place for them in the schema.
https://issues.apache.org/jira/browse/TIKA-5026
> Parse Mobipocket and Kindle KF8 ebooks (metadata, text, cover)
> --------------------------------------------------------------
>
> Key: TIKA-5026
> URL: https://issues.apache.org/jira/browse/TIKA-5026
> Project: Tika
> Issue Type: New Feature
> Reporter: Dominik Schmidt
> Priority: Major
>
> Tika detects application/x-mobipocket-ebook (BOOKMOBI magic at offset 60,
> globs *.prc and *.mobi) and application/vnd.amazon.ebook (glob *.azw only),
> but no parser module handles either: /rmeta on a 9.6 MB .azw3 returns the
> Content-Type and nothing else, /unpack/preset/thumbnail returns 204, while
> the EPUB of the same book yields title, author, text and the cover marked
> THUMBNAIL. The KF8 type application/vnd.amazon.mobi8-ebook (IANA, glob *.azw3
> in shared-mime-info) is missing from tika-mimetypes.xml altogether.
> The container is the same for all three: a Palm Database (PDB) whose record 0
> holds the PalmDOC header, the MOBI header and the EXTH metadata records. A
> parser next to EpubParser could do, in this order:
> 1. Metadata from EXTH: 100 creator, 101 publisher, 103 description, 104 ISBN,
> 105 subject, 106 publishing date, 108 contributor, 503 title (falls back to
> the full name in the MOBI header), 524 language; plus the MOBI header's
> locale and encoding (CP1252 or UTF-8). Hybrid files carry EXTH 121 (KF8
> boundary) and a second record 0 for the KF8 part with its own EXTH.
> 2. Cover: EXTH 201 (CoverOffset) and 202 (ThumbOffset) index into the image
> records starting at the MOBI header's first image index; the records are
> plain JPEG/GIF/PNG. Emit the cover as an embedded resource with
> tk:embedded-resource-type THUMBNAIL (TIKA-4856 marker), the thumbnail record
> as a second one, so /unpack/preset/thumbnail serves the cover like it does
> for EPUB.
> 3. Text: MOBI7 records with PalmDOC LZ77 (compression 2) or none (1),
> trailing entries stripped per the MOBI header's extra data flags, HTML handed
> to the HTML parser. HUFF/CDIC (17480) and the KF8 skeleton/fragment text
> format are more work and can be a follow-up; metadata and cover are the same
> for both generations and cover the common search and preview use cases.
> DRM (MOBI header encryption type != 0, EXTH 208/209): record 0, EXTH and the
> image records are not encrypted, so metadata and cover still come out; the
> text does not and should raise EncryptedDocumentException rather than emit
> garbage.
> Detection: add application/vnd.amazon.mobi8-ebook as a sub-type of
> application/x-mobipocket-ebook with glob *.azw3 (the magic is identical, the
> generation is only visible from the MOBI header version and EXTH 121, so
> content detection stays at the parent), and give application/vnd.amazon.ebook
> the same parent so one parser claims all three.
> Test files: Project Gutenberg offers every title as .mobi/.kf8 (public
> domain), Calibre converts EPUB to both .mobi and .azw3 for fixtures with
> known metadata and cover.
> Related: TIKA-1892 (the magic), TIKA-4856 (thumbnail marker), TIKA-4988
> (thumbnail preset).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
