dschmidt opened a new pull request, #3520: URL: https://github.com/apache/tika/pull/3520
Tika detects Mobipocket books (`application/x-mobipocket-ebook`, by the `BOOKMOBI` magic) and Kindle `.azw` files, but no parser handles them: `/rmeta` returns the content type and nothing else, while the EPUB of the same book yields title, author, text and the cover. The KF8 generation (`.azw3`) is not even in `tika-mimetypes.xml`. This adds `MobiParser` next to `EpubParser` in the miscoffice module. All three formats share one container, a Palm Database whose record 0 holds the PalmDOC header, the MOBI header and the EXTH metadata records, followed by the text records and the images; KF8 only has a longer MOBI header and a different text layout, so one parser serves them all: - **Metadata** from EXTH: title (503, else the full name), creators (100), publisher (101), description (103), identifiers (104 ISBN, 113/504 ASIN, deduplicated), subjects (105), publishing date (106, ISO date with or without time), contributor (108), rights (109), language (524). The content type is set from the header's file version: 8 and up is `application/vnd.amazon.mobi8-ebook`. - **Images**: every image record from the header's first image index on, recognised by its JPEG/PNG/GIF/BMP magic (index, font and boundary records sit among them and are skipped). The record EXTH 201 points at is emitted as the `THUMBNAIL` embedded document named `cover.<ext>`, the EXTH 202 thumbnail and the other images as `INLINE`. In a hybrid file (EXTH 121) the images are read up to the KF8 boundary; the two halves share them. - **Text**: the text records (PalmDOC-compressed, type 2, or stored, type 1) with their trailing entries stripped per the header's flags at 0xF2, concatenated and decoded once (a multibyte character may straddle two records; codepage 1252 or 65001), then handed to `JSoupParser.parseString`. KF8 text is taken as the records hold it, skeletons and fragments in order, without reassembly; the text comes out in reading order, the markup is not what a reader would render. HUFF/CDIC (17480) is not decoded; the parser records a warning and extracts metadata and images only. - **Encrypted books** (encryption type != 0): record 0, EXTH and the image records are not encrypted, so metadata and cover still come out; then the parser throws `EncryptedDocumentException` instead of emitting the encrypted bytes. Detection: `application/vnd.amazon.mobi8-ebook` with glob `*.azw3` and `application/vnd.amazon.ebook` (`*.azw`) both become sub-types of `application/x-mobipocket-ebook`, so a file whose name says `.azw3` and whose content says `BOOKMOBI` resolves to the specific type and reaches the parser either way. The generation is only visible from the header version, which the parser reports. The miscoffice module gains a dependency on `tika-parser-html-module` (as the mail and microsoft modules have) for the text. Tests: four fixtures of about 3 KB each, built by a script rather than a reader application and described in the test's javadoc: a version 6 book in CP1252 with two PalmDOC-compressed text records, trailing entries, a JPEG cover, a PNG thumbnail, an inline GIF and FLIS/FCIS/EOF records; the same as a version 8 book in UTF-8 with KF8-style markup; an encrypted and a HUFF/CDIC variant. `MobiParserTest` covers metadata, text, the embedded image order and types, KF8, the encrypted and HUFF/CDIC behaviour, date parsing and the trailing-entry arithmetic; `TikaDetectionTest` gets `x.azw3`. Also run against Project Gutenberg's Frankenstein as `.mobi` (version 6) and `.azw3` (version 8): metadata, cover and 450k characters of text each, and against an encrypted retail KF8 book: metadata and cover, no text. Docs: formats page, CHANGES. Follow-ups, out of scope here: HUFF/CDIC decoding, KF8 skeleton/fragment reassembly, the KF8 half of a hybrid file, and MOBI-specific metadata keys (header version, encryption type) once there is a place for them in the schema. https://issues.apache.org/jira/browse/TIKA-5026 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
