[
https://issues.apache.org/jira/browse/TIKA-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18139108#comment-18139108
]
Hudson commented on TIKA-5026:
------------------------------
SUCCESS: Integrated in Jenkins build Tika » tika-main-jdk17 #1776 (See
[https://ci-builds.apache.org/job/Tika/job/tika-main-jdk17/1776/])
TIKA-5026: Parse Mobipocket and Kindle ebooks (.mobi, .prc, .azw, .azw3)
(#3520) (github:
[https://github.com/apache/tika/commit/67645bad6b67302b10d2090d6df26051939efdbc])
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/resources/test-documents/testMOBI_encrypted.mobi
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/resources/test-documents/testMOBI.mobi
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/resources/test-documents/testMOBI_unknown_compression.mobi
* (edit) tika-core/src/test/java/org/apache/tika/TikaDetectionTest.java
* (add) tika-core/src/main/java/org/apache/tika/metadata/Mobi.java
* (edit)
tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-key-fields.json
* (edit) tika-core/src/main/resources/org/apache/tika/mime/tika-mimetypes.xml
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/main/java/org/apache/tika/parser/mobi/HuffCdic.java
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/resources/test-documents/testMOBI_huffcdic.mobi
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/python/make_mobi_fixtures.py
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/resources/test-documents/testMOBI_hybrid.mobi
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/main/java/org/apache/tika/parser/mobi/PalmDoc.java
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/main/java/org/apache/tika/parser/mobi/MobiHeader.java
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/java/org/apache/tika/parser/mobi/MobiParserTest.java
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/resources/test-documents/testMOBI_stored.mobi
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/resources/test-documents/testMOBI_kf8.azw3
* (edit) docs/modules/ROOT/pages/using-tika/server/index.adoc
* (edit) docs/modules/ROOT/pages/formats.adoc
* (edit)
tika-metadata-schema/src/main/resources/org/apache/tika/metadata/metadata-keys.json
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/main/java/org/apache/tika/parser/mobi/MobiParser.java
* (edit) CHANGES.txt
* (edit)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/pom.xml
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/main/java/org/apache/tika/parser/mobi/PalmDatabase.java
* (add)
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-miscoffice-module/src/test/resources/test-documents/testMOBI_huffcdic_broken.mobi
* (edit) docs/modules/ROOT/partials/supported-formats.adoc
> Parse Mobipocket and Kindle KF8 ebooks (metadata, text, cover)
> --------------------------------------------------------------
>
> Key: TIKA-5026
> URL: https://issues.apache.org/jira/browse/TIKA-5026
> Project: Tika
> Issue Type: New Feature
> Reporter: Dominik Schmidt
> Priority: Major
> Attachments: mobi-5026.zip
>
>
> Tika detects application/x-mobipocket-ebook (BOOKMOBI magic at offset 60,
> globs *.prc and *.mobi) and application/vnd.amazon.ebook (glob *.azw only),
> but no parser module handles either: /rmeta on a 9.6 MB .azw3 returns the
> Content-Type and nothing else, /unpack/preset/thumbnail returns 204, while
> the EPUB of the same book yields title, author, text and the cover marked
> THUMBNAIL. The KF8 type application/vnd.amazon.mobi8-ebook (IANA, glob *.azw3
> in shared-mime-info) is missing from tika-mimetypes.xml altogether.
> The container is the same for all three: a Palm Database (PDB) whose record 0
> holds the PalmDOC header, the MOBI header and the EXTH metadata records. A
> parser next to EpubParser could do, in this order:
> 1. Metadata from EXTH: 100 creator, 101 publisher, 103 description, 104 ISBN,
> 105 subject, 106 publishing date, 108 contributor, 503 title (falls back to
> the full name in the MOBI header), 524 language; plus the MOBI header's
> locale and encoding (CP1252 or UTF-8). Hybrid files carry EXTH 121 (KF8
> boundary) and a second record 0 for the KF8 part with its own EXTH.
> 2. Cover: EXTH 201 (CoverOffset) and 202 (ThumbOffset) index into the image
> records starting at the MOBI header's first image index; the records are
> plain JPEG/GIF/PNG. Emit the cover as an embedded resource with
> tk:embedded-resource-type THUMBNAIL (TIKA-4856 marker), the thumbnail record
> as a second one, so /unpack/preset/thumbnail serves the cover like it does
> for EPUB.
> 3. Text: MOBI7 records with PalmDOC LZ77 (compression 2) or none (1),
> trailing entries stripped per the MOBI header's extra data flags, HTML handed
> to the HTML parser. HUFF/CDIC (17480) and the KF8 skeleton/fragment text
> format are more work and can be a follow-up; metadata and cover are the same
> for both generations and cover the common search and preview use cases.
> DRM (MOBI header encryption type != 0, EXTH 208/209): record 0, EXTH and the
> image records are not encrypted, so metadata and cover still come out; the
> text does not and should raise EncryptedDocumentException rather than emit
> garbage.
> Detection: add application/vnd.amazon.mobi8-ebook as a sub-type of
> application/x-mobipocket-ebook with glob *.azw3 (the magic is identical, the
> generation is only visible from the MOBI header version and EXTH 121, so
> content detection stays at the parent), and give application/vnd.amazon.ebook
> the same parent so one parser claims all three.
> Test files: Project Gutenberg offers every title as .mobi/.kf8 (public
> domain), Calibre converts EPUB to both .mobi and .azw3 for fixtures with
> known metadata and cover.
> Related: TIKA-1892 (the magic), TIKA-4856 (thumbnail marker), TIKA-4988
> (thumbnail preset).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
