[
https://issues.apache.org/jira/browse/TIKA-5026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18138179#comment-18138179
]
ASF GitHub Bot commented on TIKA-5026:
--------------------------------------
dschmidt commented on PR #3520:
URL: https://github.com/apache/tika/pull/3520#issuecomment-6640469435
Thanks, all fifteen addressed in b9ad505692.
1. A hybrid yields its version 6 book only; `extractKf8` (off by default,
like `extractAllAlternatives`) emits the KF8 copy, with no resource type
instead of `ALTERNATE_FORMAT_CHUNK`.
2. Right, checked against the four hybrids in your sample: read relative to
the KF8 record 0, the indices land on `HUFF`, `FDST`, `FCIS`, `FLIS`, `INDX`
and `DATP`; read from the file start, on unrelated records. The KF8 record 0 is
now kept as it is (only the first image index is cleared), and the generator
writes the indices that way. With `extractKf8`, the three of them that use
HUFF/CDIC give 2391, 3703 and 37256 characters of KF8 text, where the KF8 half
had about 60 and a warning before; the PalmDOC one hid the bug, as you
suspected.
3. KF8 text ends at the end of the first FDST flow. Both KF8-only books in
the sample had stylesheet text before and have none now.
4. `tika:encrypted` is set alongside the warning.
5. `supported-formats.adoc` regenerated;
`testSupportedFormatsPartialIsCurrent` passes.
6. The HTML pass uses `Metadata.newInstance(context)`; the text is cut at
`maxTextLength` (now 16 MB) and at 32 times the file size.
7. EXTH is read up to 1024 records and 64 values per type.
8. The KF8 half is built only when it is extracted and the extractor wants
it, straight from the file's records into one array.
9. A record that does not decode keeps the text of the records before it,
with a warning.
10. A file that is not a Mobipocket book (another Palm Database type, or no
Palm Database at all, as a Topaz or KFX book named `.azw`) yields a warning and
nothing else. PalmDOC `TEXtREAd` text would be a cheap follow-up.
11. CHANGES says so, and a detection test covers `.mobi`, `.azw`, `.azw3`
and `.prc` with `BOOKMOBI` content.
12. `maxFileSize` (256 MB by default) is a setting and documented; of a
larger book the beginning is read as a truncated one is, so the metadata and
what fits come out.
13. EXTH 106 goes through `TikaDates.toMetadataString`: a date-only value
stays a date, non-ISO dates parse.
14. Unknown and zero codepages read as windows-1252.
15. Tests for the file limit (with a small `maxFileSize`), the fuzz test
with `extractKf8` and a `@Timeout`, the two tests that swallowed exceptions now
assert, and a new stored-text fixture covers EXTH 524 over the locale, the
title fallback and a cde-type.
Also re-run against your 19-file sample and the Gutenberg books: covers
match the index (the 1 MiB cut lacks its cover, which is behind the cut), no
container exceptions.
> Parse Mobipocket and Kindle KF8 ebooks (metadata, text, cover)
> --------------------------------------------------------------
>
> Key: TIKA-5026
> URL: https://issues.apache.org/jira/browse/TIKA-5026
> Project: Tika
> Issue Type: New Feature
> Reporter: Dominik Schmidt
> Priority: Major
> Attachments: mobi-5026.zip
>
>
> Tika detects application/x-mobipocket-ebook (BOOKMOBI magic at offset 60,
> globs *.prc and *.mobi) and application/vnd.amazon.ebook (glob *.azw only),
> but no parser module handles either: /rmeta on a 9.6 MB .azw3 returns the
> Content-Type and nothing else, /unpack/preset/thumbnail returns 204, while
> the EPUB of the same book yields title, author, text and the cover marked
> THUMBNAIL. The KF8 type application/vnd.amazon.mobi8-ebook (IANA, glob *.azw3
> in shared-mime-info) is missing from tika-mimetypes.xml altogether.
> The container is the same for all three: a Palm Database (PDB) whose record 0
> holds the PalmDOC header, the MOBI header and the EXTH metadata records. A
> parser next to EpubParser could do, in this order:
> 1. Metadata from EXTH: 100 creator, 101 publisher, 103 description, 104 ISBN,
> 105 subject, 106 publishing date, 108 contributor, 503 title (falls back to
> the full name in the MOBI header), 524 language; plus the MOBI header's
> locale and encoding (CP1252 or UTF-8). Hybrid files carry EXTH 121 (KF8
> boundary) and a second record 0 for the KF8 part with its own EXTH.
> 2. Cover: EXTH 201 (CoverOffset) and 202 (ThumbOffset) index into the image
> records starting at the MOBI header's first image index; the records are
> plain JPEG/GIF/PNG. Emit the cover as an embedded resource with
> tk:embedded-resource-type THUMBNAIL (TIKA-4856 marker), the thumbnail record
> as a second one, so /unpack/preset/thumbnail serves the cover like it does
> for EPUB.
> 3. Text: MOBI7 records with PalmDOC LZ77 (compression 2) or none (1),
> trailing entries stripped per the MOBI header's extra data flags, HTML handed
> to the HTML parser. HUFF/CDIC (17480) and the KF8 skeleton/fragment text
> format are more work and can be a follow-up; metadata and cover are the same
> for both generations and cover the common search and preview use cases.
> DRM (MOBI header encryption type != 0, EXTH 208/209): record 0, EXTH and the
> image records are not encrypted, so metadata and cover still come out; the
> text does not and should raise EncryptedDocumentException rather than emit
> garbage.
> Detection: add application/vnd.amazon.mobi8-ebook as a sub-type of
> application/x-mobipocket-ebook with glob *.azw3 (the magic is identical, the
> generation is only visible from the MOBI header version and EXTH 121, so
> content detection stays at the parent), and give application/vnd.amazon.ebook
> the same parent so one parser claims all three.
> Test files: Project Gutenberg offers every title as .mobi/.kf8 (public
> domain), Calibre converts EPUB to both .mobi and .azw3 for fixtures with
> known metadata and cover.
> Related: TIKA-1892 (the magic), TIKA-4856 (thumbnail marker), TIKA-4988
> (thumbnail preset).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
