dschmidt opened a new pull request, #3531: URL: https://github.com/apache/tika/pull/3531
Tika has no type and no parser for Jupyter notebooks, so `.ipynb` files yield neither text nor a thumbnail. `JupyterParser` (miscoffice module, `application/x-ipynb+json`, glob `*.ipynb`, sub-type of `application/json`) reads nbformat 4 notebooks with jackson-core: - **Text**: markdown and raw cells as paragraphs, code cells as preformatted text, each followed by its plain-text outputs (streams, `text/plain` results, `ename: evalue` of errors). HTML, widget and other output data is skipped while reading. - **Metadata**: new keys `jupyter:kernel` (`kernelspec.name`), `jupyter:language` (`language_info.name`, else `kernelspec.language`) and `jupyter:nbformat` (`4.5`), in a `Jupyter` interface; the registry is updated. - **Images**: every image output (SVG, else PNG, else JPEG) is an embedded document, one of them the `THUMBNAIL`, the others `INLINE`. - **Limits**: the document is read up to 256 MB and a single string (a source, an encoded image) up to 64 MB through Jackson's `StreamReadConstraints`. **Which image is the thumbnail.** Notebook images are almost always plots, so no choice is obviously right. Two conventions exist: nbsphinx, which renders notebooks for Sphinx docs, takes the last image; sphinx-gallery, which renders Python scripts, takes the first figure. On a sample of 30 notebooks from 30 GitHub repositories, 11 had a single image; of the other 19, the last image represented the notebook better in 5 (a result rather than data exploration), the first in 3, and neither in 11. The parser follows nbsphinx, the convention made for notebooks, and implements its selection exactly (`NbsphinxThumbnail`, with the source it follows in the javadoc): 1. A code cell marked `nbsphinx-thumbnail`, as cell metadata (optional `output-index`, `tooltip`) or as a tag, names the output: the one at `output-index`, else its last. 2. A marker that cannot be honoured (other keys, not a code cell, a second marked cell, no outputs, a bad index, an output without an image) means no thumbnail, as nbsphinx shows a broken one and does not fall back; the reason is recorded as a warning. 3. Otherwise the last image: code cells from the last, outputs from the last. Within an output SVG wins over PNG over JPEG, nbsphinx's display priority. Checked against nbsphinx 0.9.7 itself on 46 real notebooks, the 30 above and 16 that use the marker: the same cell and output for all of them. The ten that nbsphinx reports as broken are documentation notebooks stored without outputs, which nbsphinx executes at build time; Tika does not execute, so they get no thumbnail. A setting to take the first image instead could be added later if someone wants it. nbformat 3 notebooks (cells under `worksheets`, from before 2015) are not read beyond `jupyter:nbformat`. Tests: a small synthetic notebook (two figures, a result, an error) for text, metadata and the implicit thumbnail, and notebooks built in the test for every rule above; `TikaDetectionTest` gets `x.ipynb`. https://issues.apache.org/jira/browse/TIKA-5037 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
