mrdevel2002-ui opened a new pull request, #3567: URL: https://github.com/apache/tika/pull/3567
`ToMarkdownContentHandler` collapses line breaks in prose text and, since TIKA-5056, in math before handing them to commonmark, but not in the literals of inline `<code>`, `<pre>` code blocks, image alt text, or link and image destinations. commonmark's `MarkdownRenderer` writes a code span literal raw and splits a code block literal on LF only, so a line break from the document reaches the markdown as a real line ending. An LF inside `<code>` ends the code span's line, and if the next line starts with `<details`, `<div` or `<script` it opens a raw HTML block. A lone CR inside a `<pre>` within `<li>` or `<blockquote>` gets no list indent or `> ` prefix, yet the CommonMark spec counts CR as a line ending, so the code block and its container close there and the rest of that line is raw HTML. Both paths are reachable with the default config. Through `JSoupParser` and the default mapper, `<ul><li><pre>a <img src=x onerror=alert(document.domain)></pre></li></ul>` comes out as a list item whose code block holds `a`, CR, `<img src=x onerror=alert(document.domain)>`. An EPUB chapter with `<p>See <code>x` + LF + `<details open ontoggle=alert(document.domain)></code> here</p>` comes out as ``See `x`` followed by a line starting with the `<details>` tag. commonmark-java's `HtmlRenderer` turns them into a live `<img onerror>` and a live `<details ontoggle>`. Markdown is the default output of tika-app and of tika-server `/tika` and `/rmeta` in 4.x, and `security.adoc` says markdown output escapes document text so that it does not become HTML or markdown structure. The fix collapses line breaks in the code span literal and in alt text, which the prose and math paths already do, and turns CR and CRLF into LF in code block literals and destinations. commonmark then prefixes every code line and encodes destination line breaks as ` `, which it already does for LF. CommonMark renders line endings inside code spans and alt text as spaces anyway, and a CR in a `<pre>` was already a line ending to the parser, so ordinary documents render the same; the markdown text changes only where a document had a CR, or a line break inside inline code. The alt text and destination cases can only start a new markdown line (a heading, a list), since `<` stays escaped there. I included them because the cause is the same and a separate fix later would be the same change. This is separate from #2891 (escaping link, image and table text in the old writer) and from the math spans of TIKA-5056: those literals needed escaping, while these need line-ending handling th at commonmark does not do for them. `testLineBreakInCodeSpanCannotOpenHtml`, `testCarriageReturnInCodeBlockCannotLeaveContainer` and `testLineBreakInImageAltOrDestinationStaysInline` render the handler's output with commonmark's `HtmlRenderer`, as the TIKA-5056 math tests do. All three fail on main and pass with the fix. I also ran the two documents above through `JSoupParser` and `EpubParser` against the patched tika-core jar; the HTML now has `<img` and `<details` inside `<code>`. `./mvnw clean apache-rat:check install -Pci -pl tika-core` (1516 tests, checkstyle, spotless, forbidden-apis, rat) is green on JDK 21, as are the tests of tika-parser-text-module and tika-parser-miscoffice-module. - [ ] open issue on the Tika issue tracker: not filed yet, happy to open one or have it linked - [ ] issue ID in the PR title and commit message: pending the issue above - [x] commits are squashed into a single one - [x] Tika builds and unit tests pass with `./mvnw clean install` (affected modules) - [x] no conflicts with the recent `main` branch - [ ] new downstream module added to `tika-bom/pom.xml`: not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
