Mosiur99 opened a new pull request, #1969: URL: https://github.com/apache/wicket/pull/1969
Fixes #1953 (WICKET-5357). Wicket took the encoding of a markup file only from an XML declaration. An HTML5 template declares it with `<meta charset="...">` instead, so Wicket read it with the default markup encoding, and a UTF-8 template with `Größe: 5 €` rendered as `GröÃe: 5 â¬` when that default was something else. It also reported the file as having no encoding declared. ## What changes When a file has no XML declaration, `XmlReader` now looks for a meta tag in the first 1024 bytes and reads the file in the charset it names. Otherwise nothing changes. ## Decisions The issue asks to decide these in the pull request. I followed the encoding prescan browsers use (HTML Living Standard), so Wicket reads a template the way a browser reads the same file. - **Both present:** the XML declaration wins. Every template that declares its encoding today is read exactly as before, and it is the only form that is valid in XHTML. - **How far to look:** the first 1024 bytes, the limit browsers use. That covers a doctype, a license comment and the start of `<head>`. `XmlPullParser` already buffers 4000 bytes, so marking 1024 costs nothing. - **The older form:** `<meta http-equiv="Content-Type" content="text/html; charset=...">` is supported too. As in the browser prescan, the first meta tag that declares a charset wins, whichever form it uses. Other details: - Comments are skipped, and attributes are read with their quotes, so a meta tag inside a comment or inside another attribute value (`<div title="<meta charset=UTF-8>">`) doesn't count, and neither does `data-charset`. - A charset the JVM doesn't support is skipped instead of failing the page, and the next meta tag is tried. - A tag cut off at byte 1024 is ignored rather than read half. - Unlike the XML declaration, the meta tag stays in the output, since it belongs in the rendered page. The missing declaration message in `AbstractMarkupParser` and the Javadoc of `MarkupSettings#setThrowExceptionOnMissingXmlDeclaration` now mention the meta charset, and `migration-to-12.md` has a section on the change. No public API changes. ## Tests - `XmlReaderTest`: 14 new tests for quoting and case, attribute order, the `http-equiv` form, the XML declaration winning, the first meta tag winning, comments, attribute values, `data-charset`, unsupported charsets, and the 1024-byte limit (after it, cut off by it, ending exactly at it). `readHtml5FileWithMetaCharset` reads a UTF-8 file with ISO-8859-1 as the default and checks the decoded text. - `MetaCharsetTest` in wicket-core-tests renders an HTML5 page: its text comes out right with ISO-8859-1 as the default encoding, and with `setThrowExceptionOnMissingXmlDeclaration(true)` it no longer throws. Both tests fail without the change to `XmlReader`. - `mvn clean verify -Pjs-test` passes. ## Not in this change A UTF-16 or UTF-32 byte order mark: `BOMInputStream` only skips a UTF-8 BOM and doesn't set the encoding from it. Browsers let a BOM win over everything else, so that would be a sensible follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
