Mosiur99 opened a new pull request, #1969:
URL: https://github.com/apache/wicket/pull/1969

   Fixes #1953 (WICKET-5357).
   
   Wicket took the encoding of a markup file only from an XML declaration. An 
HTML5 template declares it with `<meta charset="...">` instead, so Wicket read 
it with the default markup encoding, and a UTF-8 template with `Größe: 5 €` 
rendered as `GröÃe: 5 â¬` when that default was something else. It also 
reported the file as having no encoding declared.
   
   ## What changes
   
   When a file has no XML declaration, `XmlReader` now looks for a meta tag in 
the first 1024 bytes and reads the file in the charset it names. Otherwise 
nothing changes.
   
   ## Decisions
   
   The issue asks to decide these in the pull request. I followed the encoding 
prescan browsers use (HTML Living Standard), so Wicket reads a template the way 
a browser reads the same file.
   
   - **Both present:** the XML declaration wins. Every template that declares 
its encoding today is read exactly as before, and it is the only form that is 
valid in XHTML.
   - **How far to look:** the first 1024 bytes, the limit browsers use. That 
covers a doctype, a license comment and the start of `<head>`. `XmlPullParser` 
already buffers 4000 bytes, so marking 1024 costs nothing.
   - **The older form:** `<meta http-equiv="Content-Type" content="text/html; 
charset=...">` is supported too. As in the browser prescan, the first meta tag 
that declares a charset wins, whichever form it uses.
   
   Other details:
   
   - Comments are skipped, and attributes are read with their quotes, so a meta 
tag inside a comment or inside another attribute value (`<div title="<meta 
charset=UTF-8>">`) doesn't count, and neither does `data-charset`.
   - A charset the JVM doesn't support is skipped instead of failing the page, 
and the next meta tag is tried.
   - A tag cut off at byte 1024 is ignored rather than read half.
   - Unlike the XML declaration, the meta tag stays in the output, since it 
belongs in the rendered page.
   
   The missing declaration message in `AbstractMarkupParser` and the Javadoc of 
`MarkupSettings#setThrowExceptionOnMissingXmlDeclaration` now mention the meta 
charset, and `migration-to-12.md` has a section on the change. No public API 
changes.
   
   ## Tests
   
   - `XmlReaderTest`: 14 new tests for quoting and case, attribute order, the 
`http-equiv` form, the XML declaration winning, the first meta tag winning, 
comments, attribute values, `data-charset`, unsupported charsets, and the 
1024-byte limit (after it, cut off by it, ending exactly at it). 
`readHtml5FileWithMetaCharset` reads a UTF-8 file with ISO-8859-1 as the 
default and checks the decoded text.
   - `MetaCharsetTest` in wicket-core-tests renders an HTML5 page: its text 
comes out right with ISO-8859-1 as the default encoding, and with 
`setThrowExceptionOnMissingXmlDeclaration(true)` it no longer throws. Both 
tests fail without the change to `XmlReader`.
   - `mvn clean verify -Pjs-test` passes.
   
   ## Not in this change
   
   A UTF-16 or UTF-32 byte order mark: `BOMInputStream` only skips a UTF-8 BOM 
and doesn't set the encoding from it. Browsers let a BOM win over everything 
else, so that would be a sensible follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to