On Sun, 16 Sep 2007, Paul Vigay wrote:
I've just discovered what I believe to be a serious bug in NetSurf2 (not
tried with NS1 yet) and I'm just off to report it in the bug tracker.
However, NetSurf is not correctly encoding characters in web forms.
Things like %91 and %92 characters are simply encoded as question marks,
which is corrupting any forms submitted via NetSurf.
I've just knocked up a quick test script at
http://www.truefacts.co.uk/cgi-bin/test which allows you to cut and paste
characters into a form and see what they're encoded as. NetSurf is not
working correctly.
The character set used for form submission is defined as follows:
If there is an accept-charset attribute on the form, then use the most
appropriate entry in the list.
Otherwise, if there is a document character set, use that.
Otherwise, fall back to ISO-8859-1.
In the case of the page above, it is served up with no character set
specified. Therefore ISO-8859-1 is fallen back to. %91 and %92 are not
defined within the ISO-8859-1 character set, and thus cannot be
represented. In this situation, NetSurf substitutes question marks (or
U+FFFD if the selected submission encoding is UTF-*) for the
unrepresentable characters.
Other browsers encode unrepresentable characters as if they were HTML
entities (e.g. ‘), then percent encode that (so it comes out as
%26%238216%3B). This approach forces server-side processors to be HTML
entity aware for no good reason, loses the 1:1 mapping between characters
entered by the user and received by the server and makes it impossible to
distinguish between user input and client fallback (i.e. there is no
distinction between the user entering "‘" and them entering a sexed
quote -- the server-side sees exactly the same thing).
John.
