Alan Kennedy wrote:
> [Bill]
> > I think the controlling reference here is RFC 3875.
>
> I think the controlling references are RFC 2616, RFC 2396 and RFC
3987.
>
> RFC 2616, the HTTP 1.1 spec, punts on the question of character
> encoding for the request URI.
>
> RFC 2396, the URI spec, says
>
> """
> It is expected that a systematic treatment of character encoding
> within URI will be developed as a future modification of this
> specification.
> """
>
> RFC 3987 is that spec, for Internationalized Resource Identifiers. It
> says
>
> """
> An IRI is a sequence of characters from the Universal Character Set
> (Unicode/ISO 10646).
> """
>
> and
>
> """
> 1.2. Applicability
>
> IRIs are designed to be compatible with recommendations for new URI
> schemes [RFC2718]. The compatibility is provided by specifying a
> well-defined and deterministic mapping from the IRI character
> sequence to the functionally equivalent URI character sequence.
> Practical use of IRIs (or IRI references) in place of URIs (or URI
> references) depends on the following conditions being met:
> """
>
> followed by
>
> """
> c. The URI corresponding to the IRI in question has to encode
> original characters into octets using UTF-8. For new URI
> schemes, this is recommended in [RFC2718]. It can apply to a
> whole scheme (e.g., IMAP URLs [RFC2192] and POP URLs [RFC2384],
> or the URN syntax [RFC2141]). It can apply to a specific part
> of
> a URI, such as the fragment identifier (e.g., [XPointer]). It
> can apply to a specific URI or part(s) thereof. For details,
> please see section 6.4.
> """
>
> I think the question is "are people using IRIs in the wild"? If so,
> then we must decide how do we best deal with the problems of
> recognising iso-8859-1+rfc2037 versus utf-8, or whatever
> server-configured encoding the user has chosen.
Agreed. The Request-URI needs to handle IRI's. The headers mostly
don't--almost all headers are of mostly type "token", which is US-ASCII.
A few are of type "TEXT", which is ISO-8859-1/RFC 2047. The remaining
(sub)values are mostly custom byte sequences:
field-name field-value
---------- -----------
Accept token
Accept-Charset token
Accept-Encoding token
Accept-Language ALPHA, plus ":", "=", "q" etc
Accept-Ranges token
Age DIGIT
Allow token
Authorization token
Cache-Control token
Connection token
Content-Encoding token
Content-Language ALPHA
Content-Length DIGIT
Content-Location absoluteURI | relativeURI
Content-MD5 base64 of 128 bit md5 digest
Content-Range DIGIT, plus "/" etc
Content-Type token
Date HTTP-date
ETag TEXT and CHAR
Expect token, quoted-string
Expires HTTP-date
>From ASCII (see RFC 822)
Host host ":" port
If-Match TEXT and CHAR
If-Modified-Since HTTP-date
If-None-Match TEXT and CHAR
If-Range TEXT and CHAR | HTTP-date
If-Unmodified-Since HTTP-date
Last-Modified HTTP-date
Location absoluteURI
Max-Forwards DIGIT
Pragma token, quoted-string
Proxy-Authenticate token
Proxy-Authorization token
Range token
Referer absoluteURI | relativeURI
Retry-After HTTP-date | DIGIT
Server token, TEXT
TE token
Trailer token
Transfer-Encoding token
Upgrade token
User-Agent token, TEXT
Vary token
Via token, host, port
Warning quoted-string, HTTP-date, host, port
WWW-Authenticate token
The Content-Location, Location, and Referer headers are problematic
since HTTP borrows those from the URI spec, which deals in characters
and not bytes, as you mentioned. Host, and maybe Via, are also special
due to possible IDNA-encoding.
Regarding extension headers, I think we should assume that the HTTP/1.1
spec implies all headers should be token (ASCII) or TEXT (ISO-8859-1).
>From section 4.2:
field-content = <the OCTETs making up the field-value
and consisting of either *TEXT or combinations
of token, separators, and quoted-string>
In addition, the httpbis effort seems to be enforcing this even more
strongly [1]:
message-header = field-name ":" OWS [ field-value ] OWS
field-name = token
field-value = *( field-content / OWS )
field-content = *( WSP / VCHAR / obs-text )
Historically, HTTP has allowed field-content with text in the ISO-
8859-1 [ISO-8859-1] character encoding (allowing other character sets
through use of [RFC2047] encoding). In practice, most HTTP header
field-values use only a subset of the US-ASCII charset [USASCII].
Newly defined header fields SHOULD constrain their field-values to
US-ASCII characters. Recipients SHOULD treat other (obs-text) octets
in field-content as opaque data.
So, from where I sit, we have:
1. Many header values which are ASCII.
2. A few header values which are ISO-8859-1 plus RFC 2047.
3. A few header values which are URI's (no specified encoding) or IRI's
(UTF-8).
I understand the desire to decode ASAP, and I agree with Guido that we
should use a default encoding which the app can override. Looking at the
above, ISO-8859-1 is the best encoding I know of for all three header
cases. ASCII can be used as a valid subset without transcoding; headers
which are ISO-8859-1 are decoded perfectly; URI/IRI headers can be
transcoded by the app if needed, but mangled opaquely by middleware.
If we make *that* call, then IMO there's no reason not to do the same to
SCRIPT_NAME, PATH_INFO, and QUERY_STRING.
Robert Brewer
fuman...@aminus.org
[1]
http://www.ietf.org/internet-drafts/draft-ietf-httpbis-p1-messaging-06.t
xt
_______________________________________________
Web-SIG mailing list
Web-SIG@python.org
Web SIG: http://www.python.org/sigs/web-sig
Unsubscribe:
http://mail.python.org/mailman/options/web-sig/archive%40mail-archive.com
