On Nov 15, 2005, at 11:31 AM, Gaetano Giunta wrote:

Very toughtful response.

Man, I love cross-linguistic typos...makes great new English words: "toughtful" = "tough thoughtfulness". Brilliant.

UTF-8 everywhere is fine and dandy but for 2 aspects:

- in fact XML-over-http without a charset declaration SHOULD be assumed to be ISO-8859-1 (there is a RFC somewhere about that, which I cannot recall now).

Hmmm. The XML 1.0 spec (http://www.w3.org/TR/2000/REC-xml-20001006) reads:

Because each XML entity not accompanied by external encoding information and not in UTF-8 or UTF-16 encoding MUST begin with an XML encoding declaration, in which the first characters must be '<? xml', any conforming processor can detect, after two to four octets of input, which of the following cases apply.

RFC 2376, however, offers suggestions for XML MIME-types sent over HTTP, but it reads (pardon the length):

Although listed as an optional parameter, the use of the charset
      parameter is STRONGLY RECOMMENDED, since this information can be
      used by XML processors to determine authoritatively the character
encoding of the XML entity. The charset parameter can also be used
      to provide protocol-specific operations, such as charset-based
      content negotiation in HTTP.  "UTF-8" [RFC-2279] is the
      recommended value, representing the UTF-8 charset. UTF-8 is
      supported by all conforming XML processors [REC-XML].

      If the XML entity is transmitted via HTTP, which uses a MIME-like
      mechanism that is exempt from the restrictions on the text top-
      level type (see section 19.4.1 of HTTP 1.1 [RFC-2068]), "UTF-16"
(Appendix C.3 of [UNICODE] and Amendment 1 of [ISO-10646]) is also recommended. UTF-16 is supported by all conforming XML processors [REC-XML]. Since the handling of CR, LF and NUL for text types in
      most MIME applications would cause undesired transformations of
      individual octets in UTF-16 multi-octet characters, gateways from
HTTP to these MIME applications MUST transform the XML entity from a text/xml; charset="utf-16" to application/xml; charset="utf-16".

      Conformant with [RFC-2046], if a text/xml entity is received with
      the charset parameter omitted, MIME processors and XML processors
      MUST use the default charset value of "us-ascii".  In cases where
      the XML entity is transmitted via HTTP, the default charset value
      is still "us-ascii".

...which implies that us-ascii, not iso-8859-1, is the default (but not really a problem if you're encoding everything outside of ASCII). But I know that my RDFParser class, for example, defaults to "utf-8" and overrides that only if the encoding is specified as something else in the xml delaration. I assume I made that decision for good reasons, though I don't remember them now!

Still, the number of factors affecting encoding and transmission are unbelievably complex. In my software, for example, there is:

1) Page encoding used when users submit data via a form (mine: UTF-8)
   a) Default charset header sent by Apache (mine:  UTF-8)
   b) Default charset set in META tags (mine: UTF-8)
   c) Charset setting of client browser (no control!)
2) Encoding of database (mine: MySQL 3.x, so limited to ISO-8859-1)
3) Encoding of page used to display data (Irrelevant to XML-RPC transfers, but 1a,1b,1c apply)
4) PHP internal encoding
5) XMLRPC library internal encoding
6) XML declaration charset (optional, but highly recommended by spec)
7) text/xml MIME type charset declaration (optional, mine: text/ xml;charset=utf-8)
8) application/xml MIME type charset declaration (optional)

...and since all of them could be set to different encodings, getting it all straight is a dizzying adventure. Add to that the complexity of handling things like users copying text from a Word document created in Windows-1252 and pasting into a form on a UTF-8 page, and...ugh! Sometimes I just want to kill myself.

While I suppose that attempting to convert all data into us-ascii through entity encoding gives us the "least common donominator" solution -- make everything 7-bit! -- it obviously isn't working perfectly. So perhaps any solution that simply makes it work, regardless of whether or not it changes the use of $xmlrpc_internalencoding, would be good. I did wonder about the utf8_encode() function, and why you didn't simply use that instead of $character = ("&#".strval($code).";"); Won't that do all the right work for you?

In any case, I think you should try to make the XMLRPC library follow as closely as possible the relevant spec/RFC "recommended" behavior, and let that be your guide.

Adding some extra settings to client/server objects is fine, but the causal user might not be used to using those, and backward compatability is a primary concern to me. Traduced in code that would probably mean adding some hacky stuff of the sort "object default charset preference is undefined, and while still undefined use global variable, otherwise use object preference" (doable but ugly). The though part is letting the client object communicate the desired charset encoding to the xmlrpcval object, since the responsibility of creating serialized content is left to the xmlrpcval object itself (and I'm surely not changing that fundamental assumption).

If you converted $xmlrpc_internalencoding to a property of xmlrpcmsg instead of a global variable, then you could simply set it to default to "iso-8859-1" in the constructor method for the class object. So you maintain your default, but allow users to reset it through scripting.

ps: the real (only ?) advantage of using variables instead of constnts for things such as internal_encoding is that you can redefine them not inside the xmlrpc lib but just after its inclusion, eg. <?php include('xmlrpc.inc'); $xmlrpc_internal_encoding = 'UTF-16'; echo 'etc...'; ?>
this way you do not have to change anything when updating...

Ah, yes, this is true, and I hadn't really thought of such a simple thing (but the same method holds true for using an object property).

How the PEAR people are handling this:
["According to RFC 3023 section 3.1, the encoding specified in the <? xml encoding=... ?> tag should be ignored for XML received over HTTP in favor of the encoding specified in the Content-Type header (e.g. "Content-Type: text/xml; charset=iso-8859-1)."]

I found another developer reflecting on these same questions, for a blogging app that uses XML-RPC:

Other messages about the default encoding of unspecified xml documents:
http://mail.zope.org/pipermail/zope-collector-monitor/2004-October/ 004361.html (in reverse chronological order)


a.h.s. boy
spud(at)nothingness.org            "as yes is to if,love is to yes"

phpxmlrpc mailing list

Reply via email to