That would not be practical, nor predictable. And note that the 700K reserved code points are also not to be interpreted as characters; by your logic all of them would need to be converted to FFFD.
And in practice, an unpaired surrogate is best treated just like a reserved (unassigned) code point. For example, a lowercase operation should convert characters with lowercase correspondants to those correspondants, and leave *everything* else alone: control characters, format characters, reserved code points, surrogates, etc. ------------------------------ Mark <https://plus.google.com/114199149796022210033> * * *— Il meglio è l’inimico del bene —* ** On Tue, Mar 27, 2012 at 08:02, Glenn Adams <[email protected]> wrote: > > > On Tue, Mar 27, 2012 at 8:39 AM, Mark Davis ☕ <[email protected]> wrote: > >> That, as Norbert explained, is not the intention of the standard. Take a >> look at the discussion of "Unicode 16-bit string" in chapter 3. The >> committee recognized that fragments may be formed when working with UTF-16, >> and that destructive changes may do more harm than good. >> >> x = a.substring(0, 5) + b + a.substring(5, a.length()); >> y = x.substring(0, 5) + x.substring(6, x.length()); >> >> After this operation is done, you want y == a, even if 5 is between D800 >> and DC00. >> > > Assuming that b.length() == 1 in this example, my interpretation of this > is that '=', '+', and 'substring' are operations whose domain and co-domain > are (currently defined) ES Strings, namely sequences of UTF-16 code units. > Since none of these operations entail interpreting the semantics of a code > point (i.e., interpreting abstract characters), then there is no violation > of C1 here. > > Or take: >> output = ""; >> for (int i = 0; i < s.length(); ++i) { >> ch = s.charAt(i); >> if (ch.equals('&')) { >> ch = '@'; >> } >> output += ch; >> } >> >> After this operation is done, you want "a&\u{10000}b" to become >> "a@\u{10000}b", >> not "a&\u{FFFD}\u{FFFD}b". >> It is also an unnecessary burden on lower-level software to always check >> this stuff. >> > > Again, in this example, I assume that the string literal "a&\u{10000}b" > maps to the UTF-16 code unit sequence: > > 0061 0026 D800 DC00 0062 > > Given that 'charAt(i)' is defined on (and is indexing) code units and not > code points, and since the 'equals' operator is also defined on code units, > this example also does not require interpreting the semantics of code > points (i.e., interpreting abstract characters). > > However, in Norbert's questions above about isUUppercase(int) and > toUpperCase(int), it is clear that the domain of these operations are code > points, not code units, and further, that they requiring interpretation as > abstract characters in order to determine the semantics of the > corresponding characters. > > My conclusion is that the determination of whether C1 is violated or not > depends upon the domain, codomain, and operation being considered. > > >> Of course, when you convert to UTF-16 (or UTF-8 or 32) for storage or >> output, then you do need to either convert to FFFD or take some other >> action. >> >> ------------------------------ >> Mark <https://plus.google.com/114199149796022210033> >> * >> * >> *— Il meglio è l’inimico del bene —* >> ** >> >> >> >> On Mon, Mar 26, 2012 at 23:11, Glenn Adams <[email protected]> wrote: >> >>> >>> On Mon, Mar 26, 2012 at 10:37 PM, Norbert Lindenberg < >>> [email protected]> wrote: >>> >>>> The conformance clause doesn't say anything about the interpretation of >>>> (UTF-16) code units as code points. To check conformance with C1, you have >>>> to look at how the resulting code points are actually further interpreted. >>>> >>> >>> True, but if the proposed language >>> >>> "A code unit that is in the range 0xD800 to 0xDFFF, but is not part of a >>> surrogate pair, is interpreted as a code point with the same value." >>> >>> is adopted, then will not this have an effect of creating unpaired >>> surrogates as code points? If so, then by my estimation, this *will* >>> increase >>> the likelihood of their being interpreted as abstract characters... e.g., >>> if the unpaired code unit is interpreted as a unpaired surrogate code >>> point, and some process/function performs *any* predicate or transform >>> on that code point, then that amounts to interpreting it as an abstract >>> character. >>> >>> I would rather see such unpaired code unit either (1) be mapped to >>> U+00FFFD, or (2) an exception raised when performing an operation that >>> requires conversion of the UTF-16 code unit sequence. >>> >>> >>>> My proposal interprets the resulting code points in the following ways: >>>> >>>> 1) In regular expressions, they can be used in both patterns and input >>>> strings to be matched. They may be compared against other code points, or >>>> against character classes, some of which will hopefully soon be defined by >>>> Unicode properties. In the case of comparing against other code points, >>>> they can't match any code points assigned to abstract characters. In the >>>> case of Unicode properties, they'll typically fall into the large bucket of >>>> have-nots, along with other unassigned code points or, for example, U+FFFD, >>>> unless you ask for their general category. >>>> >>>> 2) When parsing identifiers, they will not have the ID_Start or >>>> ID_Continue properties, so they'll be excluded, just like other unassigned >>>> code points or U+FFFD. >>>> >>>> 3) In case conversion, they won't have upper case or lower case >>>> equivalents defined, and remain as is, as would happen for unassigned code >>>> points or U+FFFD. >>>> >>>> I don't think either of these amount to interpretation as abstract >>>> characters. I mention U+FFFD because the alternative interpretation of >>>> unpaired surrogates would be to replace them with U+FFFD, but that doesn't >>>> seem to improve anything. >>>> >>>> Norbert >>>> >>>> >>>> >>>> On Mar 26, 2012, at 15:10 , Glenn Adams wrote: >>>> >>>> > On Mon, Mar 26, 2012 at 2:02 PM, Gavin Barraclough < >>>> [email protected]> wrote: >>>> > I really like the direction you're going in, but have one minor >>>> concern relating to regular expressions. >>>> > >>>> > In your proposal, you currently state: >>>> > "A code unit that is in the range 0xD800 to 0xDFFF, but is not >>>> part of a surrogate pair, is interpreted as a code point with the same >>>> value." >>>> > >>>> > Just as a reminder, this would be in explicit violation of the >>>> Unicode conformance clause C1 unless it can be guaranteed that such a code >>>> point will not be interpreted as an abstract character: >>>> > >>>> > C1 A process shall not interpret a high-surrogate code point or a >>>> low-surrogate code point as an abstract character. >>>> > >>>> > [1] http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf >>>> > >>>> > Given that such guarantee is likely impractical, this presents a >>>> problem for the above proposed language. >>>> >>>> >>> >>> _______________________________________________ >>> es-discuss mailing list >>> [email protected] >>> https://mail.mozilla.org/listinfo/es-discuss >>> >>> >> >
_______________________________________________ es-discuss mailing list [email protected] https://mail.mozilla.org/listinfo/es-discuss
