Am 19.03.2017 um 23:00 schrieb Christian Bartolomaeus via RT:
Looking at https://en.wikipedia.org/wiki/Code_point and
http://www.unicode.org/glossary/#code_point I understand that
U+10FFFF is indeed the maximum Unicode code point.


Yes, that's the maximum value you can encode in four bytes with UTF-8, see https://en.wikipedia.org/wiki/UTF-8#Description.

I was wondering how the Unicode consortium might extend this limit, so I investigated a bit.


TL;DR

I can confirm that 10ffff is going to remain the maximum for the foreseeable future.


DETAILS


Technical limits:

  UTF-8 could be extended up to 0x108430ffff [1]
  UTF-16 ("surrogate pairs") cannot be extended beyond 0x10ffff
  UTF-32 can be extended up to 0xffffffff (32 bits available)


Political limits:

Since Java chose to use surrogate pairs, and UTF-16 is not extensible,
any motion to extend the Unicode code range would be met with opposition from Oracle, and from any language community that has a JVM implementation and wants to be interoperable with Java libraries.


Code space exhaustion:

Unicode assigns code points like this:
  characters:  128,237 code points
  private use: U+e000—U+f8ff (6,400 code points) [2]
               U+f0000—U+ffffd (65,534) [2]
               U+100000—U+10fffd (65,534) [2]
  surrogates:  U+d800—U+dfff (2,048 code points) [2]
So out of the
  0x10ffff=1,114,111 available code points,
  128,237+6,400+65,534+65,534=265705 are in use, leaving
  848406 free for future character set extension.

Assuming 10,000 new characters per year (which is conservative given the numbers in [3]), the current encoding space will be exhaused in ca. 85 years.


Regards,
Jo

[1]
The Unicode consortium could extend the maximum value of UTF-8 by using more prefixes:
  111110xx for 5-byte encoding
  1111110x for 6-byte encoding
  11111110 for 7-byte encoding
  11111111 for 8-byte encoding
No prefixes are possible for a longer encoding.

Bit counts for each prefix are:
  5-byte: nr of 4-byte encoding bits (21) + 5 = 26
  6-byte: 26 + 5 = 31
  7-byte: 26 + 5 = 36
  8-byte: 36 + 6 (prefix does not lose a bit) = 42
The maximum 8-byte-encoded value is
  0x10ffff+2^26+2^31+2^36+2^42 = 0x108430FFFF

[2]
Numbers taken from
http://unicodebook.readthedocs.io/unicode.html#statistics

[3]
See https://en.wikipedia.org/wiki/Unicode#Versions

Reply via email to