Maybe you've thought of all this already, but I mention it in case it's helpful.
The differences between UTF-8, UTF-16, and UTF-32 are essentially trivial and unrelated to the significant complexities of regular expression implementation. (Disclaimer: I know more about UTF encodings than I do about regex.) Ideally, the modules that need to be different for UTF-8, UTF-16, and UTF-32 should be encapsulated to form relatively very small parts of the PCRE source code. Some things to encapsulate: * the size of code units (whether 8-bit, 16-bit, 32-bit, or even 64-bit or larger) -- use a configurable typedef like PcreCodeUnit (or USDATA if you prefer, but PcreCodeUnit is more descriptive); * the maximum value for a UCS Scalar Value (currently U+10FFFF but it has changed more than once before, and Perl 5.8 supports much larger code points) -- use a configurable macro like PCRE_MAX_USV; * the maximum length of a code, measured in code units (e.g., 4 for UTF-8, 2 for UTF-16, 1 for UTF-32, supposing PCRE_MAX_USV = 0x10FFFF) -- use a configurable macro like PCRE_MAX_UNITS_PER_CODE; * the algorithms for determining the lengths of a codes and converting between codes and USV -- encapsulate these functions or macros so that they can easily be replaced. If it's done right, the solution for UTF-32 will come practically for free along with the solution for UTF-16, and PCRE will easily be extended to support applications we can't even imagine yet, for many decades to come. To do it the unencapsulated way, only extending to UTF-16 and hard-coding numbers like 16, 0x10FFFF, and 2, and the notion of "surrogate pair", all through the source code, would be very 20th-century, a kind of built-in obsolescence. Best wishes, Tom 文林 Wenlin Institute, Inc. Software for Learning Chinese E-mail: [email protected] Web: http://www.wenlin.com Telephone: 1-877-4-WENLIN (1-877-493-6546) ☯ -- ## List details at https://lists.exim.org/mailman/listinfo/pcre-dev
