On Mon, 14 Nov 2011, Tom Bishop, Wenlin Institute wrote: > Maybe you've thought of all this already, but I mention it in case > it's helpful.
Thank you for taking the time to contribute. It's appreciated. > The differences between UTF-8, UTF-16, and UTF-32 are essentially > trivial and unrelated to the significant complexities of regular > expression implementation. (Disclaimer: I know more about UTF > encodings than I do about regex.) Ideally, the modules that need to be > different for UTF-8, UTF-16, and UTF-32 should be encapsulated to form > relatively very small parts of the PCRE source code. The problem with encapsulating too much is performance. For example, we *could* change PCRE so that every time it needs to "load next character" it calls a function. The function could check for 8,16,32, UTF status, and then load the data as required. However, I reckon this would kill PCRE's performance, which does matter in many applications. Currently, in many cases in the 8-bit environment, a simple byte load is all that is necessary. Even in UTF-8 mode when checking for metacharacters, for example, you can write code such as if *p=='*' and it works for both ascii and UTF-8 because of the clever design of UTF-8. Many functions in PCRE are large, and code is often repeated with very minor changes, and the reason is to avoid too many function calls, for better performance. > * the size of code units (whether 8-bit, 16-bit, 32-bit, or even > * 64-bit or larger) -- use a configurable typedef like PcreCodeUnit > * (or USDATA if you prefer, but PcreCodeUnit is more descriptive); ... but it's data units as well as code units... :-) > * the maximum value for a UCS Scalar Value (currently U+10FFFF but it > * has changed more than once before, and Perl 5.8 supports much larger > * code points) -- use a configurable macro like PCRE_MAX_USV; Internally PCRE supports code points up to 7fffffff in UTF-8, but you have to suppress its UTF-8 validation check to use them. > * the maximum length of a code, measured in code units (e.g., 4 for > * UTF-8, PCRE supports up to 6 for UTF-8. :-) > If it's done right, the solution for UTF-32 will come practically for > free along with the solution for UTF-16, and PCRE will easily be > extended to support applications we can't even imagine yet, for many > decades to come. To do it the unencapsulated way, only extending to > UTF-16 and hard-coding numbers like 16, 0x10FFFF, and 2, and the > notion of "surrogate pair", all through the source code, would be very > 20th-century, a kind of built-in obsolescence. These are good points, but I think PCRE is already quite well covered. Even the current code uses macros for "load next character" in UTF-8 mode, and I expect that (for example) the "surrogate pair" stuff will go into a similar macro for UTF-16. The 0x10FFFF limit is only in the "validate UTF-8" function. It would be very nice to have a single set of functions and options to specify data unit size at run time, but I don't think that is acceptable from a performance point of view. That is why the current suggestion is to have different function sets for different sizes. I agree that we should do our best so that the work to provide libpcre16 can trivially be extended to provide libpcre32 if that is ever wanted. [Aside: I've been a programmer for over 40 years. Today's machines as vastly faster, with huge memories, and yet we still worry about performance. I suspect we always will. :-] Philip -- Philip Hazel -- ## List details at https://lists.exim.org/mailman/listinfo/pcre-dev
