From: Aidan Kehoe <php-manual at parhasard dot net>
Operating system: NetBSD 1.6ZD
PHP version: 4.3.4
PHP Bug Type: PCRE related
Bug description: preg_split('//u' ...) splits into octets, not UTF-8 characters.
Description:
------------
http://php.net/manual/en/pcre.pattern.modifiers.php states that the /u
modifier "... turns on additional functionality of PCRE that is
incompatible with Perl. Pattern strings are treated as UTF-8."
The PCRE documentation itself says "In order process UTF-8 strings, you
must build PCRE to include UTF-8 support in the code, and, in addition,
you must call pcre_compile() with the PCRE_UTF8 option flag. When you do
this, both the pattern and any subject strings that are matched against it
are treated as UTF-8 strings instead of just strings of bytes." This says,
to me, that the /u modifier in our PCRE expressions maps to the PCRE_UTF8
option flag in the C.
And, sure enough, preg_match_all('/./u', $string, $matches) puts an array
of all the UTF-8 characters in $string into $matches[0].
preg_split('//u', $string) then, by this logic, should return an array
containing the UTF-8 characters in $string. It doesn't--it returns instead
an array of the octets in $string.
Reproduce code:
---------------
#!/usr/pkg/bin/php
<?php
/* The Euro sign--U+20AC--followed by " hi there",
in UTF-8. */
$teststr = "\xe2\x82\xac hi there";
/* Split it into individual characters, passing the /u flag
to tell PCRE to interpret the string as UTF-8. */
$testchars = preg_split('//u', $teststr, -1, PREG_SPLIT_NO_EMPTY);
/* Get some output that should be equivalent. */
preg_match_all('/./u', $teststr, $matches);
$goodtestchars = $matches[0];
/* The arrays should be the same length. */
print "This should be 1: '".(count($testchars)
== count($goodtestchars))."'\n";
/* And the octet count of the first entry should be
three for both arrays. */
print 'These both should be three: ';
print strlen($testchars[0]).', '.strlen($goodtestchars[0]).
"\n";
?>
Expected result:
----------------
$ ./testing.php
This should be 1: '1'
These both should be three: 3, 3
$
Actual result:
--------------
$ ./testing.php
This should be 1: ''
These both should be three: 1, 3
$
--
Edit bug report at http://bugs.php.net/?id=27103&edit=1
--
Try a CVS snapshot (php4): http://bugs.php.net/fix.php?id=27103&r=trysnapshot4
Try a CVS snapshot (php5): http://bugs.php.net/fix.php?id=27103&r=trysnapshot5
Fixed in CVS: http://bugs.php.net/fix.php?id=27103&r=fixedcvs
Fixed in release: http://bugs.php.net/fix.php?id=27103&r=alreadyfixed
Need backtrace: http://bugs.php.net/fix.php?id=27103&r=needtrace
Need Reproduce Script: http://bugs.php.net/fix.php?id=27103&r=needscript
Try newer version: http://bugs.php.net/fix.php?id=27103&r=oldversion
Not developer issue: http://bugs.php.net/fix.php?id=27103&r=support
Expected behavior: http://bugs.php.net/fix.php?id=27103&r=notwrong
Not enough info: http://bugs.php.net/fix.php?id=27103&r=notenoughinfo
Submitted twice: http://bugs.php.net/fix.php?id=27103&r=submittedtwice
register_globals: http://bugs.php.net/fix.php?id=27103&r=globals
PHP 3 support discontinued: http://bugs.php.net/fix.php?id=27103&r=php3
Daylight Savings: http://bugs.php.net/fix.php?id=27103&r=dst
IIS Stability: http://bugs.php.net/fix.php?id=27103&r=isapi
Install GNU Sed: http://bugs.php.net/fix.php?id=27103&r=gnused
Floating point limitations: http://bugs.php.net/fix.php?id=27103&r=float
