Re: Markus King's "UTF-8 decoder capability and stress test"

[ToddAndMargo via perl6-users]

On 6/9/26 10:48 PM, William Michels via perl6-users wrote:
https://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt <https:// 
www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt>

Unicode/UTF-8 Question:

Does anyone know if Rakudo has ever been tested against Markus 
King's "UTF-8 decoder capability and stress test"?

 From the intro:

    "This test file can help you examine, how your UTF-8 decoder handles
    various types of correct, malformed, or otherwise interesting UTF-8
    sequences."


Any info appreciated,

Thx, Bill.


Not sure what you are after, but here is my converter.
C strings are 16 bit and end in two chr(0)'s.  I
convert them with brute force.  I have tried with
the utf16 to utf8 converters and has never happy
with the result.

It presumes a UTF16 little endian C String and converts
to UTF8.


constant BYTE     := uint8;
constant BYTES    := CArray[BYTE];
sub c-to-raku-str( BYTES $CStr ) returns Str  is export( :c-to-raku-str ) {
   # Note: C Strings are always terminated with a nul.  This sub will 
malfunction without it.
   # Note: presumes a UTF16 little endian C String and converts to UTF8

   my Str    $SubName   = &?ROUTINE.name;
   my Str    $RakuStr   = "";
   my Str    $Msg       = "";
   my uint32 $CStrElems = $CStr.elems;
   # say $CStrElems;

   loop (my $i = 0; $i < $CStrElems - 2 ; $i += 2) {
      if  $CStr[ $i ] == 0  &&  $CStr[ $i + 1 ] == 0  { last; }

      if $i == $CStrElems - 4  {
         $Msg = "$SubName ERROR:" ~ " C Strings are required to be 
terminated with a nul\n\n" ~
                "                     Returning an empty string\n" ~
                "                     Cowardly existing\n";
         exit;
      }

      # print $CStr[ $i ] ~ "," ~ $CStr[ $i + 1 ] ~ ",  ";
      $RakuStr ~= chr( $CStr[ $i ] );
   }
   # say "";

   # say "$RakuStr";
   return $RakuStr;
}