[jira] [Created] (ASTERIXDB-2762) Use code point as the unit in string-related functions

[Rui Guo (Jira)]

Rui Guo created ASTERIXDB-2762:
----------------------------------

             Summary: Use code point as the unit in string-related functions
                 Key: ASTERIXDB-2762
                 URL: https://issues.apache.org/jira/browse/ASTERIXDB-2762
             Project: Apache AsterixDB
          Issue Type: New Feature
            Reporter: Rui Guo


Currently, we use byte or Java char (2 byte) as the unit in the string-related 
built-in functions. For example, in substr(string, offset, len) the offset and 
len are in the unit of Java char. However, for non-English characters such as 
Emoji and Korean, one character may have multiple bytes and thus it is possible 
to split one character into a few illegal parts.

Using code point as the unit would be a more natural way to deal with those 
multi-byte characters.

 

List of functions that need to be updated:


||Function||Expected output||
|substr("🇺🇸🇨🇳", 2, 2);|"🇨🇳"|
|trim("🇺🇸", "🇺");|"🇸"|
|reverse("🇨🇳");|"🇳🇨" |
|length(“🇨🇳");| 2|
| position("👩‍👩‍👧‍👦🇨🇳", "🇨🇳");| 7|
|regexp_contains("🇨🇳", "🇨🇳");regexp_like("🇨🇳", "🇨🇳");|true
true |
|string_to_codepoint("🇺🇸");| [ 127482, 127480 ]|
|codepoint_to_string([ 127482, 127480]);|"🇺🇸" |

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)