(doris) branch master updated: [fix](function) Split UTF-8 strings correctly with empty regexp (#68020)

[yanxuecheng]

This is an automated email from the ASF dual-hosted git repository.

Mryange pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new e1ba108ea72 [fix](function) Split UTF-8 strings correctly with empty 
regexp (#68020)
e1ba108ea72 is described below

commit e1ba108ea72ad107f58db5e42fcb7ee1d42dcc9a
Author: Mryange <yanxuech...@selectdb.com>
AuthorDate: Thu Sep 17 17:40:36 2026 +0800

    [fix](function) Split UTF-8 strings correctly with empty regexp (#68020)
    
    `split_by_regexp` advanced one byte at a time when the pattern was
    empty, which split multibyte UTF-8 characters into invalid array
    elements and could truncate a character when a limit was specified. Root
    cause: the empty-pattern path bypassed RE2 and treated each byte as a
    complete character. The implementation now advances by the current UTF-8
    code point length while preserving the existing byte-based behavior for
    ASCII input. Regression coverage verifies Chinese and emoji input with
    and without a split limit.
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Test <!-- At least one of them must be included. -->
        - [ ] Regression test
        - [ ] Unit Test
        - [ ] Manual test (add detailed scripts or steps below)
        - [ ] No need to test or manual test. Explain why:
    - [ ] This is a refactor/code format and no logic has been changed.
            - [ ] Previous test can cover this change.
            - [ ] No code files have been changed.
            - [ ] Other reason <!-- Add your reason?  -->
    
    - Behavior changed:
        - [ ] No.
        - [ ] Yes. <!-- Explain the behavior change -->
    
    - Does this need documentation?
        - [ ] No.
    - [ ] Yes. <!-- Add document PR link here. eg:
    https://github.com/apache/doris-website/pull/1214 -->
    
    ### Check List (For Reviewer who merge this PR)
    
    - [ ] Confirm the release note
    - [ ] Confirm test cases
    - [ ] Confirm document
    - [ ] Add branch pick label <!-- Add branch pick label that this PR
    should merge into -->
---
 be/src/exprs/function/function_split_by_regexp.cpp                  | 4 +++-
 .../sql_functions/string_functions/test_split_by_regexp.out         | 6 ++++++
 .../sql_functions/string_functions/test_split_by_regexp.groovy      | 2 ++
 3 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/be/src/exprs/function/function_split_by_regexp.cpp 
b/be/src/exprs/function/function_split_by_regexp.cpp
index d889b0fc862..292bcb64559 100644
--- a/be/src/exprs/function/function_split_by_regexp.cpp
+++ b/be/src/exprs/function/function_split_by_regexp.cpp
@@ -27,6 +27,7 @@
 #include "core/types.h"
 #include "exprs/function/function.h"
 #include "exprs/function/simple_function_factory.h"
+#include "util/simd/vstring_function.h"
 
 namespace doris {
 
@@ -115,7 +116,8 @@ bool RegexpSplit::get(const char*& token_begin, const 
char*& token_end) {
             }
         }
 
-        _pos += 1;
+        const auto utf8_byte_length = 
get_utf8_byte_length(static_cast<uint8_t>(*_pos));
+        _pos += std::min<size_t>(utf8_byte_length, _end - _pos);
         token_end = _pos;
         ++_splits;
     } else {
diff --git 
a/regression-test/data/query_p0/sql_functions/string_functions/test_split_by_regexp.out
 
b/regression-test/data/query_p0/sql_functions/string_functions/test_split_by_regexp.out
index fa6e31abd94..d9184915ab7 100644
--- 
a/regression-test/data/query_p0/sql_functions/string_functions/test_split_by_regexp.out
+++ 
b/regression-test/data/query_p0/sql_functions/string_functions/test_split_by_regexp.out
@@ -2,6 +2,12 @@
 -- !select1 --
 ["a", "b", "c", "d", "e"]
 
+-- !select_utf8_empty_pattern --
+["中", "a", "😀"]
+
+-- !select_utf8_empty_pattern_with_limit --
+["中", "a😀"]
+
 -- !select2 --
 ["a", "bc", "de", "f"]
 
diff --git 
a/regression-test/suites/query_p0/sql_functions/string_functions/test_split_by_regexp.groovy
 
b/regression-test/suites/query_p0/sql_functions/string_functions/test_split_by_regexp.groovy
index 678f7221642..9754ca6e592 100644
--- 
a/regression-test/suites/query_p0/sql_functions/string_functions/test_split_by_regexp.groovy
+++ 
b/regression-test/suites/query_p0/sql_functions/string_functions/test_split_by_regexp.groovy
@@ -17,6 +17,8 @@
 
 suite("test_split_by_regexp") {
     qt_select1 "select split_by_regexp('abcde','');"
+    order_qt_select_utf8_empty_pattern "select split_by_regexp('中a😀', '');"
+    order_qt_select_utf8_empty_pattern_with_limit "select 
split_by_regexp('中a😀', '', 2);"
     qt_select2 "select split_by_regexp('a12bc23de345f','\\\\d+');"
     qt_select3 "select split_by_regexp('a12bc23de345f',NULL);"
     qt_select4 "select split_by_regexp(NULL, 'a12bc23de345f');"


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org