EscaperLL opened a new pull request, #70549:
URL: https://github.com/apache/doris/pull/70549
### Background
Doris provides regular-expression extraction and replacement functions, but
does not expose a match-position function with occurrence and capture-group
selection. This change adds Nereids analysis and RE2-based BE execution for
that interface.
### What problem does this PR solve?
Issue Number: close #70548; related #51351 and #48203
Related PR: #51462 and #55074 (earlier closed, unmerged attempts);
apache/doris-website#4634 (documentation)
PR Type: Feature
Problem Summary:
Add `regexp_position` with two to seven arguments, plus the `regexp_instr`,
`regexpPosition`, and `regexpInstr` aliases. Users can find a match or capture
group's one-based byte position directly in SQL, selecting the occurrence and
whether to return its start or end. Most of the diff is unit and regression
coverage. The interface follows the specified ClickHouse baseline rather than
the Trino semantics originally requested in #51351.
### Details
- **Change map:** Eight files, 2,621 added lines: 495 production lines (136
FE analysis/registration/visitor; 359 BE execution), 2,023 test lines (306 FE
unit; 1,440 BE unit; 277 SQL regression), and 103 generated regression-output
lines. No documentation files or personal planning files are included in this
code PR.
- **Scope:** Add the scalar function and aliases, validate original argument
types before coercion, fill optional arguments in FE, and evaluate matching in
BE. No changes to existing regular-expression functions.
- **Design:** FE normalizes names to `regexp_position` and supplies a fixed
seven-argument BE layout. BE keeps compiled RE2 and reusable match buffers in
thread-local function state, compiling on the first row whose arguments are all
non-NULL. It searches the complete input with a byte offset to preserve anchor
context, counts non-overlapping matches, and explicitly handles
constant/nullable columns. No new configuration or third-party dependency.
- **Usage:** `regexp_position(haystack, pattern[, position[, occurrence[,
return_option[, flags[, subexpression]]]]])`. String arguments accept
STRING/VARCHAR; numeric arguments accept integer types through BIGINT. Other
types require explicit casts. Pattern and flags must be deterministic constants.
- **Default behavior:** Optional arguments default to `1, 1, 0, '', 0`.
Return BIGINT byte positions, `0` on no match, and NULL when any argument is
NULL in that row. `return_option=0` selects the start; `1` selects the position
immediately after the end. Invalid arguments fail the query. Flags support
`i/c`, `m/n`, and `s`. The unflagged empty literal pattern, general zero-width
matches, and empty captures retain the specified baseline's distinct results.
- **Compatibility:** Matching is checked against ClickHouse 26.10.2.103,
with intentional Doris differences in signed BIGINT results, row-level NULL
precedence, and error types. This is not Trino's interface: no match returns
`0` rather than `-1`, positions are byte offsets, and up to seven arguments are
supported. No storage or wire-format changes. Use the new function after
participating FE/BE nodes are upgraded; mixed-version execution has not been
tested.
- **Behavior change:** Adds the four SQL names; existing function behavior
is unchanged.
```sql
SELECT regexp_position('abcabc', 'b'); -- 2
SELECT regexp_position('abcabc', 'b', 1, 2); -- 5
SELECT regexp_position('abcabc', '(b)', 1, 2, 1, '', 1); -- 6
SELECT regexp_position('abc', 'z'); -- 0
```
### Testing
Completed locally on commit `488127868371c966f0e7d733325295b9776771a3`,
based on Apache master `adad294163c23b5f7a2da81c9ff5368af14f6d58`:
- `BUILD_TYPE=ASAN ./build.sh --be --fe --disable-dynamic-arch -j 4`:
passed; FE Checkstyle reported zero violations.
- `./run-fe-ut.sh --run
org.apache.doris.nereids.rules.analysis.RegexpPositionTest`: 10 tests passed,
no failures/errors/skips.
- `./run-be-ut.sh --run --filter='FunctionRegexpPositionTest.*' -j 4`:
rebuilt the test binary; 15 tests passed, no failures/errors/skips.
- `./run-regression-test.sh --run -d nereids_function_p0 -s
regexp_position`: one suite passed, no failures/fatal errors/skips. FE and BE
were started from the newly built output artifacts; commit identity, process
paths, and artifact hashes were verified.
- `build-support/check-format.sh` with clang-format 16.0.6 and
`build-support/check-build-hygiene.sh`: passed.
- clang-tidy: the new test file passed. The production-file invocation exits
nonzero for four unchanged diagnostics: existing `stddef.h` usage and three
Boost enum-cast diagnostics. No new diagnostics compared with the previous run.
Tests cover aliases/defaults/types, constant and nullable columns,
NULL/error precedence, multiple blocks, invalid values, UTF-8 byte offsets,
embedded NUL/invalid UTF-8, flags, capture bounds, and the reference
empty/zero-width cases. These are local results; CI results remain pending.
### Notes for Reviewers
#51351 was closed after inactivity; #51462 and #55074 were not merged.
#70548 records this implementation's distinct scope and compatibility choices.
Please review byte-offset and zero-width behavior explicitly. Performance
benchmarks and mixed-version tests have not been run.
### Release note
Add `regexp_position` and the `regexp_instr`, `regexpPosition`, and
`regexpInstr` aliases to return regex match or capture-group byte positions,
with optional start position, occurrence, return option, and matching flags.
### Check List (For Author)
- Behavior changed: Yes (see Details).
- Does this need documentation: Yes;
https://github.com/apache/doris-website/pull/4634 (open draft, not merged).
- Pre-submission status: Partially verified: local ASAN build, targeted
FE/BE unit tests, and SQL regression passed. CI and documentation merge are
pending; performance and mixed-version testing were not performed. Production
clang-tidy retains the four existing diagnostics described above.
### AI Disclosure
- Tool: OpenAI Codex.
- Generated: AI assistance was used for implementation-related work,
test/build execution and analysis, branch consolidation, the linked feature
issue, and this PR description. Precise attribution for individual
implementation lines is not recorded here.
- Human review: The author directed the workflow; the extent of manual
review of AI-assisted code and tests has not been confirmed in this session.
Agent review and local validation are described above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
