KushShriv opened a new pull request, #13278:
URL: https://github.com/apache/gluten/pull/13278
## Problem
Velox and Bolt execute regexes with RE2, which doesn't understand Java's
`\uXXXX` escapes. A pattern like `[\u4e00-\u9fa5]` fails RE2 compilation, so
the whole operator falls back to Spark (tracked in `FallbackSuite`).
Supplementary characters written as surrogate pairs (`\uD83D\uDE00`) have the
same problem.
## Solution
Two-layer translation to RE2 syntax.
* **Scala (primary).** `ExpressionConverter.withRe2PatternTranslation`
rewrites literal patterns before they are written into the Substrait plan, for
`RegExpReplace`, `RLike`, `RegExpExtract`, `RegExpExtractAll`, `RegExpInStr`
and `StringSplit`. It uses `Re2PatternUtils.translateJavaPatternToRe2`, a
single escape-aware scanner:
* `\uXXXX` → `\x{XXXX}`; a high+low surrogate pair → one `\x{cp}`
* escape pairs are copied verbatim, so `\\u0041` (escaped backslash +
`u0041`) is left alone
* returns `None` (→ `GlutenNotSupportException` → fallback to Spark) for
lookahead/lookbehind, backreferences `\1`–`\9`, and unpaired surrogates.
Escaped or in-class occurrences (`\(?=`, `[(?=]`, `\\1`) are not flagged.
* named groups `(?<n>…)` are allowed
* **Backend gating.** Translation only runs when
`SparkPlanExecApi.needsRe2PatternTranslation` is true. It is overridden to
`true` for Velox and Bolt (both RE2-based) and defaults to `false`, so
ClickHouse is unaffected.
Bolt has no C++ translation pass and relies on the Scala-side rewrite alone.
* **C++ (Velox safeguard).** `validatePattern()` calls
`translateJavaUnicodeToRe2()` before compiling with RE2, with the same
escape-pair and surrogate handling. It is idempotent. An unpaired surrogate is
left untranslated so RE2 rejects it.
## Changes
| File | Change |
|---|---|
| `gluten-substrait/.../Re2PatternUtils.scala` | New translator/validator |
| `gluten-substrait/.../ExpressionConverter.scala` |
`withRe2PatternTranslation`; explicit `RLike` / `RegExpExtract` /
`RegExpExtractAll` cases; wired into all 6 regex expressions |
| `gluten-substrait/.../SparkPlanExecApi.scala` |
`needsRe2PatternTranslation` hook (default `false`) |
| `backends-velox/.../VeloxSparkPlanExecApi.scala`,
`backends-bolt/.../BoltSparkPlanExecApi.scala` | Override to `true` |
| `cpp/velox/utils/Common.{h,cc}` | `translateJavaUnicodeToRe2`;
`validatePattern` calls it |
| `cpp/velox/tests/CommonTest.cc`, `CMakeLists.txt` | 20 C++ tests; register
`common_test` |
| `gluten-substrait/.../Re2PatternUtilsSuite.scala` | 8 unit tests |
| `backends-velox/.../FallbackSuite.scala` | `\u4e00` join-filter test now
asserts native execution; lookbehind fallback assertion tightened to the
specific reason |
| `backends-velox/.../VeloxStringFunctionsSuite.scala` | rlike tests:
Unicode escape, surrogate pair, escaped backslash run natively; lookbehind
falls back |
## Testing
- `Re2PatternUtilsSuite`: translation, surrogate pairs, escaped backslashes,
unsupported-construct detection
- `CommonTest.cc`: translation and `validatePattern` end-to-end
- `VeloxStringFunctionsSuite` / `FallbackSuite`: end-to-end native vs.
fallback, compared with vanilla Spark
## Notes
- No new config keys. Already RE2-compatible patterns are unchanged.
- Lookaround/backreference patterns still fall back to Spark on Velox and
Bolt.
- Character class union/intersection (`[a[b]]`, `[a&&[b]]`) is unaffected:
`ensureRegexIsCompatible` sees a string with the same bracket depth.
---
**Was this patch authored or co-authored using generative AI tooling?**
`Generated-by: Claude claude-sonnet-5-5`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
