KushShriv opened a new pull request, #13128:
URL: https://github.com/apache/gluten/pull/13128
## Problem
Gluten's Velox backend uses RE2 for regex execution. RE2 does not understand
Java's `\uXXXX`
Unicode escape syntax, so patterns such as `[\u4e00-\u9fa5]` (CJK Unified
Ideograph range) fail
to compile in RE2 and cause the entire operator to fall back to the Spark
JVM engine. This was a
confirmed regression tracked in `FallbackSuite`.
## Solution
Two-phase defence-in-depth translation of `\uXXXX` → `\x{XXXX}`:
**Phase 1 — Scala (primary).** Before a regex pattern literal is written
into the Substrait
protobuf, `ExpressionConverter` now calls a new
`withRe2PatternTranslation()` helper. This ensures
both the C++ validator *and* Velox's execution engine receive the
already-translated string. The
helper is wired into all six regex expression types: `RegExpReplace`,
`RLike`, `RegExpExtract`,
`RegExpExtractAll`, `RegExpInStr`, `StringSplit`. A new `VeloxRegexUtils`
object (in
`gluten-substrait`) provides `translateJavaPatternToRe2()` and
`translateUnicodeEscapes()`.
Patterns with no RE2 equivalent (lookaheads, lookbehinds, backreferences)
are detected and still
throw `GlutenNotSupportException`, preserving the existing graceful fallback
for those cases.
**Phase 2 — C++ (safeguard).** `validatePattern()` in `Common.cc` now calls
`translateJavaUnicodeToRe2()` before the `RE2::RE2()` compile attempt. This
is a last line of
defence for any future regex expression type added to `kRegexFunctions`
whose Scala transformer is
not yet hooked into the Phase 1 helper. The function is idempotent —
patterns already in RE2 syntax
pass through unchanged.
## Changes
| File | Change |
|---|---|
| `gluten-substrait/.../VeloxRegexUtils.scala` | New utility:
`translateJavaPatternToRe2`, `containsUnsupportedConstruct`,
`translateUnicodeEscapes` |
| `gluten-substrait/.../ExpressionConverter.scala` | New
`withRe2PatternTranslation` helper; explicit cases for `RLike`,
`RegExpExtract`, `RegExpExtractAll`; wired into all 6 regex expressions |
| `cpp/velox/utils/Common.h` | Declaration of `translateJavaUnicodeToRe2` |
| `cpp/velox/utils/Common.cc` | Implementation of
`translateJavaUnicodeToRe2`; updated `validatePattern` |
| `cpp/velox/tests/CommonTest.cc` | New C++ unit tests for
`translateJavaUnicodeToRe2` and `validatePattern` |
| `cpp/velox/tests/CMakeLists.txt` | Register `common_test` target |
| `backends-velox/.../FallbackSuite.scala` | Invert the `\u4e00` test —
asserts native execution now, not fallback |
| `backends-velox/.../VeloxStringFunctionsSuite.scala` | Two new regression
tests: Unicode escape runs natively; lookbehind still falls back |
## Testing
- `FallbackSuite` — existing test updated; `\uXXXX` patterns in join
post-filters now run natively
- `VeloxStringFunctionsSuite` — new test confirms `[\u0041-\u005A]+` uses
`ProjectExecTransformer`; new test confirms `(?<=x)y` still falls back
gracefully
- `CommonTest.cc` — 13 unit tests covering translation correctness and
`validatePattern` end-to-end
## Notes
- No new configuration keys introduced
- No behaviour change for patterns that were already RE2-compatible
- No behaviour change for unsupported constructs (lookaheads, lookbehinds,
backreferences) — those still fall back via `GlutenNotSupportException`
- Character class union/intersection (`[a[b]]`, `[a&&[b]]`) unaffected —
`ensureRegexIsCompatible` operates on the translated string, which has
identical bracket depth to the original
---
**Was this patch authored or co-authored using generative AI tooling?**
`Generated-by: Claude claude-sonnet-4-5`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
