david-mollitor-db opened a new pull request, #59213:
URL: https://github.com/apache/spark/pull/59213
### What changes were proposed in this pull request?
`StringSplit` (`split`) now keeps the last regex and its compiled `Pattern`
instead of compiling the regex on every row. It uses the same `lastRegex` /
`pattern` cache that `RegExpReplace` and `RegExpExtractBase` already have.
- `UTF8String`: new `split(Pattern, int)` and `splitLegacyTruncate(Pattern,
int)` overloads.
- An empty pattern delegates to the existing empty-regex branches
(SPARK-49968), so that logic isn't duplicated.
- Otherwise they split with the compiled pattern. The exception is a regex
that qualifies for `String.split`'s single-character `indexOf` fast path, on a
pattern with no flags. That case still goes through `String.split`, because the
fast path beats matching even a pre-compiled `Pattern`.
- `splitsWithoutRegex` mirrors the JDK's fast-path check. It only picks
between two calls that return the same result, since outside the fast path
`String.split` is `Pattern.compile(regex).split(this, limit)`.
- `StringSplit`: both the interpreted and the codegen paths compile
`CollationSupport.collationAwareRegex(regex, ...)` only when the regex value
changes. They then call the new overloads.
- The regex is cloned before it is cached, because the input may point
into a reused row buffer.
- `stateful` is now `true`, as for the other caching regex expressions.
- `StringFunctionsBenchmark`: adds three `split` cases.
I didn't reuse `RegExpUtils.getPatternAndLastRegex` / `initLastPatternCode`.
They compile with collation flags and wrap `PatternSyntaxException` in
`INVALID_PARAMETER_VALUE.PATTERN`. That would change two things:
- the error `split` raises for an invalid regex;
- the result for UTF8_LCASE + `spark.sql.legacy.truncateForEmptyRegexSplit`
+ an empty regex. Today the `(?ui)` prefix makes that regex non-empty, so it
takes the regex path.
Compiling the exact string `String.split` compiles today keeps both
unchanged.
### Why are the changes needed?
`UTF8String.split(UTF8String, int)` decodes the regex and calls
`String.split`, which runs `Pattern.compile(regex)` on every call unless the
regex is a single literal character. As a result, these compile a new `Pattern`
for every row even though the regex is a literal:
- `split(s, '[,;]')`
- `split(s, '\\s+')`
- `split(s, '|')`
- any UTF8_LCASE `split`, whose regex gets a `(?ui)` prefix
Local `StringFunctionsBenchmark` numbers, 1M rows like `123-246`, ns/row.
The JDK 25 numbers come from before/after runs interleaved on the same machine:
| Case | JDK 17 before | JDK 17 after | JDK 25 before | JDK 25 after |
|---|---|---|---|---|
| `split(subject, '[-]')` | 424 | 346 | 366 | 266 |
| `split(subject COLLATE UTF8_LCASE, '-')` | 477 | 353 | 395 | 281 |
| `split(subject, '-')` (single-char fast path) | 258 | 262 | 195 | 196 |
### Does this PR introduce _any_ user-facing change?
No. Results are unchanged. An invalid regex raises the same exception as
before, at the same point (the first row with non-null inputs).
### How was this patch tested?
New tests:
- `UTF8StringSuite.splitWithCompiledPattern`: checks that `split(Pattern,
int)` / `splitLegacyTruncate(Pattern, int)` match the `UTF8String` overloads
across fast-path and real-regex patterns, empty patterns and inputs, and
several limits. It also checks that a pattern compiled with `CASE_INSENSITIVE`
skips the fast path.
- `RegexpExpressionsSuite` "split refreshes its cached pattern when the
regex changes": the regex changes across rows that share a reused `UnsafeRow`
buffer, tested both interpreted and with codegen. It fails if the cached regex
isn't cloned.
Existing tests:
- `RegexpExpressionsSuite` `SPLIT`
- `CollationRegexpExpressionsSuite`
- the split tests in `StringFunctionsSuite`, `CollationSQLRegexpSuite` and
`CharVarcharTestSuite`
- `SQLQueryTestSuite` `collations-string-functions` (no golden-file changes)
The benchmark result files will be regenerated on GitHub Actions (JDK
17/21/25).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
