andygrove opened a new issue, #256:
URL: https://github.com/apache/datafusion-spark/issues/256
Part of #174 and apache/datafusion-comet#7300.
### What the function does
`regexp_replace(str, regexp, rep[, position])` replaces every substring of
`str` that matches the Java regular expression `regexp` with `rep`, starting
the search at the 1-based `position` (default 1). Spark loops `Matcher.find()`
/ `appendReplacement` / `appendTail`, so all matches are replaced (there is no
"first only" mode and no flags argument). Available since 1.5.0.
`str`, `regexp` and `rep` are implicitly cast to STRING and `position` to
INT. `position` must be foldable and greater than 0, otherwise analysis fails.
Result type is the type of `str` (STRING, with its collation in 4.0+). NULL if
any argument is NULL (including a NULL `position`).
### Examples
```sql
SELECT regexp_replace('100-200', '(\\d+)', 'num');
-- num-num
SELECT regexp_replace('100-200', r'(\d+)', 'num');
-- num-num
SELECT regexp_replace('healthy, wealthy, and wise', '\\w+thy', 'something',
2);
-- hsomething, something, and wise
SELECT regexp_replace('healthy, wealthy, and wise', '\\w+thy', 'something',
8);
-- healthy, something, and wise
SELECT regexp_replace('healthy, wealthy, and wise', '\\w', 'something', 26);
-- healthy, wealthy, and wissomething
SELECT regexp_replace('healthy, wealthy, and wise', '\\w', 'something', 27);
-- healthy, wealthy, and wise
SELECT regexp_replace('healthy, wealthy, and wise', '\\w', 'something',
null);
-- NULL
SELECT regexp_replace('', '^$', '<empty string>');
-- <empty string>
SELECT regexp_replace('healthy, wealthy, and wise', '\\w+thy', 'something',
0);
-- error DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE
```
### Edge cases to test
- NULL in each of the four arguments, literal and column, returns NULL.
- `position`: 0 or negative fails analysis with
`DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE` (valueRange `(0, 2147483647]`); a
non-foldable `position` fails with `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`. If
`position - 1 >= length(str)` (and `position != 1`) the input is returned
unchanged. With `str = ''` and position 1 the pattern is still applied (see the
`^$` example).
- `position` and the length check are in UTF-16 code units (`String.length`,
`Matcher.region`), not code points. A supplementary character before the start
shifts the position: `regexp_replace('😀ab', 'a', 'X', 3)` starts at `a` (the
emoji is two UTF-16 units) and returns `😀Xb`, while a code-point-based port
returns `😀ab`.
- The search region uses Java's default anchoring bounds, so `^` matches at
the region start: `regexp_replace('abc', '^b', 'X', 2)` returns `aXc`. Rust's
`find_at` does not treat the start offset as `^`, so a naive port returns `abc`.
- Empty pattern: matches at every position. `regexp_replace('abc', '', '-')`
gives `-a-b-c-`.
- Empty-match iteration: Java reports an empty match immediately after a
non-empty match; the `regex` crate does not (its `testdata/empty.toml` gives 3
matches for `b|` on `abc`, Java finds 4). So `regexp_replace('abc', 'b|', '-')`
differs.
- Replacement string uses Java `Matcher.appendReplacement` syntax:
- `$n` and `${name}` refer to groups: `regexp_replace('100-200', '(\\d+)',
'<$1>')` gives `<100>-<200>`.
- `\` escapes the next character, so `\$` is a literal `$` and `\1` is a
literal `1` (not a backreference as in DataFusion's `regexp_replace`).
- Java parses `$12` as group 12 only if it exists, otherwise group 1
followed by `2`; `$1a` is group 1 followed by `a`. Rust treats `$1a` as a group
named `1a` and `$12` as group 12, and substitutes an empty string for missing
groups.
- A reference to a missing group or a trailing `$` fails. 4.0+:
`INVALID_REGEXP_REPLACE` (SQLSTATE 22023) `Could not perform regexp_replace for
source = "first last", pattern = "(?<first>[a-zA-Z]+) (?<last>[a-zA-Z]+)",
replacement = "$3 $1" and position = 1.` (cause message contains `No group 3`,
`StringFunctionsSuite`). 3.4/3.5 throw the raw Java `IndexOutOfBoundsException`
/ `IllegalArgumentException`.
- Invalid pattern: `INVALID_PARAMETER_VALUE.PATTERN`, e.g.
`regexp_replace('', '[a\\d]{0, 2}', 'x')` (`StringFunctionsSuite`,
SPARK-44905). Applies to literal and column patterns.
- Column patterns and column replacements (Spark caches the last pattern and
replacement; results must not leak between rows).
- `spark.sql.parser.escapedStringLiterals` (default false) changes parsing
of the SQL literals for `regexp` and `rep` before they reach the function. In
`.slt` files, DataFusion keeps backslashes literal, so Spark `'(\\d+)'` is
`'(\d+)'`.
- Java vs Rust pattern divergences (decline or test each): lookaround /
backreferences / atomic groups / possessive quantifiers (Java only); ASCII `\d`
`\w` `\s` in Java vs Unicode in Rust (`regexp_replace('a\u0663', '\\d', 'X')`
(U+0663 ARABIC-INDIC DIGIT THREE) is unchanged in Spark); `.` excludes `\r`
`\u0085` `\u2028` `\u2029` in Java; `$` matches before a final line terminator
in Java (`regexp_replace('abc\n', 'c$', 'X')` gives `abX\n`); `(?i)` is
ASCII-only in Java (U+212A KELVIN SIGN does not match `(?i)k`); different
inline flag letters (`U`, `u`, `d`); Java-only `\Q..\E`, `\h`, `\R`, `\Z`,
octal; Rust-only `(?P<n>)`, `\<`, `[[:alpha:]]`.
- `UTF8_LCASE` collation (4.0+) compiles with `CASE_INSENSITIVE |
UNICODE_CASE`.
### Differences across Spark versions
- 4.0: replacement errors wrapped as `INVALID_REGEXP_REPLACE` (raw Java
exceptions in 3.4/3.5); collation-aware input types; return type follows `str`
(was always STRING).
- `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT` `inputName` is `position` in
3.4/3.5 and `` `position` `` in 4.0+.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
`RegExpReplace` (pattern compile in `RegExpUtils.getPatternAndLastRegex`).
- DataFusion core `regexp_replace(str, regexp, replacement[, flags])`
(`datafusion/functions/src/regex/regexpreplace.rs`) differs in every respect
that matters: the 4th argument is flags, only the first match is replaced
unless the `g` flag is given, `\1` in the replacement is rewritten to `${1}`,
and it uses Rust regex syntax. It should not be wrapped as-is.
- Use the shared Java-regex compatibility helper proposed for `rlike`
(verified subset, decline otherwise, with the decision made at planning time
for literal patterns). Also translate the replacement string from Java syntax
and decline replacement strings whose meaning cannot be reproduced exactly
(escape sequences, ambiguous `$nn`).
- Belongs in `crates/spark/src/function/string/`. Add
`crates/sqllogictest/test_files/string/regexp_replace.slt` and document with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]