andygrove opened a new issue, #256:
URL: https://github.com/apache/datafusion-spark/issues/256

   Part of #174 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `regexp_replace(str, regexp, rep[, position])` replaces every substring of 
`str` that matches the Java regular expression `regexp` with `rep`, starting 
the search at the 1-based `position` (default 1). Spark loops `Matcher.find()` 
/ `appendReplacement` / `appendTail`, so all matches are replaced (there is no 
"first only" mode and no flags argument). Available since 1.5.0.
   
   `str`, `regexp` and `rep` are implicitly cast to STRING and `position` to 
INT. `position` must be foldable and greater than 0, otherwise analysis fails. 
Result type is the type of `str` (STRING, with its collation in 4.0+). NULL if 
any argument is NULL (including a NULL `position`).
   
   ### Examples
   
   ```sql
   SELECT regexp_replace('100-200', '(\\d+)', 'num');
   -- num-num
   SELECT regexp_replace('100-200', r'(\d+)', 'num');
   -- num-num
   SELECT regexp_replace('healthy, wealthy, and wise', '\\w+thy', 'something', 
2);
   -- hsomething, something, and wise
   SELECT regexp_replace('healthy, wealthy, and wise', '\\w+thy', 'something', 
8);
   -- healthy, something, and wise
   SELECT regexp_replace('healthy, wealthy, and wise', '\\w', 'something', 26);
   -- healthy, wealthy, and wissomething
   SELECT regexp_replace('healthy, wealthy, and wise', '\\w', 'something', 27);
   -- healthy, wealthy, and wise
   SELECT regexp_replace('healthy, wealthy, and wise', '\\w', 'something', 
null);
   -- NULL
   SELECT regexp_replace('', '^$', '<empty string>');
   -- <empty string>
   SELECT regexp_replace('healthy, wealthy, and wise', '\\w+thy', 'something', 
0);
   -- error DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE
   ```
   
   ### Edge cases to test
   
   - NULL in each of the four arguments, literal and column, returns NULL.
   - `position`: 0 or negative fails analysis with 
`DATATYPE_MISMATCH.VALUE_OUT_OF_RANGE` (valueRange `(0, 2147483647]`); a 
non-foldable `position` fails with `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`. If 
`position - 1 >= length(str)` (and `position != 1`) the input is returned 
unchanged. With `str = ''` and position 1 the pattern is still applied (see the 
`^$` example).
   - `position` and the length check are in UTF-16 code units (`String.length`, 
`Matcher.region`), not code points. A supplementary character before the start 
shifts the position: `regexp_replace('😀ab', 'a', 'X', 3)` starts at `a` (the 
emoji is two UTF-16 units) and returns `😀Xb`, while a code-point-based port 
returns `😀ab`.
   - The search region uses Java's default anchoring bounds, so `^` matches at 
the region start: `regexp_replace('abc', '^b', 'X', 2)` returns `aXc`. Rust's 
`find_at` does not treat the start offset as `^`, so a naive port returns `abc`.
   - Empty pattern: matches at every position. `regexp_replace('abc', '', '-')` 
gives `-a-b-c-`.
   - Empty-match iteration: Java reports an empty match immediately after a 
non-empty match; the `regex` crate does not (its `testdata/empty.toml` gives 3 
matches for `b|` on `abc`, Java finds 4). So `regexp_replace('abc', 'b|', '-')` 
differs.
   - Replacement string uses Java `Matcher.appendReplacement` syntax:
     - `$n` and `${name}` refer to groups: `regexp_replace('100-200', '(\\d+)', 
'<$1>')` gives `<100>-<200>`.
     - `\` escapes the next character, so `\$` is a literal `$` and `\1` is a 
literal `1` (not a backreference as in DataFusion's `regexp_replace`).
     - Java parses `$12` as group 12 only if it exists, otherwise group 1 
followed by `2`; `$1a` is group 1 followed by `a`. Rust treats `$1a` as a group 
named `1a` and `$12` as group 12, and substitutes an empty string for missing 
groups.
     - A reference to a missing group or a trailing `$` fails. 4.0+: 
`INVALID_REGEXP_REPLACE` (SQLSTATE 22023) `Could not perform regexp_replace for 
source = "first last", pattern = "(?<first>[a-zA-Z]+) (?<last>[a-zA-Z]+)", 
replacement = "$3 $1" and position = 1.` (cause message contains `No group 3`, 
`StringFunctionsSuite`). 3.4/3.5 throw the raw Java `IndexOutOfBoundsException` 
/ `IllegalArgumentException`.
   - Invalid pattern: `INVALID_PARAMETER_VALUE.PATTERN`, e.g. 
`regexp_replace('', '[a\\d]{0, 2}', 'x')` (`StringFunctionsSuite`, 
SPARK-44905). Applies to literal and column patterns.
   - Column patterns and column replacements (Spark caches the last pattern and 
replacement; results must not leak between rows).
   - `spark.sql.parser.escapedStringLiterals` (default false) changes parsing 
of the SQL literals for `regexp` and `rep` before they reach the function. In 
`.slt` files, DataFusion keeps backslashes literal, so Spark `'(\\d+)'` is 
`'(\d+)'`.
   - Java vs Rust pattern divergences (decline or test each): lookaround / 
backreferences / atomic groups / possessive quantifiers (Java only); ASCII `\d` 
`\w` `\s` in Java vs Unicode in Rust (`regexp_replace('a\u0663', '\\d', 'X')` 
(U+0663 ARABIC-INDIC DIGIT THREE) is unchanged in Spark); `.` excludes `\r` 
`\u0085` `\u2028` `\u2029` in Java; `$` matches before a final line terminator 
in Java (`regexp_replace('abc\n', 'c$', 'X')` gives `abX\n`); `(?i)` is 
ASCII-only in Java (U+212A KELVIN SIGN does not match `(?i)k`); different 
inline flag letters (`U`, `u`, `d`); Java-only `\Q..\E`, `\h`, `\R`, `\Z`, 
octal; Rust-only `(?P<n>)`, `\<`, `[[:alpha:]]`.
   - `UTF8_LCASE` collation (4.0+) compiles with `CASE_INSENSITIVE | 
UNICODE_CASE`.
   
   ### Differences across Spark versions
   
   - 4.0: replacement errors wrapped as `INVALID_REGEXP_REPLACE` (raw Java 
exceptions in 3.4/3.5); collation-aware input types; return type follows `str` 
(was always STRING).
   - `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT` `inputName` is `position` in 
3.4/3.5 and `` `position` `` in 4.0+.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
 `RegExpReplace` (pattern compile in `RegExpUtils.getPatternAndLastRegex`).
   - DataFusion core `regexp_replace(str, regexp, replacement[, flags])` 
(`datafusion/functions/src/regex/regexpreplace.rs`) differs in every respect 
that matters: the 4th argument is flags, only the first match is replaced 
unless the `g` flag is given, `\1` in the replacement is rewritten to `${1}`, 
and it uses Rust regex syntax. It should not be wrapped as-is.
   - Use the shared Java-regex compatibility helper proposed for `rlike` 
(verified subset, decline otherwise, with the decision made at planning time 
for literal patterns). Also translate the replacement string from Java syntax 
and decline replacement strings whose meaning cannot be reproduced exactly 
(escape sequences, ambiguous `$nn`).
   - Belongs in `crates/spark/src/function/string/`. Add 
`crates/sqllogictest/test_files/string/regexp_replace.slt` and document with 
`#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to