andygrove opened a new issue, #257:
URL: https://github.com/apache/datafusion-spark/issues/257

   Part of #174 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `regexp_extract_all(str, regexp[, idx])` returns an array with group `idx` 
of every match of the Java regular expression `regexp` in `str`. `idx` defaults 
to 1; 0 means the whole match. Spark loops `Matcher.find()` and, for each 
match, checks `0 <= idx <= groupCount` and appends `group(idx)`, or `''` when 
the group did not participate. Available since 3.1.0.
   
   `str` and `regexp` are implicitly cast to STRING and `idx` to INT. `idx` may 
be a column. Result type is `ARRAY<STRING>` (element type follows `str`'s 
collation in 4.0+, `containsNull = true` but elements are never NULL). NULL if 
any argument is NULL. No match returns an empty array.
   
   ### Examples
   
   ```sql
   SELECT regexp_extract_all('100-200, 300-400', '(\\d+)-(\\d+)', 1);
   -- ["100","300"]
   SELECT regexp_extract_all('1a 2b 14m', '(\\d+)([a-z]+)', 0);
   -- ["1a","2b","14m"]
   SELECT regexp_extract_all('1a 2b 14m', '(\\d+)([a-z]+)', 2);
   -- ["a","b","m"]
   SELECT regexp_extract_all('a 2b 14m', '(\\d+)?([a-z]+)', 1);
   -- ["","2","14"]
   SELECT regexp_extract_all('1a 2b 14m', '(\\d+)?', 1);
   -- ["1","","","2","","","14","",""]
   SELECT regexp_extract_all('1a 2b 14m', '\\d+');
   -- error INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX (groupCount 0, groupIndex 
1)
   SELECT regexp_extract_all('abc', col0, 1) FROM VALUES('], [') AS t(col0);
   -- error INVALID_PARAMETER_VALUE.PATTERN
   ```
   
   ### Edge cases to test
   
   - NULL `str`, NULL `regexp`, NULL `idx` (literal and column): NULL.
   - No match: `[]`. `regexp_extract_all('100-200,300-400,500-600', '([a-z])', 
1)` is `[]` (`RegexpExpressionsSuite`).
   - Group index validation only happens inside the match loop, so an 
out-of-range `idx` raises no error when there is no match: 
`regexp_extract_all('xyz', '\\d+', 1)` returns `[]`.
   - Out-of-range or negative `idx` with a match: 
`INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX` (SQLSTATE 22023), message ``The 
value of parameter(s) `idx` in `regexp_extract_all` is invalid: Expects group 
index between 0 and 2, but got 3.``
   - Optional group that did not participate yields `''`, not NULL (see the 
`(\\d+)?([a-z]+)` example).
   - Empty pattern: matches at every position, so `regexp_extract_all('abc', 
'', 0)` returns four empty strings.
   - Empty matches after a non-empty match: the `(\\d+)?` example above has 9 
elements because Java reports the empty match at the end of each `\d+` match. 
The `regex` crate does not report an empty match that starts where the previous 
match ended (its `testdata/empty.toml` gives 3 matches for `b|` on `abc`, Java 
finds 4), so a direct `captures_iter` port returns fewer elements. Iteration 
must follow Java's rule.
   - Invalid pattern: `INVALID_PARAMETER_VALUE.PATTERN` with `value` the quoted 
pattern, also for per-row patterns (`'] ['`, `'], ['`).
   - Non-literal pattern column with different patterns per row (Spark caches 
the last regex).
   - `spark.sql.parser.escapedStringLiterals` (default false) changes how the 
SQL literal is unescaped; the kernel sees the result. In `.slt`, DataFusion 
keeps backslashes literal (`'(\d+)'`).
   - Java vs Rust pattern divergences to decline or test: lookaround / 
backreferences / atomic / possessive (Java only; e.g. `(?=[^ ]*[0-9])[^ ]+`); 
ASCII `\d` `\w` `\s` in Java vs Unicode in Rust (`regexp_extract_all('1\u0663', 
'\\d', 0)` (U+0663 ARABIC-INDIC DIGIT THREE) is `["1"]` in Spark); `.` excludes 
`\r`, `\u0085`, `\u2028`, `\u2029` in Java; `$` matches before a final line 
terminator in Java; `(?i)` ASCII-only in Java; inline flag letters `U` / `u` / 
`d` mean different things or are unknown; named groups: Java `(?<n>...)` only, 
Rust also `(?P<n>...)`; both number named groups in order, so `groupCount` 
matches for accepted patterns.
   - Unicode / supplementary characters in `str` (Java iterates UTF-16 but 
never splits a surrogate pair for `.`).
   - `UTF8_LCASE` collation (4.0+) compiles with `CASE_INSENSITIVE | 
UNICODE_CASE`.
   
   ### Differences across Spark versions
   
   - 3.4: group index errors were `_LEGACY_ERROR_TEMP_2006` (`The specified 
group index cannot be less than zero.`) and `_LEGACY_ERROR_TEMP_2007` (`Regex 
group count is <groupCount>, but the specified group index is <groupIndex>.`), 
both `SparkIllegalArgumentException`. 3.5+ uses 
`INVALID_PARAMETER_VALUE.REGEX_GROUP_INDEX` for both cases.
   - 4.0: collation-aware inputs; array element type follows `str`.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
 `RegExpExtractAll` (base `RegExpExtractBase`, 
`RegExpExtractBase.checkGroupIndex`).
   - DataFusion core has no equivalent. `regexp_match` returns the capture 
groups of the first match only, uses Rust syntax, and returns NULL on no match.
   - Use the shared Java-regex compatibility helper proposed for `rlike`: 
accept only a verified subset, return an "unsupported pattern" error otherwise 
(at planning time for literal patterns), and implement Java's empty-match 
iteration rule. `regexp_count` and `regexp_substr` are rewritten by Spark into 
this function and `regexp_extract`, so the same core can back them.
   - Belongs in `crates/spark/src/function/string/`. Add 
`crates/sqllogictest/test_files/string/regexp_extract_all.slt` and document 
with `#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to