andygrove opened a new issue, #258:
URL: https://github.com/apache/datafusion-spark/issues/258
Part of #174 and apache/datafusion-comet#7300.
### What the function does
`regexp_count(str, regexp)` returns the number of times the Java regular
expression `regexp` matches in `str`. It is `RuntimeReplaceable`: the analyzer
rewrites it to `size(regexp_extract_all(str, regexp, 0), legacySizeOfNull =
false)`, so it counts the matches found by repeated `Matcher.find()`. Available
since 3.4.0.
Both arguments are implicitly cast to STRING. Result is INT. NULL if either
argument is NULL. There are no `start` or `flags` arguments.
### Examples
```sql
SELECT regexp_count('Steven Jones and Stephen Smith are the best players',
'Ste(v|ph)en');
-- 2
SELECT regexp_count('abcdefghijklmnopqrstuvwxyz', '[a-z]{3}');
-- 8
SELECT regexp_count('1a 2b 14m', '\\d+');
-- 3
SELECT regexp_count('the fox', 'FOX');
-- 0
SELECT regexp_count('the fox', '(?i)FOX');
-- 1
SELECT regexp_count('passwd7 plain A1234 a1234', '(?=[^ ]*[a-z])(?=[^
]*[0-9])[^ ]+');
-- 2
SELECT regexp_count(null, 'abc');
-- NULL
```
### Edge cases to test
- NULL `str` or NULL `regexp`, literal and column: NULL.
- Matches are non-overlapping: `'[a-z]{3}'` over 26 letters is 8.
- Empty pattern: `regexp_count('abc', '')` is 4 (one empty match at each of
the 4 positions). `regexp_count('', '')` is 1.
- Empty matches after a non-empty match: Java counts them, the `regex` crate
does not (its `testdata/empty.toml` gives 3 matches for `b|` on `abc`; Java
finds 4). `regexp_count('1a 2b 14m', '(\\d+)?')` is 9 in Spark (the same
matches as the `regexp_extract_all` golden result
`["1","","","2","","","14","",""]`).
- Invalid pattern: `INVALID_PARAMETER_VALUE.PATTERN`. Because of the
rewrite, the `functionName` in the error is `` `regexp_extract_all` ``, not
`regexp_count`.
- Column patterns (different per row).
- `spark.sql.parser.escapedStringLiterals` (default false) only changes how
the SQL literal reaches the function. In `.slt`, DataFusion keeps backslashes
literal.
- Java vs Rust divergences to decline or test: lookaround (the `passwd7`
example above needs it), backreferences, atomic groups, possessive quantifiers;
ASCII `\d` `\w` `\s` in Java vs Unicode in Rust (`regexp_count('1\u0663',
'\\d')`, U+0663 ARABIC-INDIC DIGIT THREE, is 1 in Spark); `.` excludes `\r`,
`\u0085`, `\u2028`, `\u2029` in Java; `$` matches before a final line
terminator in Java (`regexp_count('a\n', 'a$')` is 1); `(?i)` ASCII-only in
Java (U+212A KELVIN SIGN does not match `(?i)k`); inline flag letters `U`, `u`,
`d`; Java-only `\Q..\E`, `\h`, `\R`, `\Z`; Rust-only `(?P<n>)`, `\<`,
`[[:alpha:]]`.
- `UTF8_LCASE` collation (4.0+) counts case-insensitively.
### Differences across Spark versions
- 4.0: collation-aware input types. Otherwise none found (3.4.0 through
4.2.0).
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
`RegExpCount` (replacement uses `RegExpExtractAll` and `Size`).
- Because of the rewrite, Spark plans (and Comet) never contain
`RegExpCount`; engines that translate Spark plans will see
`size(regexp_extract_all(...))`. A direct `regexp_count` UDF is still useful
for SQL users of `datafusion-spark` and should share the matching core with
`regexp_extract_all`.
- DataFusion core `regexp_count(str, regexp[, start[, flags]])`
(`datafusion/functions/src/regex/regexpcount.rs`) returns BIGINT, accepts
`start` and `flags`, and uses Rust regex syntax and Rust empty-match iteration.
Do not wrap it unchanged.
- Use the shared Java-regex compatibility helper proposed for `rlike`
(verified subset, decline otherwise, decided at planning time for literal
patterns).
- Belongs in `crates/spark/src/function/string/`. Add
`crates/sqllogictest/test_files/string/regexp_count.slt` and document with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]