andygrove opened a new issue, #258:
URL: https://github.com/apache/datafusion-spark/issues/258

   Part of #174 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `regexp_count(str, regexp)` returns the number of times the Java regular 
expression `regexp` matches in `str`. It is `RuntimeReplaceable`: the analyzer 
rewrites it to `size(regexp_extract_all(str, regexp, 0), legacySizeOfNull = 
false)`, so it counts the matches found by repeated `Matcher.find()`. Available 
since 3.4.0.
   
   Both arguments are implicitly cast to STRING. Result is INT. NULL if either 
argument is NULL. There are no `start` or `flags` arguments.
   
   ### Examples
   
   ```sql
   SELECT regexp_count('Steven Jones and Stephen Smith are the best players', 
'Ste(v|ph)en');
   -- 2
   SELECT regexp_count('abcdefghijklmnopqrstuvwxyz', '[a-z]{3}');
   -- 8
   SELECT regexp_count('1a 2b 14m', '\\d+');
   -- 3
   SELECT regexp_count('the fox', 'FOX');
   -- 0
   SELECT regexp_count('the fox', '(?i)FOX');
   -- 1
   SELECT regexp_count('passwd7 plain A1234 a1234', '(?=[^ ]*[a-z])(?=[^ 
]*[0-9])[^ ]+');
   -- 2
   SELECT regexp_count(null, 'abc');
   -- NULL
   ```
   
   ### Edge cases to test
   
   - NULL `str` or NULL `regexp`, literal and column: NULL.
   - Matches are non-overlapping: `'[a-z]{3}'` over 26 letters is 8.
   - Empty pattern: `regexp_count('abc', '')` is 4 (one empty match at each of 
the 4 positions). `regexp_count('', '')` is 1.
   - Empty matches after a non-empty match: Java counts them, the `regex` crate 
does not (its `testdata/empty.toml` gives 3 matches for `b|` on `abc`; Java 
finds 4). `regexp_count('1a 2b 14m', '(\\d+)?')` is 9 in Spark (the same 
matches as the `regexp_extract_all` golden result 
`["1","","","2","","","14","",""]`).
   - Invalid pattern: `INVALID_PARAMETER_VALUE.PATTERN`. Because of the 
rewrite, the `functionName` in the error is `` `regexp_extract_all` ``, not 
`regexp_count`.
   - Column patterns (different per row).
   - `spark.sql.parser.escapedStringLiterals` (default false) only changes how 
the SQL literal reaches the function. In `.slt`, DataFusion keeps backslashes 
literal.
   - Java vs Rust divergences to decline or test: lookaround (the `passwd7` 
example above needs it), backreferences, atomic groups, possessive quantifiers; 
ASCII `\d` `\w` `\s` in Java vs Unicode in Rust (`regexp_count('1\u0663', 
'\\d')`, U+0663 ARABIC-INDIC DIGIT THREE, is 1 in Spark); `.` excludes `\r`, 
`\u0085`, `\u2028`, `\u2029` in Java; `$` matches before a final line 
terminator in Java (`regexp_count('a\n', 'a$')` is 1); `(?i)` ASCII-only in 
Java (U+212A KELVIN SIGN does not match `(?i)k`); inline flag letters `U`, `u`, 
`d`; Java-only `\Q..\E`, `\h`, `\R`, `\Z`; Rust-only `(?P<n>)`, `\<`, 
`[[:alpha:]]`.
   - `UTF8_LCASE` collation (4.0+) counts case-insensitively.
   
   ### Differences across Spark versions
   
   - 4.0: collation-aware input types. Otherwise none found (3.4.0 through 
4.2.0).
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
 `RegExpCount` (replacement uses `RegExpExtractAll` and `Size`).
   - Because of the rewrite, Spark plans (and Comet) never contain 
`RegExpCount`; engines that translate Spark plans will see 
`size(regexp_extract_all(...))`. A direct `regexp_count` UDF is still useful 
for SQL users of `datafusion-spark` and should share the matching core with 
`regexp_extract_all`.
   - DataFusion core `regexp_count(str, regexp[, start[, flags]])` 
(`datafusion/functions/src/regex/regexpcount.rs`) returns BIGINT, accepts 
`start` and `flags`, and uses Rust regex syntax and Rust empty-match iteration. 
Do not wrap it unchanged.
   - Use the shared Java-regex compatibility helper proposed for `rlike` 
(verified subset, decline otherwise, decided at planning time for literal 
patterns).
   - Belongs in `crates/spark/src/function/string/`. Add 
`crates/sqllogictest/test_files/string/regexp_count.slt` and document with 
`#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to