andygrove opened a new issue, #7685:
URL: https://github.com/apache/datafusion-comet/issues/7685
### Describe the bug
The native `regexp_extract` and `regexp_extract_all` validate `idx` against
the pattern's group count before they look at any row. Spark validates it only
for a row where the pattern matches.
- `RegExpExtract.nullSafeEval` calls `checkGroupIndex` inside `if (m.find)`.
A row with no match returns `''`.
- `RegExpExtractAll.nullSafeEval` calls `checkGroupIndex` inside `while
(m.find)`. A row with no match returns an empty array.
In Comet, `parse_args` in
`native/spark-expr/src/string_funcs/regexp_extract_common.rs` computes
`group_count` and returns an error for `idx < 0 || idx > group_count` before
any row is evaluated. Both functions share it.
So a query where Spark succeeds fails in Comet. For example,
`regexp_extract(s, '\d+')` uses the default `idx` of 1 and the pattern has no
groups. Spark returns `''` for every row that does not match, and errors only
if some row matches. Comet errors on the whole query, even when no row matches.
The native path is reached only with
`spark.comet.expression.RegExpExtract.allowIncompatible=true` (or
`RegExpExtractAll`) and a literal pattern and `idx`. The default codegen
dispatcher path runs Spark's own code and is not affected.
Found while reading the review of the same bug in the `datafusion-spark`
port of this function (apache/datafusion-spark#73, since closed). I confirmed
the Spark behavior in the 4.2.0 source. I have not run the repro below against
Comet. The Comet side comes from reading the code.
### Steps to reproduce
```sql
-- spark.comet.expression.RegExpExtract.allowIncompatible=true
CREATE TABLE t(s string) USING parquet;
INSERT INTO t VALUES ('abc'), ('def');
SELECT regexp_extract(s, '\\d+') FROM t;
-- Spark: '', ''
-- Comet: error, "Expects group index between 0 and 0, but got 1."
```
The same applies to `regexp_extract_all` with an out-of-range `idx` and no
matching row.
### Expected behavior
Validate `idx` only when a row matches, as Spark does. In `extract_array`
and `extract_one`, check `group_idx` only after `captures_read` or `captures`
returns a match. Return the empty result otherwise.
Add tests in `regexp_extract.rs` and `regexp_extract_all.rs` for:
- no match with an out-of-range `idx`, which returns the empty result
- a match with an out-of-range `idx`, which still errors
- a column where only some rows match
### Additional context
A related difference is that an invalid literal pattern is compiled eagerly,
so a query over an all-NULL subject column errors in Comet. Spark evaluates a
NULL subject row without compiling the pattern and returns NULL. Fixing that
means compiling lazily on the first non-NULL row. It could go in the same
change or be tracked separately.
The doc comment on `spark_regexp_extract` says "Errors when `idx` is out of
range" and does not mention the match condition. It should be updated with the
fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]