andygrove opened a new issue, #7685:
URL: https://github.com/apache/datafusion-comet/issues/7685

   ### Describe the bug
   
   The native `regexp_extract` and `regexp_extract_all` validate `idx` against 
the pattern's group count before they look at any row. Spark validates it only 
for a row where the pattern matches.
   
   - `RegExpExtract.nullSafeEval` calls `checkGroupIndex` inside `if (m.find)`. 
A row with no match returns `''`.
   - `RegExpExtractAll.nullSafeEval` calls `checkGroupIndex` inside `while 
(m.find)`. A row with no match returns an empty array.
   
   In Comet, `parse_args` in 
`native/spark-expr/src/string_funcs/regexp_extract_common.rs` computes 
`group_count` and returns an error for `idx < 0 || idx > group_count` before 
any row is evaluated. Both functions share it.
   
   So a query where Spark succeeds fails in Comet. For example, 
`regexp_extract(s, '\d+')` uses the default `idx` of 1 and the pattern has no 
groups. Spark returns `''` for every row that does not match, and errors only 
if some row matches. Comet errors on the whole query, even when no row matches.
   
   The native path is reached only with 
`spark.comet.expression.RegExpExtract.allowIncompatible=true` (or 
`RegExpExtractAll`) and a literal pattern and `idx`. The default codegen 
dispatcher path runs Spark's own code and is not affected.
   
   Found while reading the review of the same bug in the `datafusion-spark` 
port of this function (apache/datafusion-spark#73, since closed). I confirmed 
the Spark behavior in the 4.2.0 source. I have not run the repro below against 
Comet. The Comet side comes from reading the code.
   
   ### Steps to reproduce
   
   ```sql
   -- spark.comet.expression.RegExpExtract.allowIncompatible=true
   CREATE TABLE t(s string) USING parquet;
   INSERT INTO t VALUES ('abc'), ('def');
   SELECT regexp_extract(s, '\\d+') FROM t;
   -- Spark: '', ''
   -- Comet: error, "Expects group index between 0 and 0, but got 1."
   ```
   
   The same applies to `regexp_extract_all` with an out-of-range `idx` and no 
matching row.
   
   ### Expected behavior
   
   Validate `idx` only when a row matches, as Spark does. In `extract_array` 
and `extract_one`, check `group_idx` only after `captures_read` or `captures` 
returns a match. Return the empty result otherwise.
   
   Add tests in `regexp_extract.rs` and `regexp_extract_all.rs` for:
   
   - no match with an out-of-range `idx`, which returns the empty result
   - a match with an out-of-range `idx`, which still errors
   - a column where only some rows match
   
   ### Additional context
   
   A related difference is that an invalid literal pattern is compiled eagerly, 
so a query over an all-NULL subject column errors in Comet. Spark evaluates a 
NULL subject row without compiling the pattern and returns NULL. Fixing that 
means compiling lazily on the first non-NULL row. It could go in the same 
change or be tracked separately.
   
   The doc comment on `spark_regexp_extract` says "Errors when `idx` is out of 
range" and does not mention the match condition. It should be updated with the 
fix.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to