andygrove opened a new issue, #7725:
URL: https://github.com/apache/datafusion-comet/issues/7725
### Describe the bug
With `spark.comet.expression.RegExpExtract.allowIncompatible` (or
`RegExpExtractAll`) enabled, native `regexp_extract` and `regexp_extract_all`
can still fail on an out-of-range `idx` where Spark succeeds. This is the one
case left after #7710.
Spark checks `idx` only on a row where the pattern matches, in
`RegExpExtractBase.checkGroupIndex`. #7710 makes the native functions do the
same by scanning the batch with `Regex::is_match` and erroring only if a
non-null row matches.
A scalar subject is different. Comet scalar functions do not receive the
batch row count, so a literal subject that matches the pattern errors even when
the batch has zero rows. Spark evaluates no rows and returns an empty result.
Reaching it takes a literal subject that constant folding did not remove,
and a batch with no rows. It is rare.
### Steps to reproduce
Needs a non-folded literal subject evaluated over an empty batch, with an
out-of-range `idx`:
```sql
-- spark.comet.expression.RegExpExtract.allowIncompatible=true
SELECT regexp_extract('a1', '\\d+', 1) FROM t WHERE false;
-- Spark: no rows, no error
-- Comet: "Expects group index between 0 and 0, but got 1."
```
I have not run this against Comet. It comes from the description of #7710
and from reading `parse_args` and `any_row_matches` in
`native/spark-expr/src/string_funcs/regexp_extract_common.rs`.
### Expected behavior
Return no error when the batch has no rows. Fixing it means giving Comet
scalar functions the batch row count (`number_rows`), which touches how every
scalar function is called. The same missing row count may affect other scalar
functions that decide something per row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]