Abhishek-Mohan opened a new pull request, #7710:
URL: https://github.com/apache/datafusion-comet/pull/7710

   ## Which issue does this PR close?
   
   Closes #7685.
   
   ## Rationale for this change
   
   With `spark.comet.expression.RegExpExtract.allowIncompatible` (or 
`RegExpExtractAll`) enabled, native `regexp_extract` and `regexp_extract_all` 
reject an out-of-range `idx` before looking at any row. Spark checks the index 
only on a row where the pattern matches: `RegExpExtractBase.checkGroupIndex` is 
called inside `if (m.find)` / `while (m.find)`, in both `nullSafeEval` and 
`doGenCode`, in Spark 3.5.8, 4.0.1 and 4.1.1. So on Spark, `regexp_extract(s, 
'\\d+')` over rows with no digits returns `''` for every row (and 
`regexp_extract_all` returns `[]`), while Comet fails the query.
   
   ## What changes are included in this PR?
   
   - `regexp_extract_common.rs`: when `idx` is out of range, `parse_args` scans 
the batch with `Regex::is_match` and errors only if some non-null row matches. 
Otherwise it returns a new `ParsedArgs::NoMatch`, and each function builds its 
result directly (`''` or `[]` for non-null rows, null for null rows) without 
running the regex again. A subject type other than `Utf8` / `LargeUtf8` still 
gets the existing "expects Utf8 or LargeUtf8 subject" error.
   - Checking per batch fails the same queries as Spark: Spark fails the task 
on the first matching row, and Comet fails the batch that contains it.
   - The valid-`idx` path is unchanged and does no extra work. The error 
message is unchanged.
   
   One case remains different from Spark: a matching literal (scalar) subject 
in a batch with zero rows still errors, because Comet scalar functions do not 
receive the batch row count. That case needs a non-folded literal subject and 
an empty batch, and fixing it means changing how all Comet scalar functions 
receive `number_rows`, so it is left out of this PR.
   
   ## How are these changes tested?
   
   - Rust unit tests for both functions: no match with `idx` 1 and -1, one 
matching row among non-matching rows still errors, all-null input, a scalar 
subject (matching, not matching, and null), a `LargeUtf8` subject, and an 
unsupported subject type.
   - SQL file tests in `regexp_extract_enabled.sql` and 
`regexp_extract_all_enabled.sql`, using `expect_native` so they cover the 
native path: a pattern that matches no row, a table with no digits 
(two-argument form and `idx` 1, 2 and -1), an all-null column, a literal 
subject, and an `expect_error(group index)` case where only one row matches.
   - With the native changes reverted, both `*_enabled.sql` files fail with 
"Expects group index between 0 and 0, but got 1". With the fix, `./mvnw test 
-Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite regexp_extract"` 
passes all 5 files, and `cargo test`, `cargo fmt` and `cargo clippy -D 
warnings` pass.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to