Abhishek-Mohan opened a new pull request, #7710: URL: https://github.com/apache/datafusion-comet/pull/7710
## Which issue does this PR close? Closes #7685. ## Rationale for this change With `spark.comet.expression.RegExpExtract.allowIncompatible` (or `RegExpExtractAll`) enabled, native `regexp_extract` and `regexp_extract_all` reject an out-of-range `idx` before looking at any row. Spark checks the index only on a row where the pattern matches: `RegExpExtractBase.checkGroupIndex` is called inside `if (m.find)` / `while (m.find)`, in both `nullSafeEval` and `doGenCode`, in Spark 3.5.8, 4.0.1 and 4.1.1. So on Spark, `regexp_extract(s, '\\d+')` over rows with no digits returns `''` for every row (and `regexp_extract_all` returns `[]`), while Comet fails the query. ## What changes are included in this PR? - `regexp_extract_common.rs`: when `idx` is out of range, `parse_args` scans the batch with `Regex::is_match` and errors only if some non-null row matches. Otherwise it returns a new `ParsedArgs::NoMatch`, and each function builds its result directly (`''` or `[]` for non-null rows, null for null rows) without running the regex again. A subject type other than `Utf8` / `LargeUtf8` still gets the existing "expects Utf8 or LargeUtf8 subject" error. - Checking per batch fails the same queries as Spark: Spark fails the task on the first matching row, and Comet fails the batch that contains it. - The valid-`idx` path is unchanged and does no extra work. The error message is unchanged. One case remains different from Spark: a matching literal (scalar) subject in a batch with zero rows still errors, because Comet scalar functions do not receive the batch row count. That case needs a non-folded literal subject and an empty batch, and fixing it means changing how all Comet scalar functions receive `number_rows`, so it is left out of this PR. ## How are these changes tested? - Rust unit tests for both functions: no match with `idx` 1 and -1, one matching row among non-matching rows still errors, all-null input, a scalar subject (matching, not matching, and null), a `LargeUtf8` subject, and an unsupported subject type. - SQL file tests in `regexp_extract_enabled.sql` and `regexp_extract_all_enabled.sql`, using `expect_native` so they cover the native path: a pattern that matches no row, a table with no digits (two-argument form and `idx` 1, 2 and -1), an all-null column, a literal subject, and an `expect_error(group index)` case where only one row matches. - With the native changes reverted, both `*_enabled.sql` files fail with "Expects group index between 0 and 0, but got 1". With the fix, `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite regexp_extract"` passes all 5 files, and `cargo test`, `cargo fmt` and `cargo clippy -D warnings` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
