Abhishek-Mohan opened a new issue, #7711:
URL: https://github.com/apache/datafusion-comet/issues/7711

   ### Describe the bug
   
   The native `regexp_extract` and `regexp_extract_all` compile the pattern at 
the start of every batch, before they look at any row. Spark compiles it only 
when it evaluates a row whose inputs are all non-null.
   
   - `RegExpExtractBase` is a null-intolerant `TernaryExpression`, so 
`nullSafeEval` runs only for a non-null subject. The pattern is compiled inside 
it, in `getLastMatcher` (`RegExpUtils.getPatternAndLastRegex`). The generated 
code does the same thing: `RegExpUtils.initLastMatcherCode` sits inside 
`nullSafeCodeGen`. This is the same in Spark 3.5.8, 4.0.1 and 4.1.1.
   - In Comet, `parse_args` in 
`native/spark-expr/src/string_funcs/regexp_extract_common.rs` calls 
`PatternCache::get_or_compile` and returns `The value of parameter `regexp` in 
`regexp_extract` is invalid: ...` before any row is evaluated. Both functions 
share it.
   
   So when the subject column is entirely null, Spark returns null for every 
row whatever the pattern, while Comet fails the query if the pattern does not 
compile in the Rust `regex` crate. That covers patterns invalid in both engines 
(for example an unclosed group) and patterns that are valid in Java but 
rejected by Rust (for example lookahead), which Spark would never reach on an 
all-null column.
   
   The native path is used only with 
`spark.comet.expression.RegExpExtract.allowIncompatible=true` (or 
`RegExpExtractAll`) and a literal pattern and `idx`. The default codegen 
dispatcher path runs Spark's own code and is not affected.
   
   ### Steps to reproduce
   
   ```sql
   -- spark.comet.expression.RegExpExtract.allowIncompatible=true
   -- spark.comet.expression.RegExpExtractAll.allowIncompatible=true
   CREATE TABLE t(s string) USING parquet;
   INSERT INTO t VALUES (NULL), (NULL);
   SELECT regexp_extract(s, '(unclosed', 1) FROM t;
   SELECT regexp_extract_all(s, '(unclosed', 1) FROM t;
   -- Spark: NULL, NULL
   -- Comet: CometNativeException: The value of parameter `regexp` in 
`regexp_extract` is invalid: '(unclosed' (regex parse error: ...)
   ```
   
   I reproduced the Comet error with these queries in `CometSqlFileTestSuite` 
on `main` plus #7710. The Spark result comes from reading the Spark source 
cited above.
   
   ### Expected behavior
   
   Return null for every row when no subject row is non-null, without compiling 
the pattern. In `parse_args`, before compiling, return the existing 
`ParsedArgs::NullResult` when the subject array has no non-null rows 
(`null_count() == len()`) or the subject scalar is null. A batch with at least 
one non-null row keeps today's behavior and fails on an invalid pattern, which 
matches Spark failing on the first non-null row.
   
   Add Rust unit tests for an all-null array subject and a null scalar subject 
with an invalid pattern, and a SQL file test over an all-null column in 
`regexp_extract_enabled.sql` and `regexp_extract_all_enabled.sql`.
   
   ### Additional context
   
   Noted in the additional context of #7685 and split out from its fix (#7710), 
which only changes how the group index is checked.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to