viirya opened a new pull request, #59879:
URL: https://github.com/apache/spark/pull/59879
### What changes were proposed in this pull request?
SPARK-47672 added an `expensive` heuristic to `StringRegexExpression`
(LIKE/RLIKE), used by `PushPredicateThroughNonJoin` to avoid evaluating an
expensive alias twice when `spark.sql.optimizer.avoidDoubleFilterEval` is true
(the default). The heuristic had these bugs:
1. It matched on `left`, the subject string, instead of `right`, the
pattern. For the usual `col LIKE/RLIKE 'literal'` it fell into `case _ =>
true`, so every LIKE/RLIKE over a column was classed expensive, including
`'^1'`, which the code's own comment says should be cheap.
2. The backtracking check
`Pattern.compile("\\+\\*\\{").matcher(str).matches()` only matches the literal
string `+*{`, so it never fired.
3. The regex-specific `^` / `\b` anchor checks were also applied to LIKE
patterns.
4. The detection `Pattern` was a per-instance `val`, compiled for every
Like/RLike.
This PR makes the heuristic look at the pattern:
- A pattern that is not foldable is expensive, since it is compiled for
every row.
- A foldable pattern that evaluates to null is cheap.
- A constant LIKE pattern is cheap: `%` and `_` do not nest.
- A constant RLIKE pattern is expensive when it does not start with `^` or
`\b` and contains a quantifier (`[+*{]`, now checked with `find()`). This keeps
the original intent of the heuristic.
- The detection `Pattern` moves to `object StringRegexExpression`.
The SPARK-47672 and SPARK-55014 tests in `FilterPushdownSuite` used
`$"e".rlike("magic")` as their expensive stand-in and only passed because of
bug 1. They now use a helper, `expensiveRLike(tag)`, which is an RLIKE with a
non-constant pattern, so they keep testing the rule.
### Why are the changes needed?
With the default config, a filter on an alias of a LIKE/RLIKE expression is
not pushed below the Project, which loses partition pruning and data filter
pushdown. For example, over a Parquet table partitioned by `part`:
```sql
SELECT * FROM (SELECT name, part RLIKE '^1' AS p FROM t) WHERE p
```
The scan gets `PartitionFilters: []`, whereas with
`spark.sql.optimizer.avoidDoubleFilterEval=false` it prunes partitions. The
same happens for `part LIKE '1_' OR part = '1'` and for data filters such as
`name RLIKE '^1'`. `LIKE 'x%'` is not affected, because `LikeSimplification`
rewrites it to `StartsWith` first.
### Does this PR introduce _any_ user-facing change?
Yes, compared with 4.2.0: filters on aliased LIKE/RLIKE expressions with a
cheap constant pattern are pushed down again, as they were before 4.2.0, so
partition pruning and data filter pushdown work for them. Query results do not
change.
### How was this patch tested?
New tests:
- `RegexpExpressionsSuite`: the heuristic for constant, foldable, null and
non-foldable patterns.
- `FilterPushdownSuite`: a filter on a cheap LIKE/RLIKE alias is pushed; one
on a backtracking RLIKE alias is not.
- `FileSourceStrategySuite`: partition filters and data filters on aliased
LIKE/RLIKE reach the Parquet scan. This test fails without the fix.
Existing tests: `catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.*
*RegexpExpressionsSuite`, and `sql/testOnly *FileSourceStrategySuite
*DataFrameSuite *PruneFileSourcePartitionsSuite *StringFunctionsSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
