viirya opened a new pull request, #59994:
URL: https://github.com/apache/spark/pull/59994
### What changes were proposed in this pull request?
This is a backport of #59879 to `branch-4.2`.
It fixes the `expensive` heuristic of LIKE/RLIKE added by SPARK-47672, which
matched on the subject string instead of the pattern and so classed every
LIKE/RLIKE over a column as expensive. A constant pattern is now classed by its
backtracking quantifiers. A new internal config,
`spark.sql.optimizer.regexPatternCostEstimateVersion`, is recorded in the
streaming offset log, so a query resumed from an existing checkpoint keeps its
filter placement. See #59879 for the full description.
Differences from the `master` commit:
- `branch-4.2` does not have SPARK-55014
(`splitProjectForExpensiveConditions`), so `PushPredicateThroughNonJoin` uses
`isExpensive` only in `isCheap`, and the `FilterPushdownSuite` case for
splitting the projection between two LIKE/RLIKE aliases with version `0` is
left out.
- The SPARK-47672 and SPARK-60602 tests in `FilterPushdownSuite` that used
`$"e".rlike("magic")` as their expensive stand-in now use `expensiveRLike`, as
on `master`.
- `StreamingQueryManagerSuite` gets the `rewriteOffsetLogConf` helper, which
`master` has from an earlier change.
### Why are the changes needed?
With the default config, a filter on an alias of a LIKE/RLIKE expression is
not pushed below the Project, which loses partition pruning and data filter
pushdown. This is a regression in 4.2.0.
### Does this PR introduce _any_ user-facing change?
Yes: filters on aliased LIKE/RLIKE expressions with a cheap constant pattern
are pushed down again, as they were before 4.2.0. Query results do not change.
Streaming queries resumed from an existing checkpoint keep their filter
placement.
### How was this patch tested?
`catalyst/testOnly *RegexpExpressionsSuite *FilterPushdownSuite` and
`sql/testOnly *OffsetSeqLogSuite *StreamingQueryManagerSuite
*FileSourceStrategySuite` on JDK 17, and scalastyle.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
