This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 81654b27d264 [SPARK-60632][SQL] Fix the LIKE/RLIKE expensive heuristic
to look at the pattern
81654b27d264 is described below
commit 81654b27d264533dfd3d9843e26f8a8d9b2b7a2b
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Sat Oct 31 11:11:36 2026 -0700
[SPARK-60632][SQL] Fix the LIKE/RLIKE expensive heuristic to look at the
pattern
### What changes were proposed in this pull request?
SPARK-47672 added an `expensive` heuristic to `StringRegexExpression`
(LIKE/RLIKE), used by `PushPredicateThroughNonJoin` to avoid evaluating an
expensive alias twice when `spark.sql.optimizer.avoidDoubleFilterEval` is true
(the default). The heuristic had these bugs:
1. It matched on `left`, the subject string, instead of `right`, the
pattern. For the usual `col LIKE/RLIKE 'literal'` it fell into `case _ =>
true`, so every LIKE/RLIKE over a column was classed expensive, including
`'^1'`, which the code's own comment says should be cheap.
2. The backtracking check
`Pattern.compile("\\+\\*\\{").matcher(str).matches()` only matches the literal
string `+*{`, so it never fired.
3. The regex-specific `^` / `\b` anchor checks were also applied to LIKE
patterns.
4. The detection `Pattern` was a per-instance `val`, compiled for every
Like/RLike.
This PR makes the heuristic look at the pattern:
- A pattern that is not foldable is expensive, since it is compiled for
every row. So is a foldable pattern that fails to evaluate, e.g. one that
`ConstantFolding` left in a conditional branch because it throws. The pattern
is evaluated on a fresh copy, as `ConstantFolding` does.
- A foldable pattern that evaluates to null is cheap.
- A constant pattern is classed by its repeating quantifiers that can
backtrack (`*`, `+`, `?` and `{n,m}`). A failed match tries every combination
of their lengths, so each one costs a factor of the input length, and an
unanchored RLIKE (`find`) costs one more factor for trying every start
position. On JDK 17, one 400-character value took about 40 ms for LIKE
`'%a%a%b'` or RLIKE `'^.*a.*a.*b'`, but 4.2 s for RLIKE `'.*a.*a.*b'`. So:
- An RLIKE pattern is expensive with two or more quantifiers, or three or
more when it starts with `^` or `\A` (with no leading multiline flag, no
quantifier on the anchor and no top-level `|`).
- A LIKE pattern is expensive with three or more `%` that are not at the
end. It is checked by escaping it to its regex and scanning that the same way,
as a regex that must match the whole input.
- The alternatives of a top-level `|` are tried one after another, so
each is counted on its own and the largest count is used.
- A quantifier that ends the pattern or a top-level alternative, possibly
followed by the `)` of groups that end there too, is not counted, since a
failed match never backtracks into it. Possessive quantifiers are not counted
either. `?` and a `{n,m}` range spanning fewer than 10 are not counted on their
own, but their choices multiply, together with one choice of two for each group
that holds a `|`, and they count as one quantifier each time their product
reaches 32.
- A backreference, which compares as many characters as its group holds,
and a repetition of at least 100, which adds its length to every attempt even
when it cannot backtrack, count as one quantifier each, unless they end the
pattern or a top-level alternative.
- A repeated group that holds a quantifier, a small range or a `|`, such
as `^(.*?,){11}P`, `(a+)+b` or `(a|aa){20}b`, is always expensive. So is a
pattern that turns on COMMENTS mode (`(?x)`), where whitespace can separate a
group from its quantifier, and one with a `[` that is never closed. An empty
`\Q\E` between a group and its quantifier is skipped, since `Pattern` removes
it.
- Some patterns stay expensive on purpose, although they cost no more
than their cheap relatives: alternatives inside a group, such as `(a.*b|c.*d)`,
are counted together, and a top-level `|` turns off the anchored threshold even
when every alternative is anchored, such as `^a.*a.*b|^c.*c.*d`. This errs on
the side of keeping a filter above the projection.
- Escaped characters, `\Q...\E`, character classes (where a `]` right
after `[` or `[^` is a literal) and the braces of `\p{..}`, `\x{..}` and
`\N{..}` are skipped.
- `isExpensivePattern` defaults to `true`, and `Like` and `RLike` override
it.
- A new internal config,
`spark.sql.optimizer.regexPatternCostEstimateVersion` (default `1`), selects
the estimate. With `0`, every LIKE and RLIKE is expensive, as in 4.2.0 for a
subject that is not a literal. `PushPredicateThroughNonJoin` checks it rather
than `expensive`, since a parent expression caches `hasExpensiveChild`. It is
recorded in the streaming offset log, and a checkpoint without it uses `0`. A
later change that makes any pattern cheaper needs a new version. A cheap pat
[...]
The SPARK-47672 and SPARK-55014 tests in `FilterPushdownSuite` used
`$"e".rlike("magic")` as their expensive stand-in and only passed because of
bug 1. They now use a helper, `expensiveRLike(tag)`, which is an RLIKE with a
non-constant pattern, so they keep testing the rule.
### Why are the changes needed?
With the default config, a filter on an alias of a LIKE/RLIKE expression is
not pushed below the Project, which loses partition pruning and data filter
pushdown. For example, over a Parquet table partitioned by `part`:
```sql
SELECT * FROM (SELECT name, part RLIKE '^1' AS p FROM t) WHERE p
```
The scan gets `PartitionFilters: []`, whereas with
`spark.sql.optimizer.avoidDoubleFilterEval=false` it prunes partitions. The
same happens for `part LIKE '1_' OR part = '1'` and for data filters such as
`name RLIKE '^1'`. `LIKE 'x%'` is not affected, because `LikeSimplification`
rewrites it to `StartsWith` first.
### Does this PR introduce _any_ user-facing change?
Yes, compared with 4.2.0: filters on aliased LIKE/RLIKE expressions with a
cheap constant pattern are pushed down again, as they were before 4.2.0, so
partition pruning and data filter pushdown work for them. Query results do not
change. Streaming queries resumed from an existing checkpoint keep their filter
placement.
### How was this patch tested?
New tests:
- `RegexpExpressionsSuite`: the heuristic for constant (including anchored,
possessive, nested and bounded quantifiers), foldable, null, failing and
non-foldable patterns, and for an expensive subject.
- `RegexpExpressionsSuite` also covers repeated groups with ranges or
alternations, COMMENTS mode, sequences of small ranges and `?`, trailing
quantifiers, top-level alternatives, a leading `]` in a class, groups that hold
a `|`, quantified anchors, backreferences, long repetitions and an empty `\Q\E`.
- `FilterPushdownSuite`: a filter on a cheap LIKE/RLIKE/ILIKE alias is
pushed; one on a backtracking LIKE/RLIKE alias is not, and neither is a cheap
one, nor one under `NOT`, with version `0` of the estimate, which also splits
the projection between two LIKE/RLIKE aliases.
- `OffsetSeqLogSuite`: the config is recorded as `1` for new checkpoints
and read as `0` from old ones.
- `StreamingQueryManagerSuite`: a query restarted from a checkpoint without
the config keeps the filter on a `NOT (value RLIKE '^1')` alias above the
projection, where it was pushed below before the restart.
- `FileSourceStrategySuite`: partition filters and data filters on aliased
LIKE/RLIKE reach the Parquet scan and prune the expected partitions.
Existing tests: `catalyst/testOnly
org.apache.spark.sql.catalyst.optimizer.* *RegexpExpressionsSuite`, and
`sql/testOnly *OffsetSeqLogSuite *StreamingQueryManagerSuite
*FileSourceStrategySuite *DataFrameSuite *PruneFileSourcePartitionsSuite
*StringFunctionsSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59879 from viirya/SPARK-60632.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
.../catalyst/expressions/regexpExpressions.scala | 250 +++++++++++++++++++--
.../spark/sql/catalyst/optimizer/Optimizer.scala | 21 +-
.../org/apache/spark/sql/internal/SQLConf.scala | 14 ++
.../expressions/RegexpExpressionsSuite.scala | 77 +++++++
.../catalyst/optimizer/FilterPushdownSuite.scala | 199 ++++++++++------
.../streaming/checkpointing/OffsetSeq.scala | 6 +-
.../datasources/FileSourceStrategySuite.scala | 33 ++-
.../execution/streaming/OffsetSeqLogSuite.scala | 14 ++
.../sql/streaming/StreamingQueryManagerSuite.scala | 32 +++
9 files changed, 554 insertions(+), 92 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
index b16ff1a6ed41..7d3d9249886a 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala
@@ -84,33 +84,233 @@ abstract class StringRegexExpression extends
BinaryExpression
}
}
- override def expensive: Boolean = hasExpensiveChild || _expensiveRegex
-
- // Heuristic, not designed to be perfect. Look for things likely to have
- // back tracking.
- private val detectExpensiveRegexPattern = Pattern.compile("\\+\\*\\{")
-
- private lazy val _expensiveRegex = {
- // A quick heuristic for expensive a pattern is.
- left match {
- case StringLiteral(str) =>
- // If we have a clear start limited back tracking required.
- if (str.startsWith("^") || str.startsWith("\\b")) {
- false
- } else if (detectExpensiveRegexPattern.matcher(str).matches()) {
- // Greedy matching can be tricky.
- true
- } else {
- // Default to pushdown for now.
- false
- }
- case _ =>
- true // per row regex compilation.
+ override def expensive: Boolean = hasExpensiveChild || _expensivePattern
+
+ // Heuristic, not designed to be perfect. A pattern that is not foldable is
compiled for every
+ // row; a foldable one is left to `isExpensivePattern`. Like
`ConstantFolding`, evaluate a fresh
+ // copy so that a stateful pattern keeps its state.
+ private lazy val _expensivePattern = !right.foldable || {
+ try {
+ val str = right.freshCopyIfContainsStatefulExpression().eval()
+ str != null && isExpensivePattern(str.toString)
+ } catch {
+ // E.g. a pattern that `ConstantFolding` left in a conditional branch
because it throws.
+ case NonFatal(_) => true
}
}
+
+ /** Whether matching against the constant pattern `str` is likely to be
expensive. */
+ protected def isExpensivePattern(str: String): Boolean = true
}
private[catalyst] object StringRegexExpression {
+ /**
+ * Whether matching `regex` is likely to be expensive. A failed match tries
every combination of
+ * the lengths that its repeating quantifiers take, so each one costs a
factor of the input
+ * length. `find` also tries every start position, which costs one more
factor unless the regex
+ * is anchored at the start. So two quantifiers are expensive, and three
when the regex is
+ * anchored or must match the whole input, as the one a LIKE pattern
becomes. Repeating a group
+ * that holds a quantifier or an alternation multiplies the factors, so it
is always expensive.
+ *
+ * @param matchesWhole whether `regex` must match the whole input rather
than be found in it.
+ */
+ def isExpensiveRegex(regex: String, matchesWhole: Boolean): Boolean = {
+ val (quantifiers, topLevelAlternation) = scanQuantifiers(regex)
+ val anchored = matchesWhole || (!topLevelAlternation &&
startsWithAnchor(regex))
+ quantifiers >= (if (anchored) 3 else 2)
+ }
+
+ // Leading inline flags, e.g. `(?i)` or `(?s-m)`.
+ private val leadingFlags = """(?:\(\?[a-zA-Z]*(?:-[a-zA-Z]*)?\))*""".r
+
+ // `^` or `\A`, unless a quantifier may make it optional.
+ private val leadingAnchor = """(?:\^|\\A)(?![?*{])""".r
+
+ // Whether `regex` starts with `^` or `\A`. A leading flag that mentions
MULTILINE, under which
+ // `^` also matches after every line terminator, makes it not anchored.
+ private def startsWithAnchor(regex: String): Boolean = {
+ val flags = leadingFlags.findPrefixOf(regex).getOrElse("")
+ val rest = regex.substring(flags.length)
+ !flags.contains('m') && leadingAnchor.findPrefixOf(rest).isDefined
+ }
+
+ // An inline flag, e.g. `(?x)` or `(?ix:`, that turns on COMMENTS mode,
which lets whitespace
+ // and comments separate a group from its quantifier.
+ private val commentsFlag = Pattern.compile("""\(\?[a-zA-Z]*x""")
+
+ // The product of the choices of the uncounted ranges that makes them count
as one quantifier.
+ private val smallRangesLimit = 32
+
+ /**
+ * Counts the repeating quantifiers (`*`, `+`, `?` and `{n,m}`) in a regex
that can backtrack,
+ * and tells whether the regex has a `|` outside any group. Escaped
characters, quoted text,
+ * character classes and the braces of `\p{..}`, `\x{..}` and `\N{..}` are
skipped. The
+ * alternatives of a `|` outside any group are tried one after another, so
each is counted on its
+ * own and the largest count is returned. These quantifiers are not counted:
+ * - a possessive one, which never backtracks;
+ * - one that ends the regex or an alternative of it, possibly inside
groups that end there too,
+ * which a failed match never backtracks into;
+ * - `?` and a `{n,m}` that spans fewer than 10, which add a small factor.
The choices of such
+ * quantifiers, and of groups that hold a `|`, multiply, so they count as
one quantifier each
+ * time their product reaches 32.
+ * A repetition of at least 100, which adds its length to every attempt even
when it cannot
+ * backtrack, and a backreference, which compares as many characters as its
group holds, count
+ * as one quantifier each, unless they end the regex or an alternative of it.
+ * The count is `Int.MaxValue` for a repeated group that holds a quantifier
that can backtrack or
+ * a `|`, for a regex in COMMENTS mode, and for a `[` that is never closed,
since the scan cannot
+ * follow them.
+ *
+ * A streaming query pins `SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION` in
its checkpoint, since
+ * a pattern that turns cheap can move a filter below a stateful operator.
So a change that makes
+ * any pattern cheaper needs a new version of the estimate, and the old one
has to be kept.
+ */
+ private def scanQuantifiers(regex: String): (Int, Boolean) = {
+ // The largest count of the alternatives before the current one.
+ var maxCount = 0
+ var count = 0
+ var smallRanges = 1
+ var classDepth = 0
+ var topLevelAlternation = false
+ // Whether each open group holds a choice that a repetition of it would
multiply, innermost
+ // first.
+ var groups = List.empty[Boolean]
+ var i = 0
+ def markGroup(): Unit = groups = groups match {
+ case _ :: outer => true :: outer
+ case Nil => Nil
+ }
+ // Whether nothing but the closing `)` of open groups comes between `j`
and the end of the
+ // regex or a `|` outside any group.
+ def endsAlternative(j: Int): Boolean = {
+ var end = j
+ var open = groups
+ while (end < regex.length && open.nonEmpty && regex.charAt(end) == ')') {
+ end += 1
+ open = open.tail
+ }
+ end == regex.length || (open.isEmpty && regex.charAt(end) == '|')
+ }
+ // Adds a choice among `n` to the product, which counts as one quantifier
when it gets large.
+ def addChoices(n: Int): Unit = {
+ smallRanges *= n
+ if (smallRanges >= smallRangesLimit) {
+ count += 1
+ smallRanges = 1
+ }
+ }
+ // Handles the quantifier that ends at `i`, which repeats its operand
`min` to `max` times,
+ // and moves `i` past its modifier.
+ def quantifier(min: Int, max: Int): Unit = {
+ val possessive = regex.startsWith("+", i + 1)
+ if (possessive || regex.startsWith("?", i + 1)) i += 1
+ if (!endsAlternative(i + 1)) {
+ if (!possessive && max - min >= 10) {
+ markGroup()
+ count += 1
+ } else {
+ if (!possessive && max > min) {
+ markGroup()
+ addChoices(max - min + 1)
+ }
+ if (min >= 100) count += 1
+ }
+ }
+ }
+ while (i < regex.length) {
+ regex.charAt(i) match {
+ case '\\' if i + 1 < regex.length =>
+ i += 1
+ regex.charAt(i) match {
+ case 'Q' =>
+ val end = regex.indexOf("\\E", i)
+ i = if (end < 0) regex.length else end + 1
+ // `\cX` is a control character, and `X` may be `[`.
+ case 'c' => i += 1
+ case c if ((c >= '1' && c <= '9') || c == 'k') && classDepth == 0
&&
+ !endsAlternative(i + 1) =>
+ count += 1
+ case c if "pPxN".indexOf(c) >= 0 &&
+ i + 1 < regex.length && regex.charAt(i + 1) == '{' =>
+ val end = regex.indexOf('}', i)
+ i = if (end < 0) regex.length else end
+ case _ =>
+ }
+ case '[' =>
+ classDepth += 1
+ // A `]` right after `[` or `[^` is a literal.
+ if (i + 1 < regex.length && regex.charAt(i + 1) == '^') i += 1
+ if (i + 1 < regex.length && regex.charAt(i + 1) == ']') i += 1
+ case ']' if classDepth > 0 => classDepth -= 1
+ case _ if classDepth > 0 =>
+ case '(' =>
+ if (regex.startsWith("(?", i)) {
+ if (matcherAt(commentsFlag, regex, i).lookingAt()) {
+ return (Int.MaxValue, topLevelAlternation)
+ }
+ // The `?` of `(?:`, `(?=`, `(?<name>` or `(?i)` is not a
quantifier.
+ i += 1
+ }
+ groups = false :: groups
+ case ')' if groups.nonEmpty =>
+ val holdsChoice = groups.head
+ groups = groups.tail
+ if (holdsChoice) {
+ if (repeats(regex, i + 1)) return (Int.MaxValue,
topLevelAlternation)
+ markGroup()
+ }
+ case '|' if groups.isEmpty =>
+ topLevelAlternation = true
+ maxCount = math.max(maxCount, count)
+ count = 0
+ smallRanges = 1
+ case '|' =>
+ // A group adds one factor, however many alternatives it has.
+ if (!groups.head) addChoices(2)
+ markGroup()
+ case '*' => quantifier(0, Int.MaxValue)
+ case '+' => quantifier(1, Int.MaxValue)
+ case '?' => quantifier(0, 1)
+ case '{' =>
+ bounds(regex, i).foreach { case (end, min, max) =>
+ i = end
+ quantifier(min, max)
+ }
+ case _ =>
+ }
+ i += 1
+ }
+ (if (classDepth > 0) Int.MaxValue else math.max(maxCount, count),
topLevelAlternation)
+ }
+
+ // Whether a quantifier that can repeat its operand more than once starts at
`i`.
+ private def repeats(regex: String, i: Int): Boolean = i < regex.length && {
+ regex.charAt(i) match {
+ case '*' | '+' => true
+ case '{' => bounds(regex, i).exists(_._3 >= 2)
+ // `Pattern` removes an empty quote before it parses the regex.
+ case '\\' if regex.startsWith("\\Q\\E", i) => repeats(regex, i + 4)
+ case _ => false
+ }
+ }
+
+ // A matcher of `p` over `regex` from `start`, which does not copy the rest
of `regex`.
+ private def matcherAt(p: Pattern, regex: String, start: Int): Matcher =
+ p.matcher(regex).region(start, regex.length)
+
+ private val repetition = Pattern.compile("""\{(\d+)(?:,(\d*))?\}""")
+
+ // Parses the `{n}`, `{n,}` or `{n,m}` at `start` into its closing index and
its bounds, with
+ // `Int.MaxValue` as the upper bound of `{n,}`.
+ private def bounds(regex: String, start: Int): Option[(Int, Int, Int)] = {
+ val m = matcherAt(repetition, regex, start)
+ Option.when(m.lookingAt()) {
+ def toInt(s: String): Int = s.toIntOption.getOrElse(Int.MaxValue)
+ val min = toInt(m.group(1))
+ val max = Option(m.group(2)).map(s => if (s.isEmpty) Int.MaxValue else
toInt(s))
+ (m.end - 1, min, max.getOrElse(min))
+ }
+ }
+
def expressionToEscapeChar(e: Expression): Char = e match {
case StringLiteral(v) if v.length == 1 => v.charAt(0)
case _ => throw QueryCompilationErrors.invalidEscapeChar(e)
@@ -182,6 +382,10 @@ case class Like(left: Expression, right: Expression,
escapeChar: Char)
override def matches(regex: Pattern, str: String): Boolean =
regex.matcher(str).matches()
+ // Each `%` becomes a `.*`. `LikeSimplification` rewrites most patterns with
fewer than three.
+ override protected def isExpensivePattern(str: String): Boolean =
+ StringRegexExpression.isExpensiveRegex(escape(str), matchesWhole = true)
+
final override val nodePatterns: Seq[TreePattern] = Seq(LIKE_FAMLIY)
override def toString: String = {
@@ -534,6 +738,8 @@ case class RLike(left: Expression, right: Expression)
extends StringRegexExpress
override def escape(v: String): String = v
override def matches(regex: Pattern, str: String): Boolean =
regex.matcher(str).find(0)
+ override protected def isExpensivePattern(str: String): Boolean =
+ StringRegexExpression.isExpensiveRegex(str, matchesWhole = false)
override def toString: String = s"RLIKE($left, $right)"
override def sql: String =
s"${prettyName.toUpperCase(Locale.ROOT)}(${left.sql}, ${right.sql})"
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index 853b19e4986e..210ca2ce3d66 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -2641,7 +2641,7 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// uses nothing at all, or only inexpensive items. Then we can push it
because it might
// eliminate more data quickly and it may also be something which
could be evaluated at
// the storage layer. We may wish to improve this heuristic in the
future.
- def isCheap(used: AttributeMap[Alias]): Boolean =
!used.exists(_._2.child.expensive)
+ def isCheap(used: AttributeMap[Alias]): Boolean = !used.exists(a =>
isExpensive(a._2.child))
// Split the filter's components into cheap and expensive while
keeping track of
// what each references from the projection. The cheap components
written before the
// first expensive one are all pushed. After it, a throwable component
(e.g.
@@ -2984,11 +2984,17 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
expensiveConds: Seq[(Expression, AttributeMap[Alias])])
: (LogicalPlan, AttributeSet, Seq[Expression]) = {
val expensiveAliases = AttributeSet(
- aliasMap.collect { case (attr, alias) if alias.child.expensive => attr })
+ aliasMap.collect { case (attr, alias) if isExpensive(alias.child) =>
attr })
// Expensive sub-expressions behind each expensive alias, to spot when two
aliases are the
// same piece of work. Cheap aliases have none, so we skip the (possibly
wide) cheap rest.
+ // Two aliases share work exactly when they share a regex, so version 0 of
the estimate
+ // collects the regex expressions and not their ancestors.
+ val regexIsExpensive =
conf.getConf(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION) == 0
val expensivePartsOf = AttributeMap(expensiveAliases.toSeq.map { attr =>
- attr -> aliasMap(attr).child.collect { case e if e.expensive =>
e.canonicalized }.toSet
+ attr -> aliasMap(attr).child.collect {
+ case e if e.expensive || (regexIsExpensive &&
e.isInstanceOf[StringRegexExpression]) =>
+ e.canonicalized
+ }.toSet
})
def expensiveParts(attr: Attribute): Set[Expression] =
expensivePartsOf.getOrElse(attr, Set.empty)
@@ -3052,6 +3058,15 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
(plan, computed, conditionsOf(pending))
}
+ /**
+ * Whether `e` is expensive. Version 0 of the regex pattern cost estimate
treats every LIKE and
+ * RLIKE as expensive. It is checked here rather than in
`StringRegexExpression.expensive`,
+ * because `hasExpensiveChild` caches the answer of a parent expression.
+ */
+ private def isExpensive(e: Expression): Boolean = e.expensive ||
+ (conf.getConf(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION) == 0 &&
+ e.exists(_.isInstanceOf[StringRegexExpression]))
+
/**
* Check if we can safely push a filter through a projection, by making sure
that predicate
* subqueries in the condition do not contain the same attributes as the
plan they are moved
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index b29ee4579ecd..f0584ff14905 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -670,6 +670,20 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val REGEX_PATTERN_COST_ESTIMATE_VERSION =
+ buildConf("spark.sql.optimizer.regexPatternCostEstimateVersion")
+ .internal()
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .doc(s"How ${AVOID_DOUBLE_FILTER_EVAL.key} decides whether a LIKE or RLIKE
expression is " +
+ "expensive. 0: every LIKE and RLIKE expression is expensive, as in 4.2.0
for a subject " +
+ "that is not a literal. 1: only one whose pattern is likely to be
expensive to match. A " +
+ "streaming query keeps the version recorded in its checkpoint, and one
restarted from a " +
+ "checkpoint without it uses 0.")
+ .version("4.2.2")
+ .intConf
+ .checkValue(_ >= 0, "The version of the regex pattern cost estimate must
be nonnegative.")
+ .createWithDefault(1)
+
val ATTEMPT_TRANSPILATION_OF_PYTHON_UDFS =
buildConf("spark.sql.experimental.optimizer.transpilePyUDFs")
.withBindingPolicy(ConfigBindingPolicy.SESSION)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
index 5377b9c7fc0b..0ba9fadd4bba 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala
@@ -735,4 +735,81 @@ class RegexpExpressionsSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(extractAllCopy ne extractAll,
"freshCopyIfContainsStatefulExpression should return a new instance for
RegExpExtractAll")
}
+
+ test("SPARK-60632: LIKE and RLIKE expensive heuristic looks at the pattern")
{
+ val str = $"s".string.at(0)
+ val pattern = $"p".string.at(1)
+
+ // The subject string is irrelevant; a constant pattern is checked. Two or
more repeating
+ // quantifiers that can backtrack make an RLIKE expensive, and three or
more when it is
+ // anchored at the start.
+ // scalastyle:off SingleSpaceBetweenRParenAndLCurlyBrace
+ Seq("^1", "magic", "a.b", "^a.*b", "2024-.*", "\\d{2,}", "a{3}b*",
"(?i)^abc.*",
+ "a\\+b\\*", "[+*{]x+", "\\p{Lu}+", "\\x{41}.*", "\\Q.*+\\E.*",
"[a-z&&[^b]]+x",
+ "\\(?x.*a", "[]a]*x",
+ // Possessive quantifiers and small ranges.
+ "a*+b", "a?+b?+", "a{3}+b", ".*+a.*+b", "a{1,9}b{1,9}", "(ab){3}.*",
+ ".{0,9}a.{0,9}a.{0,9}x",
+ // A quantifier that ends the pattern or an alternative, and
alternatives on their own.
+ "a.*b.*", "a{2,}b+", "\\d+\\s*", "a+?b.*", "(foo.*bar.*)", "a.*b.*|x",
"a.*b|a.*c",
+ "foo.*bar|baz.*qux", "colou?r.*x", "(.*)\\1", ".*.{1000}",
+ // Groups that hold a `|`, a long fixed repetition and a backreference
on their own.
+ ".*(k1|k2|k3|k4|k5|k6|k7|k8).*", ".{1000}x", "\\d{4}-\\d{2}-\\d{2}.*",
"(\\w)\\1x",
+ "^(\\w+)\\s\\1.*", ".*" + "(a|aa)" * 4 + "x",
+ // Anchored.
+ "^.*a.*b", "\\A.*a.*b", "(?i)^.*a.*b", "^v[0-9]+_.*",
"^(a|b).*c.*").foreach { p =>
+ assert(!RLike(str, Literal(p)).expensive, p)
+ }
+ Seq(".*a.*b", "^.*a.*a.*b", "\\b.*a.*b", "a.*b.*$", ".{0,50}a.{0,50}b",
"^a|.*b.*c",
+ "(?m)^.*a.*b", ".{0,9}a.{0,9}a.{0,9}a.{0,9}a.{0,9}a.{0,9}x",
".{0,9}a.{0,9}a.{0,8}a.*b",
+ "a?" * 24 + "b", "(a.*b.*)c", "x|a.*b.*c",
+ // Groups that hold a `|`, which multiply like `?`.
+ "(a|a)" * 24 + "b", "(a|)" * 24 + "b", "(?:a|aa)" * 24 + "b",
"(\\w|\\d)" * 24 + "x",
+ ".*" + "(a|aa)" * 5 + "x", ".*" + "a?" * 5 + "x",
+ // An anchor that a quantifier makes optional.
+ "^?.*a.*b", "\\A?.*a.*b", "^{0,1}.*a.*b",
+ // A backreference, and a long repetition next to a quantifier.
+ "(.*)\\1x", "(.+)\\1\\1x", "(?<g>.*)\\k<g>x", ".*.{1000}x",
".*.{1000}+x",
+ ".*.{1000,1005}x",
+ // A repeated group that holds a quantifier or an alternation.
+ "(a+)+b", "^(a+)+b", "^(.*?,){11}P", "(.*a){3}x", "^((.*a)b){2}",
"(a{1,9}){6}b",
+ "(a|aa){20}b", "^(\\d|\\w){24}x", "(a?a?){16}b", "^(.?.?){16}x",
+ "^(.*?,)\\Q\\E{11}P",
+ // COMMENTS mode.
+ "(?x) ^ (.*?,) {11} P", "(?x)(.*a) {3}x",
+ // A `[` that does not open a class, and a `]` that does not close one.
+ "\\c[.*a.*b", "(?x)a # [\n.*a.*b", "^[](]a|.*b.*c").foreach { p =>
+ assert(RLike(str, Literal(p)).expensive, p)
+ }
+ // scalastyle:on SingleSpaceBetweenRParenAndLCurlyBrace
+ // Three or more `%` that are not at the end make a LIKE expensive, since
it must match the
+ // whole input.
+ Seq("1_", "%magic%", "%a%b", "%a%b%", "%a\\%b%", "a_b_c",
"%a\\\\%b").foreach { p =>
+ assert(!Like(str, Literal(p), '\\').expensive, p)
+ }
+ assert(!Like(str, Literal("%a#%b#%c%"), '#').expensive)
+ Seq("%a%b%c%", "%a%a%a%b", "%(.*)%|%x").foreach { p =>
+ assert(Like(str, Literal(p), '\\').expensive, p)
+ }
+ // A pattern with an invalid escape is expensive rather than failing the
optimizer.
+ assert(Like(str, Literal("a\\b"), '\\').expensive)
+
+ // A foldable pattern that is not yet a literal.
+ assert(!RLike(str, Concat(Seq(Literal("^"), Literal("a.*")))).expensive)
+ assert(RLike(str, Concat(Seq(Literal(".*a"), Literal(".*b")))).expensive)
+ assert(!RLike(str, Literal.create(null, StringType)).expensive)
+ // A foldable pattern that fails to evaluate is expensive rather than
failing the optimizer.
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ assert(RLike(str, Cast(Divide(Literal(1), Literal(0)),
StringType)).expensive)
+ }
+
+ // A non-foldable pattern is compiled for every row.
+ assert(RLike(str, pattern).expensive)
+ assert(Like(str, pattern, '\\').expensive)
+
+ // An expensive subject wins over a cheap pattern.
+ val udf = ScalaUDF((s: String) => s, StringType, Seq(str), Seq(None))
+ assert(RLike(udf, Literal("^1")).expensive)
+ assert(Like(udf, Literal("1_"), '\\').expensive)
+ }
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index 84f68fa018c1..d10757b7219a 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -72,6 +72,10 @@ class FilterPushdownSuite extends PlanTest {
&& (("x.a".attr > 3) && ("y.a".attr > 13) || ("x.a".attr > 1) &&
("y.a".attr > 11)))).analyze
}
+ // Stands in for an expensive expression: a regex whose pattern is not a
constant is compiled
+ // for every row.
+ private def expensiveRLike(tag: String): Expression =
$"e".rlike(Concat(Seq($"e", Literal(tag))))
+
// This test already passes.
test("eliminate subqueries") {
val originalQuery =
@@ -175,15 +179,15 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-59693: Do double evaluation when configured") {
withSQLConf(SQLConf.AVOID_DOUBLE_FILTER_EVAL.key -> "false") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("notmagic") as
"j", $"b")
+ .select($"a", expensiveRLike("magic") as "f",
expensiveRLike("notmagic") as "j", $"b")
.where($"a" > 5 && $"f")
.analyze
val optimized = Optimize.execute(originalQuery)
val correctAnswer = testStringRelation
- .where($"a" > 5 && $"e".rlike("magic"))
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("notmagic") as
"j", $"b")
+ .where($"a" > 5 && expensiveRLike("magic"))
+ .select($"a", expensiveRLike("magic") as "f",
expensiveRLike("notmagic") as "j", $"b")
.analyze
comparePlans(optimized, correctAnswer)
@@ -192,7 +196,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-59693: Make sure that we handle the case where everything is
expensive") {
val originalQuery = testStringRelation
- .select($"e".rlike("magic") as "f")
+ .select(expensiveRLike("magic") as "f")
.where($"f")
.analyze
@@ -269,7 +273,7 @@ class FilterPushdownSuite extends PlanTest {
// Case 1: Multiple filters that don't reference any projection aliases -
all should be pushed
test("SPARK-59693: Case 1 - multiple filters not referencing projection
aliases") {
val originalQuery = testStringRelation
- .select($"a" as "c", $"e".rlike("magic") as "f", $"b" as "d", $"a", $"b")
+ .select($"a" as "c", expensiveRLike("magic") as "f", $"b" as "d", $"a",
$"b")
.where($"a" > 5 && $"b" < 10)
.analyze
@@ -279,7 +283,7 @@ class FilterPushdownSuite extends PlanTest {
// simple aliases (c->a, d->b) which are inexpensive
val correctAnswer = testStringRelation
.where($"a" > 5 && $"b" < 10)
- .select($"a" as "c", $"e".rlike("magic") as "f", $"b" as "d", $"a", $"b")
+ .select($"a" as "c", expensiveRLike("magic") as "f", $"b" as "d", $"a",
$"b")
.analyze
comparePlans(optimized, correctAnswer)
@@ -288,7 +292,7 @@ class FilterPushdownSuite extends PlanTest {
// Case 2: Multiple filters with inexpensive references - all should be
pushed
test("SPARK-59693: Case 2 - multiple filters with inexpensive alias
references") {
val originalQuery = testStringRelation
- .select($"a" + $"b" as "sum", $"a" - $"b" as "diff", $"e".rlike("magic")
as "f")
+ .select($"a" + $"b" as "sum", $"a" - $"b" as "diff",
expensiveRLike("magic") as "f")
.where($"sum" > 10 && $"diff" < 5)
.analyze
@@ -297,7 +301,7 @@ class FilterPushdownSuite extends PlanTest {
// Both sum and diff are inexpensive (arithmetic), so both filters should
be pushed
val correctAnswer = testStringRelation
.where($"a" + $"b" > 10 && $"a" - $"b" < 5)
- .select($"a" + $"b" as "sum", $"a" - $"b" as "diff", $"e".rlike("magic")
as "f")
+ .select($"a" + $"b" as "sum", $"a" - $"b" as "diff",
expensiveRLike("magic") as "f")
.analyze
comparePlans(optimized, correctAnswer)
@@ -306,7 +310,7 @@ class FilterPushdownSuite extends PlanTest {
// Case 3: Filter references expensive to compute references.
test("SPARK-59693: Avoid double evaluation with projections can't push past
certain items") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f")
+ .select($"a", expensiveRLike("magic") as "f")
.where($"a" > 5 || $"f")
.analyze
@@ -318,7 +322,7 @@ class FilterPushdownSuite extends PlanTest {
// Combined case 1, 2, and 3 filter pushdown
test("SPARK-59693: Case 1, 2, and 3 make sure we leave up and push down
correctly.") {
val originalQuery = testStringRelation
- .select($"a" + $"b" as "sum", $"a" - $"b" as "diff", $"e".rlike("magic")
as "f")
+ .select($"a" + $"b" as "sum", $"a" - $"b" as "diff",
expensiveRLike("magic") as "f")
.where($"sum" > 10 && $"diff" < 5 && $"f")
.analyze
@@ -328,16 +332,83 @@ class FilterPushdownSuite extends PlanTest {
// rlike magic is expensive so not pushed.
val correctAnswer = testStringRelation
.where($"a" + $"b" > 10 && $"a" - $"b" < 5)
- .select($"a" + $"b" as "sum", $"a" - $"b" as "diff", $"e".rlike("magic")
as "f")
+ .select($"a" + $"b" as "sum", $"a" - $"b" as "diff",
expensiveRLike("magic") as "f")
.where($"f")
.analyze
comparePlans(optimized, correctAnswer)
}
+ test("SPARK-60632: Push a filter on a LIKE/RLIKE alias with a cheap constant
pattern") {
+ Seq(
+ $"e".rlike("^1"),
+ $"e".rlike("magic"),
+ $"e".rlike("^a.*b"),
+ $"e".like("1_"),
+ $"e".like("%magic%"),
+ // ILIKE, as the optimizer sees it after `ReplaceExpressions`.
+ Like(Lower($"e"), Lower(Literal("%Magic%")), '\\')).foreach { pred =>
+ val originalQuery = testStringRelation
+ .select($"a", pred as "f")
+ .where($"f")
+ .analyze
+
+ val correctAnswer = testStringRelation
+ .where(pred)
+ .select($"a", pred as "f")
+ .analyze
+
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+ }
+ }
+
+ test("SPARK-60632: Do not push a filter on a LIKE/RLIKE alias with a
backtracking pattern") {
+ Seq($"e".rlike("(a+)+b"), $"e".rlike(".*a.*b"),
$"e".like("%a%b%c")).foreach { pred =>
+ val originalQuery = testStringRelation
+ .select($"a", pred as "f")
+ .where($"f")
+ .analyze
+
+ comparePlans(Optimize.execute(originalQuery), originalQuery)
+ }
+ }
+
+ test("SPARK-60632: Do not push a filter on a LIKE/RLIKE alias with version 0
of the estimate") {
+ Seq($"e".rlike("^1"), $"e".like("1_"), !$"e".rlike("^1")).foreach { pred =>
+ val originalQuery = testStringRelation
+ .select($"a", pred as "f")
+ .where($"f")
+ .analyze
+
+ // A parent expression caches whether its child is expensive, so
optimize the same plan
+ // with version 1 first.
+ Optimize.execute(originalQuery)
+ withSQLConf(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION.key -> "0") {
+ comparePlans(Optimize.execute(originalQuery), originalQuery)
+ }
+ }
+
+ // Version 0 also splits the projection between two LIKE/RLIKE aliases.
+ withSQLConf(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION.key -> "0") {
+ val originalQuery = testStringRelation
+ .select($"a", $"e".rlike("^1") as "f", $"e".rlike("^2") as "g", $"b")
+ .where($"f" && $"g")
+ .analyze
+
+ val correctAnswer = testStringRelation
+ .select($"a", $"b", $"e", $"e".rlike("^1") as "f")
+ .where($"f")
+ .select($"a", $"f", $"e".rlike("^2") as "g", $"b")
+ .where($"g")
+ .analyze
+
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+ }
+ }
+
test("SPARK-55014: split the projection so a second expensive element sees
fewer rows") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"b")
.where($"f" && $"g")
.analyze
@@ -346,9 +417,9 @@ class FilterPushdownSuite extends PlanTest {
// Both regexes are expensive and neither condition needs both, so f is
computed first and
// 'other' only runs on the rows 'magic' matched.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f")
.where($"f")
- .select($"a", $"f", $"e".rlike("other") as "g", $"b")
+ .select($"a", $"f", expensiveRLike("other") as "g", $"b")
.where($"g")
.analyze
@@ -357,7 +428,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: split the projection even for a single expensive
condition") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"b")
.where($"f")
.analyze
@@ -365,9 +436,9 @@ class FilterPushdownSuite extends PlanTest {
// g is never filtered on but is still expensive, so it is worth deferring
past the filter on f.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f")
.where($"f")
- .select($"a", $"f", $"e".rlike("other") as "g", $"b")
+ .select($"a", $"f", expensiveRLike("other") as "g", $"b")
.analyze
comparePlans(optimized, correctAnswer)
@@ -375,8 +446,8 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: split the projection once per group of expensive
elements") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"e".rlike("third") as
- "j")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g",
+ expensiveRLike("third") as "j")
.where($"f" && $"g" && $"j")
.analyze
@@ -384,11 +455,11 @@ class FilterPushdownSuite extends PlanTest {
// One projection per condition except the last -- nothing left to defer
past it.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f")
.where($"f")
- .select($"a", $"b", $"e", $"f", $"e".rlike("other") as "g")
+ .select($"a", $"b", $"e", $"f", expensiveRLike("other") as "g")
.where($"g")
- .select($"a", $"f", $"g", $"e".rlike("third") as "j")
+ .select($"a", $"f", $"g", expensiveRLike("third") as "j")
.where($"j")
.analyze
@@ -397,7 +468,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: a condition needing every expensive element can not be
split off") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"b")
.where($"f" || $"g")
.analyze
@@ -412,8 +483,8 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: a condition over several expensive elements can still be
split off") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"e".rlike("third") as
- "j")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g",
+ expensiveRLike("third") as "j")
.where($"f" || $"g")
.analyze
@@ -422,9 +493,9 @@ class FilterPushdownSuite extends PlanTest {
// A third expensive element to defer, so f and g compute together below
the filter and j
// waits above it.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f",
$"e".rlike("other") as "g")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f",
expensiveRLike("other") as "g")
.where($"f" || $"g")
- .select($"a", $"f", $"g", $"e".rlike("third") as "j")
+ .select($"a", $"f", $"g", expensiveRLike("third") as "j")
.analyze
comparePlans(optimized, correctAnswer)
@@ -432,8 +503,8 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: the least demanding condition goes lowest") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"e".rlike("third") as
- "j")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g",
+ expensiveRLike("third") as "j")
.where(($"f" || $"g") && $"j")
.analyze
@@ -442,9 +513,9 @@ class FilterPushdownSuite extends PlanTest {
// j needs one alias and (f || g) needs two, so j goes first even though
it is projected
// last -- leaving two expensive elements to run on its survivors.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("third") as "j")
+ .select($"a", $"b", $"e", expensiveRLike("third") as "j")
.where($"j")
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"j")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"j")
.where($"f" || $"g")
.analyze
@@ -453,7 +524,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: split the projection while also pushing the cheap
conditions") {
val originalQuery = testStringRelation
- .select($"a" + $"b" as "sum", $"e".rlike("magic") as "f",
$"e".rlike("other") as "g")
+ .select($"a" + $"b" as "sum", expensiveRLike("magic") as "f",
expensiveRLike("other") as "g")
.where($"sum" > 10 && $"f")
.analyze
@@ -462,9 +533,9 @@ class FilterPushdownSuite extends PlanTest {
// sum is cheap (case 2, pushed down); f is expensive (split around); g
deferred on top.
val correctAnswer = testStringRelation
.where($"a" + $"b" > 10)
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f")
.where($"f")
- .select($"a" + $"b" as "sum", $"f", $"e".rlike("other") as "g")
+ .select($"a" + $"b" as "sum", $"f", expensiveRLike("other") as "g")
.analyze
comparePlans(optimized, correctAnswer)
@@ -472,7 +543,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: no split when the only elements left to defer are cheap")
{
val originalQuery = testStringRelation
- .select($"a" + $"b" as "sum", $"e".rlike("magic") as "f")
+ .select($"a" + $"b" as "sum", expensiveRLike("magic") as "f")
.where($"f")
.analyze
@@ -486,7 +557,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: the projection on top keeps the original column
ordering") {
val originalQuery = testStringRelation
- .select($"b", $"e".rlike("magic") as "f", $"a", $"e".rlike("other") as
"g")
+ .select($"b", expensiveRLike("magic") as "f", $"a",
expensiveRLike("other") as "g")
.where($"f" && $"g")
.analyze
@@ -494,9 +565,9 @@ class FilterPushdownSuite extends PlanTest {
// f is computed below but must come back out in its original position.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f")
.where($"f")
- .select($"b", $"f", $"a", $"e".rlike("other") as "g")
+ .select($"b", $"f", $"a", expensiveRLike("other") as "g")
.where($"g")
.analyze
@@ -527,7 +598,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: no split when the expensive work is shared between
elements") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", Not($"e".rlike("magic")) as
"g")
+ .select($"a", expensiveRLike("magic") as "f",
Not(expensiveRLike("magic")) as "g")
.where($"f" && $"g")
.analyze
@@ -543,8 +614,8 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: elements sharing expensive work are split off together") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", Not($"e".rlike("magic")) as
"g",
- $"e".rlike("other") as "j")
+ .select($"a", expensiveRLike("magic") as "f",
Not(expensiveRLike("magic")) as "g",
+ expensiveRLike("other") as "j")
.where($"f" && $"j")
.analyze
@@ -553,9 +624,9 @@ class FilterPushdownSuite extends PlanTest {
// j is its own work needing one alias; f drags g along, so j splits off
first and f and g
// stay together on top.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("other") as "j")
+ .select($"a", $"b", $"e", expensiveRLike("other") as "j")
.where($"j")
- .select($"a", $"e".rlike("magic") as "f", Not($"e".rlike("magic")) as
"g", $"j")
+ .select($"a", expensiveRLike("magic") as "f",
Not(expensiveRLike("magic")) as "g", $"j")
.where($"f")
.analyze
@@ -564,8 +635,8 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: a non-deterministic condition is not split off into a
lower layer") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"e".rlike("third") as
- "j")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g",
+ expensiveRLike("third") as "j")
.where((Rand(10) > 0.5 || $"j") && ($"f" || $"g"))
.analyze
@@ -575,9 +646,9 @@ class FilterPushdownSuite extends PlanTest {
// it would go lowest and see every row. It stays in the top filter
instead, and (f || g)
// takes the layer it would have had.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f",
$"e".rlike("other") as "g")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f",
expensiveRLike("other") as "g")
.where($"f" || $"g")
- .select($"a", $"f", $"g", $"e".rlike("third") as "j")
+ .select($"a", $"f", $"g", expensiveRLike("third") as "j")
.where(Rand(10) > 0.5 || $"j")
.analyze
@@ -586,7 +657,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: no split when the expensive conditions are
non-deterministic") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"b")
.where(Rand(10) > 0.5 || $"f")
.analyze
@@ -602,7 +673,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-60602: a throwable condition is not split off ahead of the
conditions before it") {
val raiseError = IsNull(RaiseError(Literal("boom")))
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"b")
.where($"f" =!= $"g" && ($"g" || raiseError))
.analyze
@@ -616,7 +687,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-60602: a throwable condition does not keep later conditions from
being split off") {
val raiseError = IsNull(RaiseError(Literal("boom")))
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"b")
.where(($"g" || raiseError) && $"f")
.analyze
@@ -624,9 +695,9 @@ class FilterPushdownSuite extends PlanTest {
// Filtering on f first only leaves the throwable condition fewer rows to
raise on.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f")
.where($"f")
- .select($"a", $"f", $"e".rlike("other") as "g", $"b")
+ .select($"a", $"f", expensiveRLike("other") as "g", $"b")
.where($"g" || raiseError)
.analyze
@@ -638,7 +709,7 @@ class FilterPushdownSuite extends PlanTest {
Seq(true, false).foreach { split =>
withSQLConf(SQLConf.SPLIT_PROJECTION_FOR_EXPENSIVE_FILTERS.key ->
split.toString) {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"b")
+ .select($"a", expensiveRLike("magic") as "f", $"b")
.where($"f" && ($"a" > 5 || raiseError) && $"b" > 1)
.analyze
@@ -647,7 +718,7 @@ class FilterPushdownSuite extends PlanTest {
// Only the non-throwable cheap condition moves below the expensive
one.
val correctAnswer = testStringRelation
.where($"b" > 1)
- .select($"a", $"e".rlike("magic") as "f", $"b")
+ .select($"a", expensiveRLike("magic") as "f", $"b")
.where($"f" && ($"a" > 5 || raiseError))
.analyze
@@ -655,13 +726,13 @@ class FilterPushdownSuite extends PlanTest {
// Written ahead of every expensive condition, it is pushed as usual.
val throwableFirst = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"b")
+ .select($"a", expensiveRLike("magic") as "f", $"b")
.where(($"a" > 5 || raiseError) && $"f")
.analyze
val throwableFirstAnswer = testStringRelation
.where($"a" > 5 || raiseError)
- .select($"a", $"e".rlike("magic") as "f", $"b")
+ .select($"a", expensiveRLike("magic") as "f", $"b")
.where($"f")
.analyze
@@ -674,7 +745,7 @@ class FilterPushdownSuite extends PlanTest {
"expensive one") {
val raiseError = IsNull(RaiseError(Literal("boom")))
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", If($"a" > 5, Literal(true),
raiseError) as "t",
+ .select($"a", expensiveRLike("magic") as "f", If($"a" > 5,
Literal(true), raiseError) as "t",
$"b")
.where($"f" && $"t" && $"b" > 1)
.analyze
@@ -684,7 +755,7 @@ class FilterPushdownSuite extends PlanTest {
// `t` is cheap, but reading it evaluates the throwable expression behind
it.
val correctAnswer = testStringRelation
.where($"b" > 1)
- .select($"a", $"e".rlike("magic") as "f", If($"a" > 5, Literal(true),
raiseError) as "t",
+ .select($"a", expensiveRLike("magic") as "f", If($"a" > 5,
Literal(true), raiseError) as "t",
$"b")
.where($"f" && $"t")
.analyze
@@ -695,9 +766,9 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-60602: a condition over a throwable alias is not split off ahead
of the " +
"conditions before it") {
val raiseError = IsNull(RaiseError(Literal("boom")))
- val throwableAlias = If($"e".rlike("other"), Literal(true), raiseError)
+ val throwableAlias = If(expensiveRLike("other"), Literal(true), raiseError)
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("third") as "h",
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("third") as
"h",
throwableAlias as "g", $"b")
.where($"f" =!= $"h" && $"g")
.analyze
@@ -707,7 +778,7 @@ class FilterPushdownSuite extends PlanTest {
// On cost alone the condition on g would go lowest, computing g on every
row. Splitting
// f and h off instead only leaves g fewer rows to raise on.
val correctAnswer = testStringRelation
- .select($"a", $"b", $"e", $"e".rlike("magic") as "f",
$"e".rlike("third") as "h")
+ .select($"a", $"b", $"e", expensiveRLike("magic") as "f",
expensiveRLike("third") as "h")
.where($"f" =!= $"h")
.select($"a", $"f", $"h", throwableAlias as "g", $"b")
.where($"g")
@@ -727,7 +798,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-60602: the conditions left above the projection keep their
written order") {
val raiseError = IsNull(RaiseError(Literal("boom")))
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"b")
.where($"f" =!= $"g" && ($"g" || raiseError) && $"a" > 1)
.analyze
val Seq(unplaced, throwable, _) = splitConjunctivePredicates(
@@ -741,7 +812,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-60602: a non-deterministic condition left above the projection
keeps its " +
"written order") {
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other") as
"g", $"b")
.where((Rand(10) > 0.5 || $"g") && $"f" =!= $"g" && $"a" > 1)
.analyze
val Seq(nonDeterministic, unplaced, _) = splitConjunctivePredicates(
@@ -753,7 +824,7 @@ class FilterPushdownSuite extends PlanTest {
test("SPARK-55014: do not split the projection when configured") {
withSQLConf(SQLConf.SPLIT_PROJECTION_FOR_EXPENSIVE_FILTERS.key -> "false")
{
val originalQuery = testStringRelation
- .select($"a", $"e".rlike("magic") as "f", $"e".rlike("other") as "g",
$"b")
+ .select($"a", expensiveRLike("magic") as "f", expensiveRLike("other")
as "g", $"b")
.where($"f" && $"g")
.analyze
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
index 5731b9d172d5..b4947e04422f 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
@@ -207,7 +207,8 @@ object OffsetSeqMetadata extends Logging {
ENABLE_STREAMING_SOURCE_EVOLUTION,
STATEFUL_OPERATOR_ALWAYS_NULLABLE_OUTPUT,
DROP_DUPLICATES_DETERMINISTIC_KEY_ORDER,
- ALLOW_EXCEPT_ON_STREAMING_DATAFRAME
+ ALLOW_EXCEPT_ON_STREAMING_DATAFRAME,
+ REGEX_PATTERN_COST_ESTIMATE_VERSION
)
/**
@@ -280,7 +281,8 @@ object OffsetSeqMetadata extends Logging {
ENABLE_STREAMING_SOURCE_EVOLUTION.key -> "false",
STATEFUL_OPERATOR_ALWAYS_NULLABLE_OUTPUT.key -> "false",
DROP_DUPLICATES_DETERMINISTIC_KEY_ORDER.key -> "false",
- ALLOW_EXCEPT_ON_STREAMING_DATAFRAME.key -> "true"
+ ALLOW_EXCEPT_ON_STREAMING_DATAFRAME.key -> "true",
+ REGEX_PATTERN_COST_ESTIMATE_VERSION.key -> "0"
)
def readValue[T](metadataLog: OffsetSeqMetadataBase, confKey:
ConfigEntry[T]): String = {
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala
index f38996adbd99..d5754d34a521 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategySuite.scala
@@ -31,7 +31,7 @@ import org.apache.spark.paths.SparkPath.{fromUrlString => sp}
import org.apache.spark.sql.{execution, DataFrame, Row, SparkSession}
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.catalog.BucketSpec
-import org.apache.spark.sql.catalyst.expressions.{Expression, ExpressionSet}
+import org.apache.spark.sql.catalyst.expressions.{Expression, ExpressionSet,
StringRegexExpression}
import org.apache.spark.sql.catalyst.util
import org.apache.spark.sql.classic.Dataset
import org.apache.spark.sql.execution.{DataSourceScanExec, FileSourceScanExec,
SparkPlan}
@@ -606,6 +606,37 @@ class FileSourceStrategySuite extends SharedSparkSession {
checkDataFilters(Set.empty)
}
+ test("SPARK-60632: Filters on an aliased LIKE/RLIKE column are pushed into
the file scan") {
+ withTempPath { path =>
+ spark.range(20)
+ .selectExpr("CAST(id AS STRING) AS name", "CONCAT('v', id % 4) AS
part")
+ .write.partitionBy("part").parquet(path.getCanonicalPath)
+ val table = spark.read.parquet(path.getCanonicalPath)
+
+ def scanOf(df: DataFrame): FileSourceScanExec =
+ df.queryExecution.sparkPlan.collectFirst { case s: FileSourceScanExec
=> s }.get
+ def hasRegex(filters: Seq[Expression]): Boolean =
+ filters.exists(_.exists(_.isInstanceOf[StringRegexExpression]))
+
+ Seq(
+ "part RLIKE '^v1'" -> 1,
+ "part RLIKE 'v[12].*'" -> 2,
+ "part LIKE '_1' OR part = 'v2'" -> 2).foreach { case (pred,
numPartitions) =>
+ withClue(pred) {
+ val df = table.selectExpr("name", s"$pred AS p").where("p")
+ val scan = scanOf(df)
+ assert(hasRegex(scan.partitionFilters))
+ assert(scan.selectedPartitions.partitionCount == numPartitions)
+ checkAnswer(df, table.where(pred).selectExpr("name", "true AS p"))
+ }
+ }
+
+ val df = table.selectExpr("part", "name RLIKE '^1' AS p").where("p")
+ assert(hasRegex(scanOf(df).dataFilters))
+ checkAnswer(df, table.where("name RLIKE '^1'").selectExpr("part", "true
AS p"))
+ }
+ }
+
test(s"SPARK-44021: Test ${SQLConf.FILES_MAX_PARTITION_NUM.key} works as
expected") {
withSQLConf(SQLConf.FILES_MAX_PARTITION_BYTES.key -> "128MB") {
val files =
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
index b47c0dc5b372..0a31aba4aa8f 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
@@ -52,6 +52,20 @@ class OffsetSeqLogSuite extends SharedSparkSession {
assert(metadata.conf.get(SQLConf.ALLOW_EXCEPT_ON_STREAMING_DATAFRAME.key).contains("false"))
}
+ test("SPARK-60632: regex pattern cost estimate is version 0 for old
checkpoints") {
+ val (_, offsetSeq) = readFromResource("offset-log-version-2.1.0")
+ val metadata = offsetSeq.metadataOpt.get
+
assert(!metadata.conf.contains(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION.key))
+ assert(OffsetSeqMetadata.readValue(
+ metadata, SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION) == "0")
+ }
+
+ test("SPARK-60632: regex pattern cost estimate is version 1 for new
checkpoints") {
+ val metadata = OffsetSeqMetadata(
+ batchWatermarkMs = 0, batchTimestampMs = 0, spark.conf)
+
assert(metadata.conf.get(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION.key).contains("1"))
+ }
+
test("SPARK-59919: v1 and v2 metadata persist rebound stateful shuffle
partitions") {
withSQLConf(
SQLConf.SHUFFLE_PARTITIONS.key -> "10",
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
index 219640ea3eb3..da010e3c4ec5 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
@@ -32,6 +32,7 @@ import org.scalatest.time.SpanSugar._
import org.apache.spark.SparkException
import org.apache.spark.sql.{AnalysisException, Dataset, Encoders}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, Project}
import
org.apache.spark.sql.execution.datasources.v2.StreamingDataSourceV2ScanRelation
import org.apache.spark.sql.execution.streaming.checkpointing.{OffsetMap,
OffsetSeq, OffsetSeqLog,
OffsetSeqMetadata, OffsetSeqMetadataV2}
@@ -168,6 +169,37 @@ class StreamingQueryManagerSuite extends StreamTest {
}
}
+ test("SPARK-60632: regex pattern cost estimate version 0 for old
checkpoints") {
+ withTempDir { checkpointDir =>
+ val input = MemoryStream[String]
+ // `NOT` caches whether its child is expensive, and both queries share
this plan.
+ val result = input.toDF().select($"value", !$"value".rlike("^1") as
"p").where($"p")
+
+ def run(data: String): Boolean = {
+ val query = result.writeStream
+ .format("noop")
+ .option("checkpointLocation", checkpointDir.getCanonicalPath)
+ .start()
+ try {
+ input.addData(data)
+ query.processAllAvailable()
+ // Whether the filter stays above the projection.
+
query.asInstanceOf[StreamingQueryWrapper].streamingQuery.lastExecution.optimizedPlan
+ .exists {
+ case Filter(_, _: Project) => true
+ case _ => false
+ }
+ } finally {
+ query.stop()
+ }
+ }
+
+ assert(!run("a"))
+ rewriteOffsetLogConf(checkpointDir)(_ -
SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION.key)
+ assert(run("b"))
+ }
+ }
+
test("streaming EXCEPT compatibility uses the query session checkpoint
location") {
withTempDir { checkpointRoot =>
withTempDir { otherCheckpointRoot =>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]