This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 16340d91b03b [SPARK-59614][SQL] Compile the sort-merge ASOF join
conditions and drop the output row copy
16340d91b03b is described below
commit 16340d91b03b88ee1ba8f32ae8ef12a140bfb26c
Author: Luka Zdravić <[email protected]>
AuthorDate: Fri Nov 6 03:38:19 2026 +0000
[SPARK-59614][SQL] Compile the sort-merge ASOF join conditions and drop the
output row copy
### What changes were proposed in this pull request?
Two small changes to `SortMergeAsOfJoinScanner`:
- It compiles the as-of and residual conditions, only when a scan first
checks a right row. Before, each check walked the bound expression tree and
boxed every value it read. The generated code skips subexpression elimination,
which would compute `x / y` first in `y <> 0 AND x / y > 1 AND x / y < 10` and
divide by zero. The interpreted fallback keeps subexpression elimination, but
computes a shared part only when an operand reaches it. So AND, OR and IF still
short-circuit on both paths.
- It returns its output row without a copy. A consumer that keeps rows
copies them.
### Why are the changes needed?
The scan checks the as-of condition for every right row it reads, so these
checks are its hot path.
Local benchmark (best of 5, ms; 400K rows per side and 2000 keys, except
the no-key cases with 10K rows per side):
| Case | Before | After |
|---|---|---|
| backward | 970 | 613 |
| forward | 990 | 600 |
| left outer | 942 | 611 |
| backward with tolerance | 2271 | 1159 |
| nearest | 782 | 773 |
| no key | 723 | 437 |
| no key, with an extra ON condition | 2155 | 610 |
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests: `SortMergeAsOfJoinScannerSuite`,
`AsOfJoinSortMergeSQLSuite`, `SortMergeAsOfJoinSuite`, `AsOfJoinSQLSuite`,
`DataFrameAsOfJoinSuite`, the ASOF test in `SQLMetricsSuite`, `PlannerSuite`,
and the `join-asof-*.sql` golden files. Tests such as "null as-of keys" check
that a NULL condition result still counts as no match.
To check that no row is lost without the copy, a local test fed the join
output into operators that keep rows (sort, limit, window, shuffle, distinct,
union, `collect_list`, a second sort-merge join, cache, `toLocalIterator`),
with whole-stage codegen on and off. All 37 cases matched the correlated
subquery path. All 37 failed when the scanner was changed to overwrite its row
early.
New tests put a guard before a part that throws in ANSI mode: in the ON
condition, a division behind an AND guard, the same division behind an OR
guard, and a cast behind a left-only guard in a LEFT ASOF JOIN; in the
MATCH_CONDITION, a division behind an AND guard inside an IF. They run with
compiled and with interpreted conditions, each with subexpression elimination
on and off. Scanner tests check that a repeated part behind a FALSE guard never
runs in either condition, and that a t [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #60029 from LukaZdravic/asof-jvm-per-row-overhead.
Authored-by: Luka Zdravić <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit e657651890acdc26185c1eec910473e4127de66e)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../execution/joins/SortMergeAsOfJoinExec.scala | 53 +++++--
.../spark/sql/AsOfJoinSortMergeSQLSuite.scala | 87 +++++++++-
.../joins/SortMergeAsOfJoinScannerSuite.scala | 176 ++++++++++++++++++---
3 files changed, 276 insertions(+), 40 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala
index ef1f8650ee52..448a31dfcb09 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala
@@ -24,7 +24,8 @@ import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.expressions.BindReferences.bindReference
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
GenerateOrdering}
+import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
GenerateOrdering,
+ GeneratePredicate}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical.AsOfJoin
import org.apache.spark.sql.catalyst.plans.physical._
@@ -175,7 +176,8 @@ private[joins] class SortMergeAsOfJoinScanner(
private val joinedOutput = leftOutput ++ rightOutput
private val joinedRow = new JoinedRow()
// Use nullable bound references so outer-join null padding is safe even when
- // right attributes are NOT NULL in the catalog schema.
+ // right attributes are NOT NULL in the catalog schema. Every output row
reuses the
+ // projection's row.
private val resultProjection = {
val nullableRefs = joinedOutput.zipWithIndex.map { case (attr, i) =>
BoundReference(i, attr.dataType, nullable = true)
@@ -183,9 +185,14 @@ private[joins] class SortMergeAsOfJoinScanner(
UnsafeProjection.create(nullableRefs, joinedOutput)
}
- private val boundAsOfCond = bindReference(asOfCondition, joinedOutput)
- private val boundResidualCond =
- residualCondition.map(bindReference(_, joinedOutput))
+ // A repeated part must not run before its guard: `y <> 0 AND x / y > 1 AND
x / y < 10` would
+ // divide by zero when y is 0. So the generated predicates skip
subexpression elimination. The
+ // interpreted fallback keeps it, but runs a shared part only when an
operand reaches it, so
+ // AND, OR and IF still short-circuit. Both are lazy, so a task that checks
no right row
+ // compiles neither condition.
+ private lazy val boundAsOfCond = ShortCircuitPredicate.create(asOfCondition,
joinedOutput)
+ private lazy val boundResidualCond =
+ residualCondition.map(ShortCircuitPredicate.create(_, joinedOutput))
private val equiKeyOrdering: Option[BaseOrdering] =
if (leftKeys.nonEmpty) {
@@ -291,7 +298,7 @@ private[joins] class SortMergeAsOfJoinScanner(
if (joinType == LeftOuter) {
numOutputRows += 1
joinedRow.withLeft(leftRow).withRight(nullRightRow)
- return resultProjection(joinedRow).copy()
+ return resultProjection(joinedRow)
}
} else {
val bestMatch = if (useCursorScan) {
@@ -304,11 +311,11 @@ private[joins] class SortMergeAsOfJoinScanner(
if (bestMatch != null) {
numOutputRows += 1
joinedRow.withLeft(leftRow).withRight(bestMatch)
- return resultProjection(joinedRow).copy()
+ return resultProjection(joinedRow)
} else if (joinType == LeftOuter) {
numOutputRows += 1
joinedRow.withLeft(leftRow).withRight(nullRightRow)
- return resultProjection(joinedRow).copy()
+ return resultProjection(joinedRow)
}
}
}
@@ -407,13 +414,14 @@ private[joins] class SortMergeAsOfJoinScanner(
}
/** Whether `cond` is TRUE on the current joined row. FALSE and NULL both
fail. */
- private def holds(cond: Expression): Boolean = {
- val result = cond.eval(joinedRow)
- result != null && result.asInstanceOf[Boolean]
- }
+ private def holds(cond: BasePredicate): Boolean = cond.eval(joinedRow)
/** Whether the residual ON condition, if any, holds on the current joined
row. */
- private def residualHolds: Boolean = boundResidualCond.forall(holds)
+ private def residualHolds: Boolean = boundResidualCond match {
+ // A match, not `forall`, so the per-row check makes no closure.
+ case Some(cond) => cond.eval(joinedRow)
+ case None => true
+ }
/**
* Backward joins keep the last row that passes the as-of and residual
conditions.
@@ -695,3 +703,22 @@ private[execution] object SortMergeAsOfJoinScanner {
case _ => false
}
}
+
+/**
+ * Builds a predicate whose AND, OR and IF still skip the parts they do not
need. The generated
+ * predicate skips subexpression elimination, which would run a repeated part
first. The
+ * interpreted fallback, used when code generation is off or fails, keeps
subexpression
+ * elimination, but runs a shared part only when an operand reaches it.
+ */
+private[joins] object ShortCircuitPredicate
+ extends CodeGeneratorWithInterpretedFallback[Expression, BasePredicate] {
+
+ def create(e: Expression, inputSchema: Seq[Attribute]): BasePredicate =
+ createObject(bindReference(e, inputSchema))
+
+ override protected def createCodeGeneratedObject(in: Expression):
BasePredicate =
+ GeneratePredicate.generate(in, useSubexprElimination = false)
+
+ override protected def createInterpretedObject(in: Expression):
BasePredicate =
+ InterpretedPredicate(in)
+}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSortMergeSQLSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSortMergeSQLSuite.scala
index da6caa107c1b..3a2ee710b50f 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSortMergeSQLSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSortMergeSQLSuite.scala
@@ -22,7 +22,7 @@ import java.sql.{Date, Timestamp}
import scala.util.Random
import org.apache.spark.SparkThrowable
-import org.apache.spark.sql.catalyst.expressions.Cast
+import org.apache.spark.sql.catalyst.expressions.{Cast,
CodegenObjectFactoryMode}
import org.apache.spark.sql.catalyst.plans.{AsOfJoinDirection, Backward,
Forward}
import org.apache.spark.sql.catalyst.plans.logical.AsOfJoin
import org.apache.spark.sql.execution.FileSourceScanExec
@@ -121,6 +121,25 @@ class AsOfJoinSortMergeSQLSuite extends QueryTest
}
}
+ /**
+ * Runs `f` in ANSI mode, so a division by zero or a bad cast throws. Runs
it with compiled
+ * and with interpreted conditions, each with subexpression elimination on
and off.
+ */
+ private def withGuardedConditionConfs(f: => Unit): Unit = {
+ val factoryModes =
+ Seq(CodegenObjectFactoryMode.CODEGEN_ONLY,
CodegenObjectFactoryMode.NO_CODEGEN)
+ for (factoryMode <- factoryModes; subexprElimination <- Seq("true",
"false")) {
+ withClue(s"factoryMode=$factoryMode,
subexprElimination=$subexprElimination: ") {
+ withSQLConf(
+ SQLConf.ANSI_ENABLED.key -> "true",
+ SQLConf.CODEGEN_FACTORY_MODE.key -> factoryMode.toString,
+ SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key ->
subexprElimination) {
+ f
+ }
+ }
+ }
+ }
+
/** The value of `metric` in the one SortMergeAsOfJoinExec of `df`, read
after `df` ran. */
private def asOfJoinMetric(df: DataFrame, metric: String): Long = {
val plan = df.queryExecution.executedPlan
@@ -1231,6 +1250,72 @@ class AsOfJoinSortMergeSQLSuite extends QueryTest
Row(Timestamp.valueOf("2026-06-29 10:00:12"), null)))
}
+ test("ON condition does not divide when an AND guard before it is FALSE") {
+ withGuardedConditionConfs {
+ // The closer right row has the same y, so `t.y <> r.y` must stop the
division by zero.
+ checkSortMergeAsOf(
+ sql(
+ """
+ |SELECT r.tag
+ |FROM VALUES (10, 6, 5) AS t(k, x, y)
+ |ASOF JOIN VALUES (5, 3, 'differs'), (8, 5, 'same') AS r(k, y, tag)
+ | MATCH_CONDITION (t.k >= r.k)
+ | ON t.y <> r.y AND t.x / (t.y - r.y) > 1 AND t.x / (t.y - r.y) <
10
+ |""".stripMargin),
+ Row("differs") :: Nil)
+ }
+ }
+
+ test("ON condition does not divide when an OR guard before it is TRUE") {
+ withGuardedConditionConfs {
+ // The closer right row has the same y, so `t.y = r.y` must match it
without the division.
+ checkSortMergeAsOf(
+ sql(
+ """
+ |SELECT r.tag
+ |FROM VALUES (10, 6, 5) AS t(k, x, y)
+ |ASOF JOIN VALUES (5, 3, 'differs'), (8, 5, 'same') AS r(k, y, tag)
+ | MATCH_CONDITION (t.k >= r.k)
+ | ON t.y = r.y OR (t.x / (t.y - r.y) > 1 AND t.x / (t.y - r.y) <
10)
+ |""".stripMargin),
+ Row("same") :: Nil)
+ }
+ }
+
+ test("LEFT ASOF JOIN ON condition does not cast when a left-only guard is
FALSE") {
+ withGuardedConditionConfs {
+ // Left row 2 fails `t.kind <> 'text'`, so its candidate 'abc' must
never be cast to INT.
+ // The guard is not `t.kind = 'num'`, which the planner would make a
join key.
+ checkSortMergeAsOf(
+ sql(
+ """
+ |SELECT t.id, r.v
+ |FROM VALUES (1, 6, 'num', 0, 100), (2, 10, 'text', 0, 100) AS
t(id, k, kind, lo, hi)
+ |LEFT ASOF JOIN VALUES (5, '42'), (8, 'abc') AS r(k, v)
+ | MATCH_CONDITION (t.k >= r.k)
+ | ON t.kind <> 'text' AND CAST(r.v AS INT) >= t.lo AND CAST(r.v
AS INT) <= t.hi
+ |""".stripMargin),
+ Seq(Row(1, "42"), Row(2, null)))
+ }
+ }
+
+ test("MATCH_CONDITION does not divide when an AND guard inside an IF is
FALSE") {
+ withGuardedConditionConfs {
+ // Row 1 has y = 0, so its IF must take t.b without the division. Row 2
takes t.a, which
+ // is below every right row.
+ checkSortMergeAsOf(
+ sql(
+ """
+ |SELECT t.id, r.v
+ |FROM VALUES (1, 1, 6, 0, 10, 20), (2, 1, 6, 2, 10, 20) AS t(id,
k, x, y, a, b)
+ |ASOF JOIN VALUES (1, 15, 'm') AS r(k, ts, v)
+ | MATCH_CONDITION (IF(t.y <> 0 AND t.x / t.y > 1 AND t.x / t.y <
5, t.a, t.b) >= r.ts)
+ | ON t.k = r.k
+ |""".stripMargin),
+ Row(1, "m") :: Nil)
+ }
+ }
+
test("LEFT ASOF JOIN null-pads NOT NULL right catalog columns") {
withTable("right_nn", "left_nn") {
sql("CREATE TABLE right_nn (k INT NOT NULL, v STRING NOT NULL) USING
parquet")
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinScannerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinScannerSuite.scala
index 9a69b8599958..73c67fafef24 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinScannerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinScannerSuite.scala
@@ -23,14 +23,19 @@ import org.apache.spark.{LocalSparkContext, SparkConf,
SparkContext, SparkFunSui
import org.apache.spark.sql.YearUDT
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext,
CodegenFallback, ExprCode}
import org.apache.spark.sql.catalyst.plans.{AsOfJoinDirection, Backward,
Forward, Inner, LeftOuter,
Nearest}
import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
import org.apache.spark.unsafe.types.UTF8String
class SortMergeAsOfJoinScannerSuite extends SparkFunSuite with
LocalSparkContext {
+ import SortMergeAsOfJoinScannerSuite._
+
+ private val factoryModes =
+ Seq(CodegenObjectFactoryMode.CODEGEN_ONLY,
CodegenObjectFactoryMode.NO_CODEGEN)
test("nearest scan stops at the first eligible row after the left key") {
sc = new SparkContext("local", "SortMergeAsOfJoinScannerSuite", new
SparkConf(false))
@@ -39,31 +44,9 @@ class SortMergeAsOfJoinScannerSuite extends SparkFunSuite
with LocalSparkContext
val leftKey = AttributeReference("leftKey", IntegerType)()
val rightKey = AttributeReference("rightKey", IntegerType)()
val count = new AtomicInteger()
- val scanner = new SortMergeAsOfJoinScanner(
- leftIter = Iterator(InternalRow(98)),
- rightIter = (0 to 200 by 10).iterator.map(i => InternalRow(i)),
- leftOutput = Seq(leftKey),
- rightOutput = Seq(rightKey),
- leftKeys = Nil,
- rightKeys = Nil,
- leftSortExprs = Seq(leftKey),
- rightSortExprs = Seq(rightKey),
- asOfCondition = CountedEvaluation(Literal.TrueLiteral, count),
- joinType = Inner,
- residualCondition = None,
- numOutputRows = new SQLMetric("sum"),
- spillSize = new SQLMetric("sum"),
- direction = Nearest,
- inMemoryThreshold = 256,
- sizeInBytesSpillThreshold = Long.MaxValue,
- spillThreshold = 256,
- useCursorScan = false,
- numAsOfCursorTasks = new SQLMetric("sum"))
- try {
- Iterator.single((scanner.iterator.map(_.getInt(1)).toList,
count.get()))
- } finally {
- scanner.close()
- }
+ val scanner = newScanner(leftKey, rightKey, Seq(98), 0 to 200 by 10,
+ asOfCondition = CountedEvaluation(Literal.TrueLiteral, count),
direction = Nearest)
+ Iterator.single((matchedRightKeys(scanner), count.get()))
}.collect().head
// The scan reads the right rows 0 to 100 and stops at 100, the first row
after 98.
@@ -71,6 +54,77 @@ class SortMergeAsOfJoinScannerSuite extends SparkFunSuite
with LocalSparkContext
assert(evaluations == 11, s"evaluated the as-of condition $evaluations
times")
}
+ gridTest("residual condition skips a repeated part behind a FALSE
guard")(factoryModes) { mode =>
+ sc = new SparkContext("local", "SortMergeAsOfJoinScannerSuite", new
SparkConf(false))
+ val (matchedKeys, evaluations) = sc.parallelize(Seq(0), 1)
+ .mapPartitions { _ =>
+ val leftKey = AttributeReference("leftKey", IntegerType)()
+ val rightKey = AttributeReference("rightKey", IntegerType)()
+ val count = new AtomicInteger()
+ val residual = falseGuardBeforeRepeatedPart(leftKey,
CountedEvaluation(rightKey, count))
+ withFactoryMode(mode) {
+ val scanner = newScanner(leftKey, rightKey, Seq(10), 0 until 10,
+ asOfCondition = GreaterThanOrEqual(leftKey, rightKey),
+ residualCondition = Some(residual))
+ Iterator.single((matchedRightKeys(scanner), count.get()))
+ }
+ }.collect().head
+
+ // All 10 right rows pass the as-of condition and then fail the guard.
+ assert(matchedKeys.isEmpty)
+ assert(evaluations == 0, s"evaluated the repeated part $evaluations times")
+ }
+
+ gridTest("as-of condition skips a repeated part behind a FALSE
guard")(factoryModes) { mode =>
+ sc = new SparkContext("local", "SortMergeAsOfJoinScannerSuite", new
SparkConf(false))
+ val (matchedKeys, evaluations) = sc.parallelize(Seq(0), 1)
+ .mapPartitions { _ =>
+ val leftKey = AttributeReference("leftKey", IntegerType)()
+ val rightKey = AttributeReference("rightKey", IntegerType)()
+ val count = new AtomicInteger()
+ val guard = falseGuardBeforeRepeatedPart(leftKey,
CountedEvaluation(rightKey, count))
+ withFactoryMode(mode) {
+ val scanner = newScanner(leftKey, rightKey, Seq(10), 0 until 10,
+ asOfCondition = GreaterThanOrEqual(If(guard, Literal(-1),
leftKey), rightKey))
+ Iterator.single((matchedRightKeys(scanner), count.get()))
+ }
+ }.collect().head
+
+ // The guard is FALSE, so the IF compares the left key 10, and the closest
right row is 9.
+ assert(matchedKeys == List(9))
+ assert(evaluations == 0, s"evaluated the repeated part $evaluations times")
+ }
+
+ test("conditions are built only when a scan checks a right row") {
+ sc = new SparkContext("local", "SortMergeAsOfJoinScannerSuite", new
SparkConf(false))
+ val compiled = sc.parallelize(Seq(0), 1)
+ .mapPartitions { _ =>
+ val leftKey = AttributeReference("leftKey", IntegerType)()
+ val rightKey = AttributeReference("rightKey", IntegerType)()
+ val failing = GreaterThanOrEqual(FailsToCompile(leftKey), rightKey)
+ // Whether the scan compiled `failing`, as its as-of or as its
residual condition.
+ def compiles(left: Seq[Int], right: Seq[Int], asOf: Expression,
+ residual: Option[Expression]): Boolean = {
+ try {
+ matchedRightKeys(newScanner(leftKey, rightKey, left, right, asOf,
residual))
+ false
+ } catch {
+ case _: FailsToCompileException => true
+ }
+ }
+ val sides = Seq((Seq.empty[Int], Seq(1)), (Seq(1), Seq.empty[Int]),
(Seq(1), Seq(1)))
+ withFactoryMode(CodegenObjectFactoryMode.CODEGEN_ONLY) {
+ Iterator.single(sides.map { case (left, right) =>
+ (compiles(left, right, failing, None),
+ compiles(left, right, Literal.TrueLiteral, Some(failing)))
+ })
+ }
+ }.collect().head
+
+ // An empty left or right side checks no right row, so neither condition
is compiled.
+ assert(compiled == Seq((false, false), (false, false), (true, true)))
+ }
+
test("cursor scan evaluates the as-of condition twice per left row") {
val leftKey = AttributeReference("leftKey", IntegerType)()
val rightKey = AttributeReference("rightKey", IntegerType)()
@@ -320,6 +374,62 @@ class SortMergeAsOfJoinScannerSuite extends SparkFunSuite
with LocalSparkContext
}
}
+private object SortMergeAsOfJoinScannerSuite {
+
+ /** A buffered scanner over one INT column per side, with no equi-keys. */
+ def newScanner(
+ leftKey: Attribute,
+ rightKey: Attribute,
+ leftKeys: Seq[Int],
+ rightKeys: Seq[Int],
+ asOfCondition: Expression,
+ residualCondition: Option[Expression] = None,
+ direction: AsOfJoinDirection = Backward): SortMergeAsOfJoinScanner = {
+ new SortMergeAsOfJoinScanner(
+ leftIter = leftKeys.iterator.map(k => InternalRow(k)),
+ rightIter = rightKeys.iterator.map(k => InternalRow(k)),
+ leftOutput = Seq(leftKey),
+ rightOutput = Seq(rightKey),
+ leftKeys = Nil,
+ rightKeys = Nil,
+ leftSortExprs = Seq(leftKey),
+ rightSortExprs = Seq(rightKey),
+ asOfCondition = asOfCondition,
+ joinType = Inner,
+ residualCondition = residualCondition,
+ numOutputRows = new SQLMetric("sum"),
+ spillSize = new SQLMetric("sum"),
+ direction = direction,
+ inMemoryThreshold = 256,
+ sizeInBytesSpillThreshold = Long.MaxValue,
+ spillThreshold = 256,
+ useCursorScan = false,
+ numAsOfCursorTasks = new SQLMetric("sum"))
+ }
+
+ /** Runs the scan to the end and returns the right key of each output row. */
+ def matchedRightKeys(scanner: SortMergeAsOfJoinScanner): List[Int] = {
+ try {
+ scanner.iterator.map(_.getInt(1)).toList
+ } finally {
+ scanner.close()
+ }
+ }
+
+ /** `leftKey < 0 AND part > 0 AND part < 100`: the guard is FALSE for the
left keys used here. */
+ def falseGuardBeforeRepeatedPart(leftKey: Expression, part: Expression):
Expression =
+ And(And(LessThan(leftKey, Literal(0)), GreaterThan(part, Literal(0))),
+ LessThan(part, Literal(100)))
+
+ /** Runs `f` with subexpression elimination on and the given codegen factory
mode. */
+ def withFactoryMode[T](mode: CodegenObjectFactoryMode.Value)(f: => T): T = {
+ val conf = new SQLConf
+ conf.setConfString(SQLConf.CODEGEN_FACTORY_MODE.key, mode.toString)
+ conf.setConfString(SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key, "true")
+ SQLConf.withExistingConf(conf)(f)
+ }
+}
+
private[joins] case class CountedEvaluation(child: Expression, counter:
AtomicInteger)
extends UnaryExpression with CodegenFallback {
override def nullable: Boolean = child.nullable
@@ -334,3 +444,17 @@ private[joins] case class CountedEvaluation(child:
Expression, counter: AtomicIn
override protected def withNewChildInternal(newChild: Expression):
CountedEvaluation =
copy(child = newChild)
}
+
+private[joins] class FailsToCompileException extends
RuntimeException("compiled")
+
+/** Evaluates its child, but throws when compiled, so a test can see when a
condition is built. */
+private[joins] case class FailsToCompile(child: Expression) extends
UnaryExpression {
+ override def dataType: DataType = child.dataType
+ override protected def nullSafeEval(input: Any): Any = input
+
+ override protected def doGenCode(ctx: CodegenContext, ev: ExprCode):
ExprCode =
+ throw new FailsToCompileException
+
+ override protected def withNewChildInternal(newChild: Expression):
FailsToCompile =
+ copy(child = newChild)
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]