This is an automated email from the ASF dual-hosted git repository.

cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 16340d91b03b [SPARK-59614][SQL] Compile the sort-merge ASOF join 
conditions and drop the output row copy
16340d91b03b is described below

commit 16340d91b03b88ee1ba8f32ae8ef12a140bfb26c
Author: Luka Zdravić <[email protected]>
AuthorDate: Fri Nov 6 03:38:19 2026 +0000

    [SPARK-59614][SQL] Compile the sort-merge ASOF join conditions and drop the 
output row copy
    
    ### What changes were proposed in this pull request?
    Two small changes to `SortMergeAsOfJoinScanner`:
    - It compiles the as-of and residual conditions, only when a scan first 
checks a right row. Before, each check walked the bound expression tree and 
boxed every value it read. The generated code skips subexpression elimination, 
which would compute `x / y` first in `y <> 0 AND x / y > 1 AND x / y < 10` and 
divide by zero. The interpreted fallback keeps subexpression elimination, but 
computes a shared part only when an operand reaches it. So AND, OR and IF still 
short-circuit on both paths.
    - It returns its output row without a copy. A consumer that keeps rows 
copies them.
    
    ### Why are the changes needed?
    The scan checks the as-of condition for every right row it reads, so these 
checks are its hot path.
    
    Local benchmark (best of 5, ms; 400K rows per side and 2000 keys, except 
the no-key cases with 10K rows per side):
    
    | Case | Before | After |
    |---|---|---|
    | backward | 970 | 613 |
    | forward | 990 | 600 |
    | left outer | 942 | 611 |
    | backward with tolerance | 2271 | 1159 |
    | nearest | 782 | 773 |
    | no key | 723 | 437 |
    | no key, with an extra ON condition | 2155 | 610 |
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Existing tests: `SortMergeAsOfJoinScannerSuite`, 
`AsOfJoinSortMergeSQLSuite`, `SortMergeAsOfJoinSuite`, `AsOfJoinSQLSuite`, 
`DataFrameAsOfJoinSuite`, the ASOF test in `SQLMetricsSuite`, `PlannerSuite`, 
and the `join-asof-*.sql` golden files. Tests such as "null as-of keys" check 
that a NULL condition result still counts as no match.
    
    To check that no row is lost without the copy, a local test fed the join 
output into operators that keep rows (sort, limit, window, shuffle, distinct, 
union, `collect_list`, a second sort-merge join, cache, `toLocalIterator`), 
with whole-stage codegen on and off. All 37 cases matched the correlated 
subquery path. All 37 failed when the scanner was changed to overwrite its row 
early.
    
    New tests put a guard before a part that throws in ANSI mode: in the ON 
condition, a division behind an AND guard, the same division behind an OR 
guard, and a cast behind a left-only guard in a LEFT ASOF JOIN; in the 
MATCH_CONDITION, a division behind an AND guard inside an IF. They run with 
compiled and with interpreted conditions, each with subexpression elimination 
on and off. Scanner tests check that a repeated part behind a FALSE guard never 
runs in either condition, and that a t [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Claude Code
    
    Closes #60029 from LukaZdravic/asof-jvm-per-row-overhead.
    
    Authored-by: Luka Zdravić <[email protected]>
    Signed-off-by: Wenchen Fan <[email protected]>
    (cherry picked from commit e657651890acdc26185c1eec910473e4127de66e)
    Signed-off-by: Wenchen Fan <[email protected]>
---
 .../execution/joins/SortMergeAsOfJoinExec.scala    |  53 +++++--
 .../spark/sql/AsOfJoinSortMergeSQLSuite.scala      |  87 +++++++++-
 .../joins/SortMergeAsOfJoinScannerSuite.scala      | 176 ++++++++++++++++++---
 3 files changed, 276 insertions(+), 40 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala
index ef1f8650ee52..448a31dfcb09 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala
@@ -24,7 +24,8 @@ import org.apache.spark.rdd.RDD
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions._
 import org.apache.spark.sql.catalyst.expressions.BindReferences.bindReference
-import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, 
GenerateOrdering}
+import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, 
GenerateOrdering,
+  GeneratePredicate}
 import org.apache.spark.sql.catalyst.plans._
 import org.apache.spark.sql.catalyst.plans.logical.AsOfJoin
 import org.apache.spark.sql.catalyst.plans.physical._
@@ -175,7 +176,8 @@ private[joins] class SortMergeAsOfJoinScanner(
   private val joinedOutput = leftOutput ++ rightOutput
   private val joinedRow = new JoinedRow()
   // Use nullable bound references so outer-join null padding is safe even when
-  // right attributes are NOT NULL in the catalog schema.
+  // right attributes are NOT NULL in the catalog schema. Every output row 
reuses the
+  // projection's row.
   private val resultProjection = {
     val nullableRefs = joinedOutput.zipWithIndex.map { case (attr, i) =>
       BoundReference(i, attr.dataType, nullable = true)
@@ -183,9 +185,14 @@ private[joins] class SortMergeAsOfJoinScanner(
     UnsafeProjection.create(nullableRefs, joinedOutput)
   }
 
-  private val boundAsOfCond = bindReference(asOfCondition, joinedOutput)
-  private val boundResidualCond =
-    residualCondition.map(bindReference(_, joinedOutput))
+  // A repeated part must not run before its guard: `y <> 0 AND x / y > 1 AND 
x / y < 10` would
+  // divide by zero when y is 0. So the generated predicates skip 
subexpression elimination. The
+  // interpreted fallback keeps it, but runs a shared part only when an 
operand reaches it, so
+  // AND, OR and IF still short-circuit. Both are lazy, so a task that checks 
no right row
+  // compiles neither condition.
+  private lazy val boundAsOfCond = ShortCircuitPredicate.create(asOfCondition, 
joinedOutput)
+  private lazy val boundResidualCond =
+    residualCondition.map(ShortCircuitPredicate.create(_, joinedOutput))
 
   private val equiKeyOrdering: Option[BaseOrdering] =
     if (leftKeys.nonEmpty) {
@@ -291,7 +298,7 @@ private[joins] class SortMergeAsOfJoinScanner(
           if (joinType == LeftOuter) {
             numOutputRows += 1
             joinedRow.withLeft(leftRow).withRight(nullRightRow)
-            return resultProjection(joinedRow).copy()
+            return resultProjection(joinedRow)
           }
         } else {
           val bestMatch = if (useCursorScan) {
@@ -304,11 +311,11 @@ private[joins] class SortMergeAsOfJoinScanner(
           if (bestMatch != null) {
             numOutputRows += 1
             joinedRow.withLeft(leftRow).withRight(bestMatch)
-            return resultProjection(joinedRow).copy()
+            return resultProjection(joinedRow)
           } else if (joinType == LeftOuter) {
             numOutputRows += 1
             joinedRow.withLeft(leftRow).withRight(nullRightRow)
-            return resultProjection(joinedRow).copy()
+            return resultProjection(joinedRow)
           }
         }
       }
@@ -407,13 +414,14 @@ private[joins] class SortMergeAsOfJoinScanner(
   }
 
   /** Whether `cond` is TRUE on the current joined row. FALSE and NULL both 
fail. */
-  private def holds(cond: Expression): Boolean = {
-    val result = cond.eval(joinedRow)
-    result != null && result.asInstanceOf[Boolean]
-  }
+  private def holds(cond: BasePredicate): Boolean = cond.eval(joinedRow)
 
   /** Whether the residual ON condition, if any, holds on the current joined 
row. */
-  private def residualHolds: Boolean = boundResidualCond.forall(holds)
+  private def residualHolds: Boolean = boundResidualCond match {
+    // A match, not `forall`, so the per-row check makes no closure.
+    case Some(cond) => cond.eval(joinedRow)
+    case None => true
+  }
 
   /**
    * Backward joins keep the last row that passes the as-of and residual 
conditions.
@@ -695,3 +703,22 @@ private[execution] object SortMergeAsOfJoinScanner {
     case _ => false
   }
 }
+
+/**
+ * Builds a predicate whose AND, OR and IF still skip the parts they do not 
need. The generated
+ * predicate skips subexpression elimination, which would run a repeated part 
first. The
+ * interpreted fallback, used when code generation is off or fails, keeps 
subexpression
+ * elimination, but runs a shared part only when an operand reaches it.
+ */
+private[joins] object ShortCircuitPredicate
+    extends CodeGeneratorWithInterpretedFallback[Expression, BasePredicate] {
+
+  def create(e: Expression, inputSchema: Seq[Attribute]): BasePredicate =
+    createObject(bindReference(e, inputSchema))
+
+  override protected def createCodeGeneratedObject(in: Expression): 
BasePredicate =
+    GeneratePredicate.generate(in, useSubexprElimination = false)
+
+  override protected def createInterpretedObject(in: Expression): 
BasePredicate =
+    InterpretedPredicate(in)
+}
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSortMergeSQLSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSortMergeSQLSuite.scala
index da6caa107c1b..3a2ee710b50f 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSortMergeSQLSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/AsOfJoinSortMergeSQLSuite.scala
@@ -22,7 +22,7 @@ import java.sql.{Date, Timestamp}
 import scala.util.Random
 
 import org.apache.spark.SparkThrowable
-import org.apache.spark.sql.catalyst.expressions.Cast
+import org.apache.spark.sql.catalyst.expressions.{Cast, 
CodegenObjectFactoryMode}
 import org.apache.spark.sql.catalyst.plans.{AsOfJoinDirection, Backward, 
Forward}
 import org.apache.spark.sql.catalyst.plans.logical.AsOfJoin
 import org.apache.spark.sql.execution.FileSourceScanExec
@@ -121,6 +121,25 @@ class AsOfJoinSortMergeSQLSuite extends QueryTest
     }
   }
 
+  /**
+   * Runs `f` in ANSI mode, so a division by zero or a bad cast throws. Runs 
it with compiled
+   * and with interpreted conditions, each with subexpression elimination on 
and off.
+   */
+  private def withGuardedConditionConfs(f: => Unit): Unit = {
+    val factoryModes =
+      Seq(CodegenObjectFactoryMode.CODEGEN_ONLY, 
CodegenObjectFactoryMode.NO_CODEGEN)
+    for (factoryMode <- factoryModes; subexprElimination <- Seq("true", 
"false")) {
+      withClue(s"factoryMode=$factoryMode, 
subexprElimination=$subexprElimination: ") {
+        withSQLConf(
+            SQLConf.ANSI_ENABLED.key -> "true",
+            SQLConf.CODEGEN_FACTORY_MODE.key -> factoryMode.toString,
+            SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key -> 
subexprElimination) {
+          f
+        }
+      }
+    }
+  }
+
   /** The value of `metric` in the one SortMergeAsOfJoinExec of `df`, read 
after `df` ran. */
   private def asOfJoinMetric(df: DataFrame, metric: String): Long = {
     val plan = df.queryExecution.executedPlan
@@ -1231,6 +1250,72 @@ class AsOfJoinSortMergeSQLSuite extends QueryTest
         Row(Timestamp.valueOf("2026-06-29 10:00:12"), null)))
   }
 
+  test("ON condition does not divide when an AND guard before it is FALSE") {
+    withGuardedConditionConfs {
+      // The closer right row has the same y, so `t.y <> r.y` must stop the 
division by zero.
+      checkSortMergeAsOf(
+        sql(
+          """
+            |SELECT r.tag
+            |FROM VALUES (10, 6, 5) AS t(k, x, y)
+            |ASOF JOIN VALUES (5, 3, 'differs'), (8, 5, 'same') AS r(k, y, tag)
+            |  MATCH_CONDITION (t.k >= r.k)
+            |  ON t.y <> r.y AND t.x / (t.y - r.y) > 1 AND t.x / (t.y - r.y) < 
10
+            |""".stripMargin),
+        Row("differs") :: Nil)
+    }
+  }
+
+  test("ON condition does not divide when an OR guard before it is TRUE") {
+    withGuardedConditionConfs {
+      // The closer right row has the same y, so `t.y = r.y` must match it 
without the division.
+      checkSortMergeAsOf(
+        sql(
+          """
+            |SELECT r.tag
+            |FROM VALUES (10, 6, 5) AS t(k, x, y)
+            |ASOF JOIN VALUES (5, 3, 'differs'), (8, 5, 'same') AS r(k, y, tag)
+            |  MATCH_CONDITION (t.k >= r.k)
+            |  ON t.y = r.y OR (t.x / (t.y - r.y) > 1 AND t.x / (t.y - r.y) < 
10)
+            |""".stripMargin),
+        Row("same") :: Nil)
+    }
+  }
+
+  test("LEFT ASOF JOIN ON condition does not cast when a left-only guard is 
FALSE") {
+    withGuardedConditionConfs {
+      // Left row 2 fails `t.kind <> 'text'`, so its candidate 'abc' must 
never be cast to INT.
+      // The guard is not `t.kind = 'num'`, which the planner would make a 
join key.
+      checkSortMergeAsOf(
+        sql(
+          """
+            |SELECT t.id, r.v
+            |FROM VALUES (1, 6, 'num', 0, 100), (2, 10, 'text', 0, 100) AS 
t(id, k, kind, lo, hi)
+            |LEFT ASOF JOIN VALUES (5, '42'), (8, 'abc') AS r(k, v)
+            |  MATCH_CONDITION (t.k >= r.k)
+            |  ON t.kind <> 'text' AND CAST(r.v AS INT) >= t.lo AND CAST(r.v 
AS INT) <= t.hi
+            |""".stripMargin),
+        Seq(Row(1, "42"), Row(2, null)))
+    }
+  }
+
+  test("MATCH_CONDITION does not divide when an AND guard inside an IF is 
FALSE") {
+    withGuardedConditionConfs {
+      // Row 1 has y = 0, so its IF must take t.b without the division. Row 2 
takes t.a, which
+      // is below every right row.
+      checkSortMergeAsOf(
+        sql(
+          """
+            |SELECT t.id, r.v
+            |FROM VALUES (1, 1, 6, 0, 10, 20), (2, 1, 6, 2, 10, 20) AS t(id, 
k, x, y, a, b)
+            |ASOF JOIN VALUES (1, 15, 'm') AS r(k, ts, v)
+            |  MATCH_CONDITION (IF(t.y <> 0 AND t.x / t.y > 1 AND t.x / t.y < 
5, t.a, t.b) >= r.ts)
+            |  ON t.k = r.k
+            |""".stripMargin),
+        Row(1, "m") :: Nil)
+    }
+  }
+
   test("LEFT ASOF JOIN null-pads NOT NULL right catalog columns") {
     withTable("right_nn", "left_nn") {
       sql("CREATE TABLE right_nn (k INT NOT NULL, v STRING NOT NULL) USING 
parquet")
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinScannerSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinScannerSuite.scala
index 9a69b8599958..73c67fafef24 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinScannerSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinScannerSuite.scala
@@ -23,14 +23,19 @@ import org.apache.spark.{LocalSparkContext, SparkConf, 
SparkContext, SparkFunSui
 import org.apache.spark.sql.YearUDT
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.codegen.CodegenFallback
+import org.apache.spark.sql.catalyst.expressions.codegen.{CodegenContext, 
CodegenFallback, ExprCode}
 import org.apache.spark.sql.catalyst.plans.{AsOfJoinDirection, Backward, 
Forward, Inner, LeftOuter,
   Nearest}
 import org.apache.spark.sql.execution.metric.SQLMetric
+import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types._
 import org.apache.spark.unsafe.types.UTF8String
 
 class SortMergeAsOfJoinScannerSuite extends SparkFunSuite with 
LocalSparkContext {
+  import SortMergeAsOfJoinScannerSuite._
+
+  private val factoryModes =
+    Seq(CodegenObjectFactoryMode.CODEGEN_ONLY, 
CodegenObjectFactoryMode.NO_CODEGEN)
 
   test("nearest scan stops at the first eligible row after the left key") {
     sc = new SparkContext("local", "SortMergeAsOfJoinScannerSuite", new 
SparkConf(false))
@@ -39,31 +44,9 @@ class SortMergeAsOfJoinScannerSuite extends SparkFunSuite 
with LocalSparkContext
         val leftKey = AttributeReference("leftKey", IntegerType)()
         val rightKey = AttributeReference("rightKey", IntegerType)()
         val count = new AtomicInteger()
-        val scanner = new SortMergeAsOfJoinScanner(
-          leftIter = Iterator(InternalRow(98)),
-          rightIter = (0 to 200 by 10).iterator.map(i => InternalRow(i)),
-          leftOutput = Seq(leftKey),
-          rightOutput = Seq(rightKey),
-          leftKeys = Nil,
-          rightKeys = Nil,
-          leftSortExprs = Seq(leftKey),
-          rightSortExprs = Seq(rightKey),
-          asOfCondition = CountedEvaluation(Literal.TrueLiteral, count),
-          joinType = Inner,
-          residualCondition = None,
-          numOutputRows = new SQLMetric("sum"),
-          spillSize = new SQLMetric("sum"),
-          direction = Nearest,
-          inMemoryThreshold = 256,
-          sizeInBytesSpillThreshold = Long.MaxValue,
-          spillThreshold = 256,
-          useCursorScan = false,
-          numAsOfCursorTasks = new SQLMetric("sum"))
-        try {
-          Iterator.single((scanner.iterator.map(_.getInt(1)).toList, 
count.get()))
-        } finally {
-          scanner.close()
-        }
+        val scanner = newScanner(leftKey, rightKey, Seq(98), 0 to 200 by 10,
+          asOfCondition = CountedEvaluation(Literal.TrueLiteral, count), 
direction = Nearest)
+        Iterator.single((matchedRightKeys(scanner), count.get()))
       }.collect().head
 
     // The scan reads the right rows 0 to 100 and stops at 100, the first row 
after 98.
@@ -71,6 +54,77 @@ class SortMergeAsOfJoinScannerSuite extends SparkFunSuite 
with LocalSparkContext
     assert(evaluations == 11, s"evaluated the as-of condition $evaluations 
times")
   }
 
+  gridTest("residual condition skips a repeated part behind a FALSE 
guard")(factoryModes) { mode =>
+    sc = new SparkContext("local", "SortMergeAsOfJoinScannerSuite", new 
SparkConf(false))
+    val (matchedKeys, evaluations) = sc.parallelize(Seq(0), 1)
+      .mapPartitions { _ =>
+        val leftKey = AttributeReference("leftKey", IntegerType)()
+        val rightKey = AttributeReference("rightKey", IntegerType)()
+        val count = new AtomicInteger()
+        val residual = falseGuardBeforeRepeatedPart(leftKey, 
CountedEvaluation(rightKey, count))
+        withFactoryMode(mode) {
+          val scanner = newScanner(leftKey, rightKey, Seq(10), 0 until 10,
+            asOfCondition = GreaterThanOrEqual(leftKey, rightKey),
+            residualCondition = Some(residual))
+          Iterator.single((matchedRightKeys(scanner), count.get()))
+        }
+      }.collect().head
+
+    // All 10 right rows pass the as-of condition and then fail the guard.
+    assert(matchedKeys.isEmpty)
+    assert(evaluations == 0, s"evaluated the repeated part $evaluations times")
+  }
+
+  gridTest("as-of condition skips a repeated part behind a FALSE 
guard")(factoryModes) { mode =>
+    sc = new SparkContext("local", "SortMergeAsOfJoinScannerSuite", new 
SparkConf(false))
+    val (matchedKeys, evaluations) = sc.parallelize(Seq(0), 1)
+      .mapPartitions { _ =>
+        val leftKey = AttributeReference("leftKey", IntegerType)()
+        val rightKey = AttributeReference("rightKey", IntegerType)()
+        val count = new AtomicInteger()
+        val guard = falseGuardBeforeRepeatedPart(leftKey, 
CountedEvaluation(rightKey, count))
+        withFactoryMode(mode) {
+          val scanner = newScanner(leftKey, rightKey, Seq(10), 0 until 10,
+            asOfCondition = GreaterThanOrEqual(If(guard, Literal(-1), 
leftKey), rightKey))
+          Iterator.single((matchedRightKeys(scanner), count.get()))
+        }
+      }.collect().head
+
+    // The guard is FALSE, so the IF compares the left key 10, and the closest 
right row is 9.
+    assert(matchedKeys == List(9))
+    assert(evaluations == 0, s"evaluated the repeated part $evaluations times")
+  }
+
+  test("conditions are built only when a scan checks a right row") {
+    sc = new SparkContext("local", "SortMergeAsOfJoinScannerSuite", new 
SparkConf(false))
+    val compiled = sc.parallelize(Seq(0), 1)
+      .mapPartitions { _ =>
+        val leftKey = AttributeReference("leftKey", IntegerType)()
+        val rightKey = AttributeReference("rightKey", IntegerType)()
+        val failing = GreaterThanOrEqual(FailsToCompile(leftKey), rightKey)
+        // Whether the scan compiled `failing`, as its as-of or as its 
residual condition.
+        def compiles(left: Seq[Int], right: Seq[Int], asOf: Expression,
+            residual: Option[Expression]): Boolean = {
+          try {
+            matchedRightKeys(newScanner(leftKey, rightKey, left, right, asOf, 
residual))
+            false
+          } catch {
+            case _: FailsToCompileException => true
+          }
+        }
+        val sides = Seq((Seq.empty[Int], Seq(1)), (Seq(1), Seq.empty[Int]), 
(Seq(1), Seq(1)))
+        withFactoryMode(CodegenObjectFactoryMode.CODEGEN_ONLY) {
+          Iterator.single(sides.map { case (left, right) =>
+            (compiles(left, right, failing, None),
+              compiles(left, right, Literal.TrueLiteral, Some(failing)))
+          })
+        }
+      }.collect().head
+
+    // An empty left or right side checks no right row, so neither condition 
is compiled.
+    assert(compiled == Seq((false, false), (false, false), (true, true)))
+  }
+
   test("cursor scan evaluates the as-of condition twice per left row") {
     val leftKey = AttributeReference("leftKey", IntegerType)()
     val rightKey = AttributeReference("rightKey", IntegerType)()
@@ -320,6 +374,62 @@ class SortMergeAsOfJoinScannerSuite extends SparkFunSuite 
with LocalSparkContext
   }
 }
 
+private object SortMergeAsOfJoinScannerSuite {
+
+  /** A buffered scanner over one INT column per side, with no equi-keys. */
+  def newScanner(
+      leftKey: Attribute,
+      rightKey: Attribute,
+      leftKeys: Seq[Int],
+      rightKeys: Seq[Int],
+      asOfCondition: Expression,
+      residualCondition: Option[Expression] = None,
+      direction: AsOfJoinDirection = Backward): SortMergeAsOfJoinScanner = {
+    new SortMergeAsOfJoinScanner(
+      leftIter = leftKeys.iterator.map(k => InternalRow(k)),
+      rightIter = rightKeys.iterator.map(k => InternalRow(k)),
+      leftOutput = Seq(leftKey),
+      rightOutput = Seq(rightKey),
+      leftKeys = Nil,
+      rightKeys = Nil,
+      leftSortExprs = Seq(leftKey),
+      rightSortExprs = Seq(rightKey),
+      asOfCondition = asOfCondition,
+      joinType = Inner,
+      residualCondition = residualCondition,
+      numOutputRows = new SQLMetric("sum"),
+      spillSize = new SQLMetric("sum"),
+      direction = direction,
+      inMemoryThreshold = 256,
+      sizeInBytesSpillThreshold = Long.MaxValue,
+      spillThreshold = 256,
+      useCursorScan = false,
+      numAsOfCursorTasks = new SQLMetric("sum"))
+  }
+
+  /** Runs the scan to the end and returns the right key of each output row. */
+  def matchedRightKeys(scanner: SortMergeAsOfJoinScanner): List[Int] = {
+    try {
+      scanner.iterator.map(_.getInt(1)).toList
+    } finally {
+      scanner.close()
+    }
+  }
+
+  /** `leftKey < 0 AND part > 0 AND part < 100`: the guard is FALSE for the 
left keys used here. */
+  def falseGuardBeforeRepeatedPart(leftKey: Expression, part: Expression): 
Expression =
+    And(And(LessThan(leftKey, Literal(0)), GreaterThan(part, Literal(0))),
+      LessThan(part, Literal(100)))
+
+  /** Runs `f` with subexpression elimination on and the given codegen factory 
mode. */
+  def withFactoryMode[T](mode: CodegenObjectFactoryMode.Value)(f: => T): T = {
+    val conf = new SQLConf
+    conf.setConfString(SQLConf.CODEGEN_FACTORY_MODE.key, mode.toString)
+    conf.setConfString(SQLConf.SUBEXPRESSION_ELIMINATION_ENABLED.key, "true")
+    SQLConf.withExistingConf(conf)(f)
+  }
+}
+
 private[joins] case class CountedEvaluation(child: Expression, counter: 
AtomicInteger)
     extends UnaryExpression with CodegenFallback {
   override def nullable: Boolean = child.nullable
@@ -334,3 +444,17 @@ private[joins] case class CountedEvaluation(child: 
Expression, counter: AtomicIn
   override protected def withNewChildInternal(newChild: Expression): 
CountedEvaluation =
     copy(child = newChild)
 }
+
+private[joins] class FailsToCompileException extends 
RuntimeException("compiled")
+
+/** Evaluates its child, but throws when compiled, so a test can see when a 
condition is built. */
+private[joins] case class FailsToCompile(child: Expression) extends 
UnaryExpression {
+  override def dataType: DataType = child.dataType
+  override protected def nullSafeEval(input: Any): Any = input
+
+  override protected def doGenCode(ctx: CodegenContext, ev: ExprCode): 
ExprCode =
+    throw new FailsToCompileException
+
+  override protected def withNewChildInternal(newChild: Expression): 
FailsToCompile =
+    copy(child = newChild)
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to