This is an automated email from the ASF dual-hosted git repository.

sunchao pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 753a68f61fb9 [SPARK-58265][SQL] Reuse projected broadcast values for 
dynamic partition pruning
753a68f61fb9 is described below

commit 753a68f61fb97d2926ed0d63918a3a4273f63c9c
Author: Chao Sun <[email protected]>
AuthorDate: Mon Aug 3 17:17:18 2026 -0700

    [SPARK-58265][SQL] Reuse projected broadcast values for dynamic partition 
pruning
    
    ### What changes were proposed in this pull request?
    
    Tracks [SPARK-58265](https://issues.apache.org/jira/browse/SPARK-58265).
    
    Extend dynamic partition pruning so Spark can derive partition values from 
the complete rows of a broadcast hash relation it has already built, rather 
than only from the hash keys of that relation.
    
    Spark continues to prefer its existing, exact broadcast-key reuse. When 
that cannot satisfy a broadcast-only pruning filter, it looks for a broadcast 
already required by an earlier inner equi-join. If the broadcast comes from the 
expected source and uses the complete ordered join keys and the correct hash 
mode, Spark evaluates a supported deterministic partition expression against 
its stored value rows and feeds the resulting values into the existing 
partition-pruning path.
    
    The key difference is that Spark reuses work the query already has to 
perform. It does not create a second broadcast or rerun the filtering-side join 
just to discover the partitions to read. The same approach works with adaptive 
and nonadaptive planning and with V1 and V2 partition filtering.
    
    Correctness takes priority over pruning. The projected values must include 
every partition the original query could need; they may also include values 
from broadcast rows that an earlier join later rejects. If Spark cannot verify 
the broadcast, cannot safely evaluate the expression, cannot obtain runtime 
statistics, or exceeds a configured row or size limit, it discards the entire 
projected domain and continues without this optimization. An unavailable domain 
is never interpreted as a [...]
    
    ### Why are the changes needed?
    
    Broadcast hash relations contain full build-side rows, not just hash keys. 
Existing dynamic partition pruning can reuse the keys, but it cannot use 
another column from the same broadcast even when that column identifies the 
partitions of a later scan.
    
    For example, suppose `sales` is partitioned by `category`:
    
    ```sql
    SELECT s.*
    FROM sales s
    JOIN (
      SELECT /*+ BROADCAST(p) */
        p.category
      FROM products p
      JOIN promotions promo
        ON p.product_id = promo.product_id
      WHERE promo.active
    ) eligible
      ON s.category = eligible.category
    ```
    
    The first join broadcasts `products` using `product_id` as its hash key. 
Each broadcast row also contains `category`, but `category` is not part of that 
hash key:
    
    ```text
    product_id   category
    ----------   --------
    101          books
    202          toys
    303          clothing
    ```
    
    Suppose only products `101` and `202` have active promotions, and `sales` 
has four category partitions: `books`, `toys`, `clothing`, and `electronics`.
    
    Without this change, Spark cannot derive the `sales` partition values from 
the existing `product_id` broadcast. With 
`spark.sql.optimizer.dynamicPartitionPruning.reuseBroadcastOnly=true`, it drops 
the pruning filter and scans every `sales` partition:
    
    ```text
    Before:                 books, toys, clothing, electronics
    Exact active products:  books, toys
    Reused broadcast rows:  books, toys, clothing
    ```
    
    With this change, Spark reads `category` directly from the existing 
broadcast and avoids the `electronics` partition. It may still read `clothing`: 
the broadcast contains product `303` even though the promotions join will later 
eliminate it. That extra partition is a safe superset, not a regression: Spark 
previously scanned `clothing` as part of the full scan, and the original join 
still excludes inactive products from the final result.
    
    If Spark can already construct an exact reusable filter, it keeps using 
that filter first. The projected-broadcast path is for queries that would 
otherwise receive no broadcast-only pruning; it does not replace a more 
selective existing pruning strategy.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. The optimization is disabled by default and controlled only by internal 
SQL configurations. It does not introduce public configuration, SQL syntax, 
APIs, or changes to query results.
    
    ### How was this patch tested?
    
    Regression tests cover derived partition expressions; composite and 
single-column broadcast keys; dense and sparse long-key hash relations; 
duplicate, null, and unmatched broadcast rows; safe-superset pruning; 
feature-disabled behavior; row and byte limit fallback; preservation of 
Catalyst case-class identity and metadata across rewrites; and both passes of 
iterative V2 partition filtering.
    
    The shared dynamic partition pruning tests run against V1, V2, and 
runtime-filtering V2 sources with adaptive execution both enabled and disabled. 
Enhanced V2 connector tests separately verify that a valid projected domain 
prunes partitions and that an unavailable domain leaves all original partitions 
intact.
    
    The focused pruning, ordinary subquery, runtime-filter injection, and 
existing Bloom filter suites passed locally: **390 executed tests passed, with 
five skipped**.
    
    ```bash
    SERIAL_SBT_TESTS=1 ./build/sbt \
      'catalyst/testOnly 
org.apache.spark.sql.catalyst.expressions.DynamicPruningSubquerySuite' \
      'sql/testOnly org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOff 
org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOn 
org.apache.spark.sql.DynamicPartitionPruningV2SuiteAEOff 
org.apache.spark.sql.DynamicPartitionPruningV2SuiteAEOn 
org.apache.spark.sql.DynamicPartitionPruningV2FilterSuiteAEOff 
org.apache.spark.sql.DynamicPartitionPruningV2FilterSuiteAEOn 
org.apache.spark.sql.connector.DataSourceV2EnhancedRuntimePartitionFilterSuite'
    
    SERIAL_SBT_TESTS=1 ./build/sbt \
      'sql/testOnly org.apache.spark.sql.SubquerySuite 
org.apache.spark.sql.InjectRuntimeFilterSuite 
org.apache.spark.sql.BloomFilterAggregateQuerySuite'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-5)
    
    Closes #57437 from sunchao/dev/chao/codex/spark-projected-broadcast-dpp.
    
    Authored-by: Chao Sun <[email protected]>
    Signed-off-by: Chao Sun <[email protected]>
    (cherry picked from commit b2ce7f728571f0ce421834e6d61dfa020d76585e)
    Signed-off-by: Chao Sun <[email protected]>
---
 .../expressions/BroadcastValueProjection.scala     |  31 ++
 .../sql/catalyst/expressions/DynamicPruning.scala  |  34 +-
 .../ReusableBroadcastValueProjection.scala         | 111 ++++++
 .../org/apache/spark/sql/internal/SQLConf.scala    |  59 +++
 .../expressions/DynamicPruningSubquerySuite.scala  |  99 ++++-
 .../sql/execution/BroadcastValueProjector.scala    | 158 ++++++++
 .../ProjectedBroadcastValueSubqueryExec.scala      | 136 +++++++
 .../execution/SubqueryAdaptiveBroadcastExec.scala  |  10 +-
 .../PlanAdaptiveDynamicPruningFilters.scala        | 119 ++++--
 .../adaptive/PlanAdaptiveSubqueries.scala          |  11 +-
 .../datasources/v2/DataSourceV2Strategy.scala      |   3 +
 .../execution/datasources/v2/PushDownUtils.scala   |   4 +-
 .../dynamicpruning/PartitionPruning.scala          |   9 +-
 .../dynamicpruning/PlanDynamicPruningFilters.scala | 110 +++--
 .../metric/SQLLastAttemptAccumulator.scala         |   4 +-
 .../org/apache/spark/sql/execution/subquery.scala  |  38 +-
 .../spark/sql/DynamicPartitionPruningSuite.scala   | 443 ++++++++++++++++++++-
 .../org/apache/spark/sql/PlanStabilitySuite.scala  |   2 +
 .../scala/org/apache/spark/sql/SubquerySuite.scala |  14 +-
 ...urceV2EnhancedRuntimePartitionFilterSuite.scala |  93 ++++-
 20 files changed, 1374 insertions(+), 114 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BroadcastValueProjection.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BroadcastValueProjection.scala
new file mode 100644
index 000000000000..0058414987a9
--- /dev/null
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/BroadcastValueProjection.scala
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.expressions
+
+import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+
+/**
+ * Describes a value projection from the rows of an existing standard hash 
broadcast.
+ *
+ * The source hash keys are the complete, ordered keys of the broadcast join. 
The value expression
+ * is evaluated against the rows stored in the resulting hashed relation.
+ */
+case class BroadcastValueProjection(
+    sourcePlan: LogicalPlan,
+    sourceHashKeys: Seq[Expression],
+    valueExpression: Expression)
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
index 1c33ed65c1df..959acbc762b4 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruning.scala
@@ -39,6 +39,7 @@ trait DynamicPruning extends Predicate
  *  broadcast through ReuseExchange; otherwise, it will use the filter only if 
it
  *  can reuse the results of the broadcast through ReuseExchange
  * @param broadcastKeyIndices the indices of the filtering keys collected from 
the broadcast
+ * @param broadcastValueProjection an optional value projection from an 
existing broadcast
  */
 case class DynamicPruningSubquery(
     pruningKey: Expression,
@@ -47,28 +48,49 @@ case class DynamicPruningSubquery(
     broadcastKeyIndices: Seq[Int],
     onlyInBroadcast: Boolean,
     exprId: ExprId = NamedExpression.newExprId,
-    hint: Option[HintInfo] = None)
+    hint: Option[HintInfo] = None)(
+    @transient private[sql] val broadcastValueProjection: 
Option[BroadcastValueProjection] = None)
   extends SubqueryExpression(buildQuery, Seq(pruningKey), exprId, Seq.empty, 
hint)
   with DynamicPruning
   with Unevaluable
   with UnaryLike[Expression] {
 
+  override protected def otherCopyArgs: Seq[AnyRef] = 
Seq(broadcastValueProjection)
+
   override def child: Expression = pruningKey
 
   override def plan: LogicalPlan = buildQuery
 
   override def nullable: Boolean = false
 
-  override def withNewPlan(plan: LogicalPlan): DynamicPruningSubquery = 
copy(buildQuery = plan)
+  override def withNewPlan(plan: LogicalPlan): DynamicPruningSubquery =
+    copy(buildQuery = plan)(broadcastValueProjection)
 
   override def withNewOuterAttrs(outerAttrs: Seq[Expression]): 
DynamicPruningSubquery = {
     // Updating outer attrs of DynamicPruningSubquery is unsupported; assert 
that they match
     // pruningKey and return a copy without any changes.
     assert(outerAttrs.size == 1 && outerAttrs.head.semanticEquals(pruningKey))
-    copy()
+    copy()(broadcastValueProjection)
   }
 
-  override def withNewHint(hint: Option[HintInfo]): SubqueryExpression = 
copy(hint = hint)
+  override def withNewHint(hint: Option[HintInfo]): SubqueryExpression =
+    copy(hint = hint)(broadcastValueProjection)
+
+  private[sql] def usableBroadcastValueProjection: 
Option[BroadcastValueProjection] = {
+    broadcastValueProjection.filter { projection =>
+      projection.sourcePlan.resolved &&
+        projection.sourcePlan.deterministic &&
+        projection.sourceHashKeys.nonEmpty &&
+        projection.sourceHashKeys.forall(_.resolved) &&
+        projection.sourceHashKeys.forall(_.deterministic) &&
+        projection.sourceHashKeys.forall(
+          _.references.subsetOf(projection.sourcePlan.outputSet)) &&
+        projection.valueExpression.resolved &&
+        projection.valueExpression.deterministic &&
+        
projection.valueExpression.references.subsetOf(projection.sourcePlan.outputSet) 
&&
+        projection.valueExpression.dataType == pruningKey.dataType
+    }
+  }
 
   override lazy val resolved: Boolean = {
     pruningKey.resolved &&
@@ -92,11 +114,11 @@ case class DynamicPruningSubquery(
       pruningKey = pruningKey.canonicalized,
       buildQuery = buildQuery.canonicalized,
       buildKeys = buildKeys.map(QueryPlan.normalizeExpressions(_, 
buildQuery.output)),
-      exprId = ExprId(0))
+      exprId = ExprId(0))(None)
   }
 
   override protected def withNewChildInternal(newChild: Expression): 
DynamicPruningSubquery =
-    copy(pruningKey = newChild)
+    copy(pruningKey = newChild)(broadcastValueProjection)
 }
 
 /**
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReusableBroadcastValueProjection.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReusableBroadcastValueProjection.scala
new file mode 100644
index 000000000000..8841396adbbe
--- /dev/null
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/ReusableBroadcastValueProjection.scala
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys
+import org.apache.spark.sql.catalyst.plans.Inner
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.trees.TreePattern.{CTE, 
DYNAMIC_PRUNING_SUBQUERY}
+import org.apache.spark.sql.types.{DateType, StringType, TimestampNTZType, 
TimestampType}
+
+/**
+ * Finds an existing broadcast whose stored rows provide a safe superset of a 
pruning domain.
+ */
+private[sql] object ReusableBroadcastValueProjection extends PredicateHelper {
+
+  // Keep this an allowlist: projected values are evaluated directly from 
broadcast rows, so
+  // supported expressions must be deterministic and row-local, without 
subqueries or outer
+  // references, and preserve their time-zone, collation, and equality 
semantics. Extend it only
+  // when those properties hold; runtime limits and recoverable failures 
disable projection.
+  private def isSafeValueExpression(expression: Expression): Boolean = {
+    expression.deterministic && !expression.exists {
+      case _: OuterReference | _: SubqueryExpression => true
+      case _ => false
+    } && (expression match {
+      case attribute: Attribute =>
+        UnsafeRow.isFixedLength(attribute.dataType) || attribute.dataType == 
StringType
+      case _: Literal => true
+      case DateAdd(startDate, days: Literal) =>
+        isSafeValueExpression(startDate) && isSafeValueExpression(days)
+      case cast: Cast if cast.child.dataType == DateType &&
+          (cast.dataType == TimestampType || cast.dataType == 
TimestampNTZType) =>
+        isSafeValueExpression(cast.child)
+      case DateFormatClass(timestamp, format: Literal, Some(_))
+          if format.dataType == StringType && format.value != null =>
+        isSafeValueExpression(timestamp)
+      case _ => false
+    })
+  }
+
+  private def isSafeSourceHashKey(expression: Expression): Boolean =
+    expression.isInstanceOf[Attribute] && expression.deterministic
+
+  /**
+   * Follows deterministic projections to the first inner equality join. 
Unmatched broadcast rows
+   * can add partition values, but cannot remove a partition needed by the 
actual join.
+   */
+  def find(
+      valueExpression: Expression,
+      valuePlan: LogicalPlan,
+      excludedPlan: LogicalPlan): Option[BroadcastValueProjection] = {
+
+    def descend(
+        value: Expression,
+        plan: LogicalPlan): Option[BroadcastValueProjection] = {
+      plan match {
+        case project: Project if project.projectList.forall(_.deterministic) =>
+          val rewritten = replaceAlias(value, getAliasMap(project))
+          if (rewritten.references.subsetOf(project.child.outputSet) &&
+              isSafeValueExpression(rewritten)) {
+            descend(rewritten, project.child)
+          } else {
+            None
+          }
+
+        case ExtractEquiJoinKeys(Inner, leftKeys, rightKeys, _, _, left, 
right, _) =>
+          val valueFromLeft = value.references.nonEmpty &&
+            value.references.subsetOf(left.outputSet)
+          val valueFromRight = value.references.nonEmpty &&
+            value.references.subsetOf(right.outputSet)
+          val candidate = (valueFromLeft, valueFromRight) match {
+            case (true, false) => Some((left, leftKeys))
+            case (false, true) => Some((right, rightKeys))
+            case _ => None
+          }
+
+          candidate.filter { case (source, hashKeys) =>
+            !source.isStreaming && source.deterministic &&
+              hashKeys.nonEmpty && hashKeys.forall(isSafeSourceHashKey) &&
+              !source.containsAnyPattern(CTE, DYNAMIC_PRUNING_SUBQUERY) &&
+              !source.exists(_.sameResult(excludedPlan))
+          }.map { case (source, hashKeys) =>
+            BroadcastValueProjection(source, hashKeys, value)
+          }
+
+        case _ => None
+      }
+    }
+
+    if (valuePlan.deterministic && isSafeValueExpression(valueExpression)) {
+      descend(valueExpression, valuePlan)
+    } else {
+      None
+    }
+  }
+}
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 8a5b54e5cbcd..8a8611ba9684 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -828,6 +828,53 @@ object SQLConf {
       .booleanConf
       .createWithDefault(true)
 
+  val DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED =
+    
buildConf("spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.enabled")
+      .internal()
+      .doc("When true, dynamic partition pruning may project a bounded value 
domain from the " +
+        "full rows of an existing ancestor broadcast. When false, Spark uses 
its existing " +
+        "broadcast-key reuse and subquery fallback behavior.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .booleanConf
+      .createWithDefault(false)
+
+  val DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_ROWS =
+    
buildConf("spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.maxRows")
+      .internal()
+      .doc("Maximum number of broadcast value rows visited when projecting a 
dynamic partition " +
+        "pruning domain. If the limit is exceeded, the projection fails open 
rather than " +
+        "using a partial domain.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .intConf
+      .checkValue(_ >= 0, "The maximum number of DPP broadcast rows must be 
nonnegative.")
+      .createWithDefault(10000)
+
+  val DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_BYTES =
+    
buildConf("spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.maxBytes")
+      .internal()
+      .doc("Maximum cumulative size of distinct values projected from a 
broadcast for dynamic " +
+        "partition pruning. If the limit is exceeded, the projection fails 
open rather than " +
+        "using a partial domain.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .bytesConf(ByteUnit.BYTE)
+      .checkValue(_ >= 0, "The maximum size of DPP broadcast values must be 
nonnegative.")
+      .createWithDefaultString("8MB")
+
+  val DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_SOURCE_BYTES =
+    
buildConf("spark.sql.optimizer.dynamicPartitionPruning.broadcastProjection.maxSourceBytes")
+      .internal()
+      .doc("Maximum materialized broadcast size that may be rehydrated on the 
driver when " +
+        "projecting dynamic partition pruning values. If runtime statistics 
are unavailable " +
+        "or exceed the limit, the projection fails open.")
+      .version("4.3.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .bytesConf(ByteUnit.BYTE)
+      .checkValue(_ >= 0, "The maximum DPP source broadcast size must be 
nonnegative.")
+      .createWithDefaultString("128MB")
+
   val RUNTIME_FILTER_NUMBER_THRESHOLD =
     buildConf("spark.sql.optimizer.runtimeFilter.number.threshold")
       .doc("The total number of injected runtime filters (non-DPP) for a 
single " +
@@ -8380,6 +8427,18 @@ class SQLConf extends Serializable with Logging with 
SqlApiConf {
   def dynamicPartitionPruningReuseBroadcastOnly: Boolean =
     getConf(DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY)
 
+  def dynamicPartitionPruningBroadcastProjectionEnabled: Boolean =
+    getConf(DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED)
+
+  def dynamicPartitionPruningBroadcastProjectionMaxRows: Int =
+    getConf(DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_ROWS)
+
+  def dynamicPartitionPruningBroadcastProjectionMaxBytes: Long =
+    getConf(DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_BYTES)
+
+  def dynamicPartitionPruningBroadcastProjectionMaxSourceBytes: Long =
+    getConf(DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_SOURCE_BYTES)
+
   def runtimeFilterBloomFilterEnabled: Boolean =
     getConf(RUNTIME_BLOOM_FILTER_ENABLED)
 
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruningSubquerySuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruningSubquerySuite.scala
index 614a29c5ac4a..61700aa0d860 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruningSubquerySuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/DynamicPruningSubquerySuite.scala
@@ -17,8 +17,12 @@
 
 package org.apache.spark.sql.catalyst.expressions
 
+import java.lang.reflect.Modifier
+
 import org.apache.spark.SparkFunSuite
-import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, Project}
+import org.apache.spark.sql.catalyst.optimizer.ReusableBroadcastValueProjection
+import org.apache.spark.sql.catalyst.plans.Inner
+import org.apache.spark.sql.catalyst.plans.logical.{Join, JoinHint, 
LocalRelation, Project}
 import org.apache.spark.sql.types.IntegerType
 
 class DynamicPruningSubquerySuite extends SparkFunSuite {
@@ -31,59 +35,59 @@ class DynamicPruningSubquerySuite extends SparkFunSuite {
     buildKeys = Seq(pruningKeyExpression),
     broadcastKeyIndices = Seq(0),
     onlyInBroadcast = false
-  )
+  )()
 
   test("pruningKey data type matches single buildKey") {
     val dynamicPruningSubquery = validDynamicPruningSubquery
-      .copy(buildKeys = Seq(Literal(2023)))
+      .copy(buildKeys = Seq(Literal(2023)))(None)
     assert(dynamicPruningSubquery.resolved == true)
   }
 
   test("pruningKey data type is a Struct and matches with Struct buildKey") {
     val dynamicPruningSubquery = validDynamicPruningSubquery
       .copy(pruningKey = CreateStruct(Seq(Literal(1), Literal.FalseLiteral)),
-        buildKeys = Seq(CreateStruct(Seq(Literal(2), Literal.TrueLiteral))))
+        buildKeys = Seq(CreateStruct(Seq(Literal(2), 
Literal.TrueLiteral))))(None)
     assert(dynamicPruningSubquery.resolved == true)
   }
 
   test("multiple buildKeys but only one broadcastKeyIndex") {
     val dynamicPruningSubquery = validDynamicPruningSubquery
       .copy(buildKeys = Seq(Literal(0), Literal(2), Literal(0), Literal(9)),
-        broadcastKeyIndices = Seq(1))
+        broadcastKeyIndices = Seq(1))(None)
     assert(dynamicPruningSubquery.resolved == true)
   }
 
   test("pruningKey data type does not match the single buildKey") {
     val dynamicPruningSubquery = validDynamicPruningSubquery.copy(
       pruningKey = Literal.TrueLiteral,
-      buildKeys = Seq(Literal(2013)))
+      buildKeys = Seq(Literal(2013)))(None)
     assert(dynamicPruningSubquery.resolved == false)
   }
 
   test("pruningKey data type is a Struct but mismatch with Struct buildKey") {
     val dynamicPruningSubquery = validDynamicPruningSubquery
       .copy(pruningKey = CreateStruct(Seq(Literal(1), Literal.FalseLiteral)),
-        buildKeys = Seq(CreateStruct(Seq(Literal.TrueLiteral, Literal(2)))))
+        buildKeys = Seq(CreateStruct(Seq(Literal.TrueLiteral, 
Literal(2)))))(None)
     assert(dynamicPruningSubquery.resolved == false)
   }
 
   test("DynamicPruningSubquery should only have a single broadcasting key") {
     val dynamicPruningSubquery = validDynamicPruningSubquery
       .copy(buildKeys = Seq(Literal(2025), Literal(2), Literal(1809)),
-        broadcastKeyIndices = Seq(0, 2))
+        broadcastKeyIndices = Seq(0, 2))(None)
     assert(dynamicPruningSubquery.resolved == false)
   }
 
   test("duplicates in broadcastKeyIndices, and also should not be allowed") {
     val dynamicPruningSubquery = validDynamicPruningSubquery
       .copy(buildKeys = Seq(Literal(2)),
-        broadcastKeyIndices = Seq(0, 0))
+        broadcastKeyIndices = Seq(0, 0))(None)
     assert(dynamicPruningSubquery.resolved == false)
   }
 
   test("broadcastKeyIndex out of bounds") {
     val dynamicPruningSubquery = validDynamicPruningSubquery
-      .copy(broadcastKeyIndices = Seq(1))
+      .copy(broadcastKeyIndices = Seq(1))(None)
     assert(dynamicPruningSubquery.resolved == false)
   }
 
@@ -98,17 +102,88 @@ class DynamicPruningSubquerySuite extends SparkFunSuite {
       buildQuery = LocalRelation(attr1),
       buildKeys = Seq(attr1),
       broadcastKeyIndices = Seq(0),
-      onlyInBroadcast = false)
+      onlyInBroadcast = false)()
 
     val dpq2 = DynamicPruningSubquery(
       pruningKey = Literal(1),
       buildQuery = LocalRelation(attr2),
       buildKeys = Seq(attr2),
       broadcastKeyIndices = Seq(0),
-      onlyInBroadcast = false)
+      onlyInBroadcast = false)()
 
     assert(dpq1.canonicalized == dpq2.canonicalized,
       "DynamicPruningSubquery with identical build queries but different 
ExprIds " +
       "must produce identical canonicalized forms so PlanMerger can 
deduplicate them")
   }
+
+  test("transient broadcast value projection survives Catalyst copies without 
changing identity") {
+    val key = AttributeReference("key", IntegerType)()
+    val source = LocalRelation(key)
+    val projection = BroadcastValueProjection(source, Seq(key), key)
+    val pruning = DynamicPruningSubquery(
+      key, source, Seq(key), Seq(0), onlyInBroadcast = true)(Some(projection))
+
+    assert(pruning.resolved)
+    assert(pruning.usableBroadcastValueProjection.contains(projection))
+    assert(pruning.productArity === 7)
+    assert(DynamicPruningSubquery.unapply(pruning).exists(_.productArity == 7))
+    assert(Modifier.isTransient(
+      
classOf[DynamicPruningSubquery].getDeclaredField("broadcastValueProjection").getModifiers))
+
+    Seq(
+      pruning.copy()(pruning.broadcastValueProjection),
+      pruning.withNewPlan(source),
+      pruning.withNewOuterAttrs(Seq(key)),
+      pruning.withNewHint(None).asInstanceOf[DynamicPruningSubquery],
+      pruning.withNewChildren(Seq(key)).asInstanceOf[DynamicPruningSubquery],
+      
pruning.makeCopy(pruning.productIterator.map(_.asInstanceOf[AnyRef]).toArray)
+        .asInstanceOf[DynamicPruningSubquery]
+    ).foreach { rewritten =>
+      assert(rewritten.broadcastValueProjection.contains(projection))
+    }
+
+    val unprojected = pruning.copy()(None)
+    assert(pruning === unprojected)
+    assert(pruning.canonicalized.asInstanceOf[DynamicPruningSubquery]
+      .broadcastValueProjection.isEmpty)
+    assert(pruning.canonicalized === unprojected.canonicalized)
+
+    val missing = AttributeReference("missing", IntegerType)()
+    Seq(
+      projection.copy(sourceHashKeys = Seq(missing)),
+      projection.copy(valueExpression = missing),
+      projection.copy(sourceHashKeys = Seq.empty),
+      projection.copy(valueExpression = Literal(1L))
+    ).foreach { invalidProjection =>
+      val rewritten = pruning.copy()(Some(invalidProjection))
+      assert(rewritten.resolved)
+      assert(rewritten.broadcastValueProjection.contains(invalidProjection))
+      assert(rewritten.usableBroadcastValueProjection.isEmpty)
+    }
+  }
+
+  test("extract broadcast hash keys without rejecting residual join 
predicates") {
+    val leftKey = AttributeReference("left_key", IntegerType)()
+    val leftRegion = AttributeReference("left_region", IntegerType)()
+    val leftValue = AttributeReference("left_value", IntegerType)()
+    val rightKey = AttributeReference("right_key", IntegerType)()
+    val rightRegion = AttributeReference("right_region", IntegerType)()
+    val rightValue = AttributeReference("right_value", IntegerType)()
+    val left = LocalRelation(leftKey, leftRegion, leftValue)
+    val right = LocalRelation(rightKey, rightRegion, rightValue)
+    val excluded = LocalRelation(AttributeReference("excluded", IntegerType)())
+    val residual = LessThanOrEqual(leftValue, rightValue)
+    val condition = And(
+      And(EqualTo(leftKey, rightKey), EqualTo(rightRegion, leftRegion)), 
residual)
+    val join = Join(left, right, Inner, Some(condition), JoinHint.NONE)
+
+    assert(ReusableBroadcastValueProjection.find(leftValue, join, 
excluded).contains(
+      BroadcastValueProjection(left, Seq(leftKey, leftRegion), leftValue)))
+    assert(ReusableBroadcastValueProjection.find(rightValue, join, 
excluded).contains(
+      BroadcastValueProjection(right, Seq(rightKey, rightRegion), rightValue)))
+
+    val nullSafeJoin = Join(
+      left, right, Inner, Some(And(EqualNullSafe(leftKey, rightKey), 
residual)), JoinHint.NONE)
+    assert(ReusableBroadcastValueProjection.find(leftValue, nullSafeJoin, 
excluded).isEmpty)
+  }
 }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/BroadcastValueProjector.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/BroadcastValueProjector.scala
new file mode 100644
index 000000000000..bbff16a10f19
--- /dev/null
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/BroadcastValueProjector.scala
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import java.io.InterruptedIOException
+import java.nio.channels.ClosedByInterruptException
+import java.util.IdentityHashMap
+import java.util.concurrent.CancellationException
+
+import scala.collection.mutable
+import scala.util.control.{ControlThrowable, NonFatal}
+
+import org.apache.spark.TaskKilledException
+import org.apache.spark.broadcast.Broadcast
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{BindReferences, Expression, 
UnsafeProjection, UnsafeRow}
+import org.apache.spark.sql.catalyst.plans.logical.Statistics
+import org.apache.spark.sql.execution.adaptive.{AdaptiveSparkPlanExec, 
BroadcastQueryStageExec, ResultQueryStageExec}
+import org.apache.spark.sql.execution.exchange.{BroadcastExchangeLike, 
ReusedExchangeExec}
+import org.apache.spark.sql.execution.joins.{EmptyHashedRelation, 
HashedRelation, HashedRelationWithAllNullKeys, LongHashedRelation}
+
+private[sql] case class BroadcastValueProjectionLimits(
+    maxInputRows: Long,
+    maxOutputBytes: Long,
+    maxSourceBytes: Long)
+
+private[sql] sealed trait BroadcastValueResult[+T]
+
+private[sql] object BroadcastValueResult {
+  case class Available[T](value: T) extends BroadcastValueResult[T]
+  case object Unavailable extends BroadcastValueResult[Nothing]
+}
+
+/** Projects a complete, bounded value domain from rows already stored in a 
hash broadcast. */
+private[sql] object BroadcastValueProjector {
+  import BroadcastValueResult._
+
+  def collectExactValueDomain(
+      broadcast: Broadcast[HashedRelation],
+      child: SparkPlan,
+      valueExpression: Expression,
+      limits: BroadcastValueProjectionLimits,
+      onInputRow: () => Unit,
+      onError: () => Unit): BroadcastValueResult[Array[InternalRow]] = {
+    try {
+      if (sourceBroadcastCannotBeSafelyRehydrated(child, limits)) {
+        return Unavailable
+      }
+
+      val broadcastRelation = broadcast.value
+      if (broadcastRelation == HashedRelationWithAllNullKeys) {
+        return Unavailable
+      }
+      if (broadcastRelation == EmptyHashedRelation) {
+        return Available(Array.empty[InternalRow])
+      }
+
+      val relation = broadcastRelation.asReadOnlyCopy()
+      val projection = UnsafeProjection.create(
+        BindReferences.bindReference(valueExpression, child.output))
+      val projectedRows = mutable.LinkedHashSet.empty[UnsafeRow]
+      val valueRows = relation match {
+        case longRelation: LongHashedRelation =>
+          longRelation.keys().flatMap { key =>
+            val rows = longRelation.get(key)
+            if (rows == null) Iterator.empty else rows
+          }
+        case otherRelation =>
+          otherRelation.valuesWithKeyIndex().map(_.getValue)
+      }
+      var visitedRows = 0L
+      var projectedBytes = 0L
+
+      while (valueRows.hasNext) {
+        if (visitedRows >= limits.maxInputRows) {
+          return Unavailable
+        }
+        val valueRow = valueRows.next()
+        visitedRows += 1
+        onInputRow()
+        val projected = projection(valueRow)
+        if (!projected.isNullAt(0) && !projectedRows.contains(projected)) {
+          val rowSize = projected.getSizeInBytes.toLong
+          if (rowSize > limits.maxOutputBytes - projectedBytes) {
+            return Unavailable
+          }
+          projectedBytes += rowSize
+          projectedRows += projected.copy()
+        }
+      }
+
+      Available(projectedRows.toArray[InternalRow])
+    } catch {
+      case NonFatal(error) if !mustPropagateFailure(error) =>
+        onError()
+        Unavailable
+    }
+  }
+
+  private def sourceBroadcastRuntimeStatistics(plan: SparkPlan): 
Option[Statistics] = plan match {
+    case exchange: BroadcastExchangeLike => Some(exchange.runtimeStatistics)
+    case ReusedExchangeExec(_, exchange: BroadcastExchangeLike) =>
+      Some(exchange.runtimeStatistics)
+    case stage: BroadcastQueryStageExec => Some(stage.getRuntimeStatistics)
+    case stage: ResultQueryStageExec =>
+      sourceBroadcastRuntimeStatistics(stage.plan)
+    case adaptive: AdaptiveSparkPlanExec =>
+      sourceBroadcastRuntimeStatistics(adaptive.executedPlan)
+    case _ => None
+  }
+
+  private def sourceBroadcastCannotBeSafelyRehydrated(
+      child: SparkPlan,
+      limits: BroadcastValueProjectionLimits): Boolean = {
+    val maxRows = BigInt(limits.maxInputRows)
+    val maxSourceBytes = BigInt(limits.maxSourceBytes)
+    sourceBroadcastRuntimeStatistics(child) match {
+      case Some(statistics) =>
+        statistics.rowCount.forall(_ > maxRows) || statistics.sizeInBytes > 
maxSourceBytes
+      case None => true
+    }
+  }
+
+  private[sql] def mustPropagateFailure(error: Throwable): Boolean = {
+    if (Thread.currentThread().isInterrupted) {
+      return true
+    }
+
+    val seen = new IdentityHashMap[Throwable, java.lang.Boolean]()
+    var current = error
+    while (current != null && !seen.containsKey(current)) {
+      seen.put(current, java.lang.Boolean.TRUE)
+      current match {
+        case _: InterruptedException | _: InterruptedIOException |
+            _: ClosedByInterruptException | _: CancellationException | _: 
TaskKilledException |
+            _: VirtualMachineError | _: ThreadDeath | _: LinkageError | _: 
ControlThrowable =>
+          return true
+        case _ => current = current.getCause
+      }
+    }
+    false
+  }
+}
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ProjectedBroadcastValueSubqueryExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ProjectedBroadcastValueSubqueryExec.scala
new file mode 100644
index 000000000000..25132c38f9c2
--- /dev/null
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ProjectedBroadcastValueSubqueryExec.scala
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution
+
+import java.util.concurrent.{Future => JFuture}
+
+import scala.concurrent.duration.Duration
+
+import org.apache.spark.SparkException
+import org.apache.spark.rdd.RDD
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{Attribute, 
AttributeReference, Cast, Expression, NamedExpression, UnsafeRow}
+import org.apache.spark.sql.catalyst.plans.QueryPlan
+import org.apache.spark.sql.errors.QueryExecutionErrors
+import org.apache.spark.sql.execution.BroadcastValueResult.{Available, 
Unavailable}
+import org.apache.spark.sql.execution.joins.HashedRelation
+import org.apache.spark.sql.execution.metric.SQLMetrics
+import org.apache.spark.util.ThreadUtils
+
+/** Collects pruning values from the full rows of an already required hash 
broadcast. */
+case class ProjectedBroadcastValueSubqueryExec(
+    name: String,
+    valueExpression: Expression,
+    child: SparkPlan) extends BaseSubqueryExec with UnaryExecNode {
+
+  override def output: Seq[Attribute] = {
+    val outputName = valueExpression match {
+      case named: NamedExpression => named.name
+      case Cast(named: NamedExpression, _, _, _) => named.name
+      case _ => "key"
+    }
+    Seq(AttributeReference(outputName, valueExpression.dataType, 
valueExpression.nullable)())
+  }
+
+  override lazy val metrics = Map(
+    "numInputRows" -> SQLMetrics.createMetric(sparkContext, "number of input 
rows"),
+    "numOutputRows" -> SQLMetrics.createMetric(sparkContext, "number of output 
rows"),
+    "dataSize" -> SQLMetrics.createMetric(sparkContext, "data size (bytes)"),
+    "projectionDisabled" -> SQLMetrics.createMetric(sparkContext, "projection 
disabled"),
+    "projectionErrors" -> SQLMetrics.createMetric(sparkContext, "projection 
errors"),
+    "collectTime" -> SQLMetrics.createMetric(sparkContext, "time to collect 
(ms)"))
+
+  override def doCanonicalize(): SparkPlan = {
+    ProjectedBroadcastValueSubqueryExec(
+      "dpp",
+      QueryPlan.normalizeExpressions(valueExpression, child.output),
+      child.canonicalized)
+  }
+
+  @transient
+  private lazy val relationFuture: 
JFuture[BroadcastValueResult[Array[InternalRow]]] = {
+    val executionId = 
sparkContext.getLocalProperty(SQLExecution.EXECUTION_ID_KEY)
+    
SQLExecution.withThreadLocalCaptured[BroadcastValueResult[Array[InternalRow]]](
+      session, SubqueryBroadcastExec.executionContext) {
+      SQLExecution.withExecutionId(session, executionId) {
+        val beforeCollect = System.nanoTime()
+        val broadcast = child.executeBroadcast[HashedRelation]()
+        val limits = BroadcastValueProjectionLimits(
+          maxInputRows = 
conf.dynamicPartitionPruningBroadcastProjectionMaxRows.toLong,
+          maxOutputBytes = 
conf.dynamicPartitionPruningBroadcastProjectionMaxBytes,
+          maxSourceBytes = 
conf.dynamicPartitionPruningBroadcastProjectionMaxSourceBytes)
+        val result = BroadcastValueProjector.collectExactValueDomain(
+          broadcast,
+          child,
+          valueExpression,
+          limits,
+          onInputRow = () => longMetric("numInputRows") += 1,
+          onError = () => longMetric("projectionErrors") += 1)
+
+        longMetric("collectTime") += (System.nanoTime() - beforeCollect) / 
1000000
+        result match {
+          case Available(rows) =>
+            longMetric("numOutputRows") += rows.length
+            longMetric("dataSize") +=
+              
rows.iterator.map(_.asInstanceOf[UnsafeRow].getSizeInBytes.toLong).sum
+          case Unavailable =>
+            longMetric("projectionDisabled") += 1
+        }
+        SQLMetrics.postDriverMetricUpdates(sparkContext, executionId, 
metrics.values.toSeq)
+        result
+      }
+    }
+  }
+
+  override protected def doPrepare(): Unit = {
+    relationFuture
+  }
+
+  override protected def doExecute(): RDD[InternalRow] = {
+    throw QueryExecutionErrors.executeCodePathUnsupportedError(
+      "ProjectedBroadcastValueSubqueryExec")
+  }
+
+  override def executeCollect(): Array[InternalRow] = executeCollectResult() 
match {
+    case Available(rows) => rows
+    case Unavailable =>
+      throw SparkException.internalError(
+        "An unavailable projected broadcast value domain must be consumed 
through " +
+          "executeCollectResult.")
+  }
+
+  private[execution] def executeCollectResult(): 
BroadcastValueResult[Array[InternalRow]] =
+    ThreadUtils.awaitResult(relationFuture, Duration.Inf)
+
+  override def stringArgs: Iterator[Any] = super.stringArgs ++ 
Iterator(s"[id=#$id]")
+
+  override protected def withNewChildInternal(
+      newChild: SparkPlan): ProjectedBroadcastValueSubqueryExec = copy(child = 
newChild)
+}
+
+object ProjectedBroadcastValueSubqueryExec {
+  private[execution] def resultOf(
+      plan: BaseSubqueryExec): 
Option[BroadcastValueResult[Array[InternalRow]]] = {
+    plan match {
+      case projected: ProjectedBroadcastValueSubqueryExec =>
+        Some(projected.executeCollectResult())
+      case ReusedSubqueryExec(child) => resultOf(child)
+      case _ => None
+    }
+  }
+}
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala
index 555f4f41d3cd..c435f06ffddc 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala
@@ -38,7 +38,11 @@ case class SubqueryAdaptiveBroadcastExec(
     onlyInBroadcast: Boolean,
     @transient buildPlan: LogicalPlan,
     buildKeys: Seq[Expression],
-    child: SparkPlan) extends BaseSubqueryExec with UnaryExecNode {
+    child: SparkPlan)(
+    @transient private[sql] val broadcastValueProjection: 
Option[BroadcastValueProjection] = None)
+  extends BaseSubqueryExec with UnaryExecNode {
+
+  override protected def otherCopyArgs: Seq[AnyRef] = 
Seq(broadcastValueProjection)
 
   protected override def doExecute(): RDD[InternalRow] = {
     throw 
QueryExecutionErrors.executeCodePathUnsupportedError("SubqueryAdaptiveBroadcastExec")
@@ -46,9 +50,9 @@ case class SubqueryAdaptiveBroadcastExec(
 
   protected override def doCanonicalize(): SparkPlan = {
     val keys = buildKeys.map(k => QueryPlan.normalizeExpressions(k, 
child.output))
-    copy(name = "dpp", buildKeys = keys, child = child.canonicalized)
+    copy(name = "dpp", buildKeys = keys, child = child.canonicalized)(None)
   }
 
   override protected def withNewChildInternal(newChild: SparkPlan): 
SubqueryAdaptiveBroadcastExec =
-    copy(child = newChild)
+    copy(child = newChild)(broadcastValueProjection)
 }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
index 6b721d5442e2..9374758d4731 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveDynamicPruningFilters.scala
@@ -17,7 +17,7 @@
 
 package org.apache.spark.sql.execution.adaptive
 
-import org.apache.spark.sql.catalyst.expressions.{Alias, BindReferences, 
DynamicPruningExpression, Literal}
+import org.apache.spark.sql.catalyst.expressions.{Alias, BindReferences, 
BroadcastValueProjection, DynamicPruningExpression, Expression, Literal}
 import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
 import org.apache.spark.sql.catalyst.plans.logical.Aggregate
 import org.apache.spark.sql.catalyst.rules.Rule
@@ -35,6 +35,61 @@ case class PlanAdaptiveDynamicPruningFilters(
 
   override def conf: SQLConf = rootPlan.context.session.sessionState.conf
 
+  private def reusableBroadcast(
+      name: String,
+      indices: Seq[Int],
+      buildKeys: Seq[Expression],
+      valueExpression: Option[Expression],
+      adaptivePlan: AdaptiveSparkPlanExec): Option[BaseSubqueryExec] = {
+    if (!conf.exchangeReuseEnabled || buildKeys.isEmpty) {
+      return None
+    }
+
+    val packedKeys = BindReferences.bindReferences(
+      HashJoin.rewriteKeyExpr(buildKeys), adaptivePlan.executedPlan.output)
+    val exchange = BroadcastExchangeExec(
+      HashedRelationBroadcastMode(packedKeys, isNullAware = false),
+      adaptivePlan.executedPlan)
+    val canReuseExchange = find(rootPlan) {
+      case join: BroadcastHashJoinExec if !join.isNullAwareAntiJoin =>
+        val candidatePlan = join.buildSide match {
+          case BuildLeft => join.left
+          case BuildRight => join.right
+        }
+        candidatePlan.sameResult(exchange)
+      case _ => false
+    }.isDefined
+
+    if (canReuseExchange) {
+      adaptivePlan.executedPlan.logicalLink.foreach(exchange.setLogicalLink)
+      val newAdaptivePlan = adaptivePlan.copy(inputPlan = exchange)
+      Some(valueExpression match {
+        case Some(value) =>
+          ProjectedBroadcastValueSubqueryExec(name, value, newAdaptivePlan)
+        case None =>
+          SubqueryBroadcastExec(name, indices, buildKeys, newAdaptivePlan)
+      })
+    } else {
+      None
+    }
+  }
+
+  private def projectedBroadcast(
+      name: String,
+      projection: BroadcastValueProjection,
+      context: AdaptiveExecutionContext): Option[BaseSubqueryExec] = {
+    QueryExecution.prepareExecutedPlan(projection.sourcePlan, context) match {
+      case adaptive: AdaptiveSparkPlanExec =>
+        reusableBroadcast(
+          name,
+          Seq(0),
+          projection.sourceHashKeys,
+          Some(projection.valueExpression),
+          adaptive)
+      case _ => None
+    }
+  }
+
   def apply(plan: SparkPlan): SparkPlan = {
     if (!conf.dynamicPartitionPruningEnabled) {
       return plan
@@ -43,42 +98,34 @@ case class PlanAdaptiveDynamicPruningFilters(
     plan.transformAllExpressionsWithPruning(
       _.containsAllPatterns(DYNAMIC_PRUNING_EXPRESSION, IN_SUBQUERY_EXEC)) {
       case DynamicPruningExpression(InSubqueryExec(
-          value, SubqueryAdaptiveBroadcastExec(name, indices, onlyInBroadcast, 
buildPlan, buildKeys,
-          adaptivePlan: AdaptiveSparkPlanExec), exprId, _, _, _)) =>
-        val packedKeys = BindReferences.bindReferences(
-          HashJoin.rewriteKeyExpr(buildKeys), adaptivePlan.executedPlan.output)
-        val mode = HashedRelationBroadcastMode(packedKeys)
-        // plan a broadcast exchange of the build side of the join
-        val exchange = BroadcastExchangeExec(mode, adaptivePlan.executedPlan)
-
-        val canReuseExchange = conf.exchangeReuseEnabled && buildKeys.nonEmpty 
&&
-          find(rootPlan) {
-            case BroadcastHashJoinExec(_, _, _, BuildLeft, _, left, _, _, _) =>
-              left.sameResult(exchange)
-            case BroadcastHashJoinExec(_, _, _, BuildRight, _, _, right, _, _) 
=>
-              right.sameResult(exchange)
-            case _ => false
-          }.isDefined
-
-        if (canReuseExchange) {
-          exchange.setLogicalLink(adaptivePlan.executedPlan.logicalLink.get)
-          val newAdaptivePlan = adaptivePlan.copy(inputPlan = exchange)
-
-          val broadcastValues = SubqueryBroadcastExec(
-            name, indices, buildKeys, newAdaptivePlan)
-          DynamicPruningExpression(InSubqueryExec(value, broadcastValues, 
exprId))
-        } else if (onlyInBroadcast) {
-          DynamicPruningExpression(Literal.TrueLiteral)
-        } else {
-          // we need to apply an aggregate on the buildPlan in order to be 
column pruned
-          val aliases = indices.map(idx => Alias(buildKeys(idx), 
buildKeys(idx).toString)())
-          val aggregate = Aggregate(aliases, aliases, buildPlan)
+          value, subquery @ SubqueryAdaptiveBroadcastExec(
+            name, indices, onlyInBroadcast, buildPlan, buildKeys,
+            adaptivePlan: AdaptiveSparkPlanExec), exprId, _, _, _)) =>
+        val directBroadcast = reusableBroadcast(
+          name, indices, buildKeys, None, adaptivePlan)
+        val reusedBroadcast = directBroadcast.orElse {
+          if (onlyInBroadcast) {
+            subquery.broadcastValueProjection
+              .flatMap(projectedBroadcast(name, _, adaptivePlan.context))
+          } else {
+            None
+          }
+        }
 
-          val sparkPlan = QueryExecution.prepareExecutedPlan(aggregate, 
adaptivePlan.context)
-          assert(sparkPlan.isInstanceOf[AdaptiveSparkPlanExec])
-          val newAdaptivePlan = sparkPlan.asInstanceOf[AdaptiveSparkPlanExec]
-          val values = SubqueryExec(name, newAdaptivePlan)
-          DynamicPruningExpression(InSubqueryExec(value, values, exprId))
+        reusedBroadcast match {
+          case Some(broadcastValues) =>
+            DynamicPruningExpression(InSubqueryExec(value, broadcastValues, 
exprId))
+          case None if onlyInBroadcast =>
+            DynamicPruningExpression(Literal.TrueLiteral)
+          case None =>
+            val aliases = indices.map(idx =>
+              Alias(buildKeys(idx), buildKeys(idx).toString)())
+            val aggregate = Aggregate(aliases, aliases, buildPlan)
+            val sparkPlan = QueryExecution.prepareExecutedPlan(aggregate, 
adaptivePlan.context)
+            assert(sparkPlan.isInstanceOf[AdaptiveSparkPlanExec])
+            val newAdaptivePlan = sparkPlan.asInstanceOf[AdaptiveSparkPlanExec]
+            val values = SubqueryExec(name, newAdaptivePlan)
+            DynamicPruningExpression(InSubqueryExec(value, values, exprId))
         }
     }
   }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveSubqueries.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveSubqueries.scala
index 5f2638655c37..f97339cde51f 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveSubqueries.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/PlanAdaptiveSubqueries.scala
@@ -46,11 +46,16 @@ case class PlanAdaptiveSubqueries(
         }
         val subquery = SubqueryExec(s"subquery#${exprId.id}", 
subqueryMap(exprId.id))
         InSubqueryExec(expr, subquery, exprId, isDynamicPruning = false)
-      case expressions.DynamicPruningSubquery(value, buildPlan,
+      case pruning @ expressions.DynamicPruningSubquery(value, buildPlan,
           buildKeys, broadcastKeyIndices, onlyInBroadcast, exprId, _) =>
         val name = s"dynamicpruning#${exprId.id}"
-        val subquery = SubqueryAdaptiveBroadcastExec(name, 
broadcastKeyIndices, onlyInBroadcast,
-          buildPlan, buildKeys, subqueryMap(exprId.id))
+        val subquery = SubqueryAdaptiveBroadcastExec(
+          name,
+          broadcastKeyIndices,
+          onlyInBroadcast,
+          buildPlan,
+          buildKeys,
+          subqueryMap(exprId.id))(pruning.usableBroadcastValueProjection)
         DynamicPruningExpression(InSubqueryExec(value, subquery, exprId))
     }
   }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
index bcdbf5fe6ee2..72807a5242c7 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
@@ -955,6 +955,9 @@ private[sql] object DataSourceV2Strategy extends Logging {
    * If the underlying subquery hasn't completed yet, this method will throw 
an exception.
    */
   protected[sql] def translateRuntimeFilterV2(expr: Expression): 
Option[Predicate] = expr match {
+    case TrueLiteral => None
+    case in: InSubqueryExec if in.isResultUnavailable =>
+      None
     case in @ InSubqueryExec(PushableColumnAndNestedColumn(name), _, _, _, _, 
_) =>
       val values = in.values().getOrElse {
         throw SparkException.internalError(
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala
index b74a785aafc3..46c4e9859582 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/PushDownUtils.scala
@@ -32,7 +32,7 @@ import org.apache.spark.sql.connector.catalog.Table
 import org.apache.spark.sql.connector.expressions.{IdentityTransform, 
SortOrder, Transform}
 import org.apache.spark.sql.connector.expressions.filter.Predicate
 import org.apache.spark.sql.connector.read.{HasPartitionKey, InputPartition, 
SampleMethod => SampleMethodV2, Scan, ScanBuilder, SupportsPushDownFilters, 
SupportsPushDownLimit, SupportsPushDownOffset, SupportsPushDownRequiredColumns, 
SupportsPushDownTableSample, SupportsPushDownTopN, SupportsPushDownV2Filters, 
SupportsRuntimeV2Filtering}
-import org.apache.spark.sql.execution.{ScalarSubquery => ExecScalarSubquery}
+import org.apache.spark.sql.execution.{InSubqueryExec, ScalarSubquery => 
ExecScalarSubquery}
 import org.apache.spark.sql.execution.datasources.{DataSourceStrategy, 
DataSourceUtils}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.internal.connector.{PartitionPredicateField, 
PartitionPredicateImpl, SupportsPushDownCatalystFilters}
@@ -434,6 +434,8 @@ object PushDownUtils extends Logging {
       runtimeFilters: Seq[Expression],
       partitionFields: Seq[PartitionPredicateField]): 
Seq[PartitionPredicateImpl] = {
     val catalystExprs = runtimeFilters.flatMap {
+      case DynamicPruningExpression(in: InSubqueryExec) if 
in.isResultUnavailable =>
+        None
       case DynamicPruningExpression(e) => Some(e)
       case _: DynamicPruning => None
       case f => Some(f.transform { case s: ExecScalarSubquery => s.toLiteral })
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala
index 3e1b39b564c7..0e885b994ede 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PartitionPruning.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution.dynamicpruning
 
 import org.apache.spark.sql.catalyst.catalog.HiveTableRelation
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.optimizer.JoinSelectionHelper
+import org.apache.spark.sql.catalyst.optimizer.{JoinSelectionHelper, 
ReusableBroadcastValueProjection}
 import org.apache.spark.sql.catalyst.planning.ExtractEquiJoinKeys
 import org.apache.spark.sql.catalyst.plans.logical._
 import org.apache.spark.sql.catalyst.rules.Rule
@@ -110,6 +110,11 @@ object PartitionPruning extends Rule[LogicalPlan] with 
PredicateHelper with Join
     require(filteringKeys.size == 1, "DPP Filters should only have a single 
broadcasting key " +
       "since there are no usage for multiple broadcasting keys at the moment.")
     val indices = Seq(joinKeys.indexOf(filteringKeys.head))
+    val broadcastValueProjection = if 
(conf.dynamicPartitionPruningBroadcastProjectionEnabled) {
+      ReusableBroadcastValueProjection.find(filteringKeys.head, filteringPlan, 
partScan)
+    } else {
+      None
+    }
     lazy val hasBenefit = pruningHasBenefit(
       pruningKey, partScan, filteringKeys.head, filteringPlan, 
hasSelectivePredicate(filteringPlan))
     if (reuseEnabled || hasBenefit) {
@@ -120,7 +125,7 @@ object PartitionPruning extends Rule[LogicalPlan] with 
PredicateHelper with Join
           filteringPlan,
           joinKeys,
           indices,
-          conf.dynamicPartitionPruningReuseBroadcastOnly || !hasBenefit),
+          conf.dynamicPartitionPruningReuseBroadcastOnly || 
!hasBenefit)(broadcastValueProjection),
         pruningPlan)
     } else {
       // abort dynamic partition pruning
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
index fdcb78fdb55a..ca7407eb2a88 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/dynamicpruning/PlanDynamicPruningFilters.scala
@@ -17,14 +17,14 @@
 
 package org.apache.spark.sql.execution.dynamicpruning
 
-import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeSeq, 
BindReferences, DynamicPruningExpression, DynamicPruningSubquery, Expression, 
Literal}
+import org.apache.spark.sql.catalyst.expressions.{Alias, AttributeSeq, 
BindReferences, BroadcastValueProjection, DynamicPruningExpression, 
DynamicPruningSubquery, Expression, Literal}
 import org.apache.spark.sql.catalyst.optimizer.{BuildLeft, BuildRight}
-import org.apache.spark.sql.catalyst.plans.logical.Aggregate
+import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, LogicalPlan}
 import org.apache.spark.sql.catalyst.plans.physical.BroadcastMode
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.TreePattern.DYNAMIC_PRUNING_SUBQUERY
 import org.apache.spark.sql.classic.SparkSession
-import org.apache.spark.sql.execution.{InSubqueryExec, QueryExecution, 
SparkPlan, SubqueryBroadcastExec, SubqueryExec}
+import org.apache.spark.sql.execution.{BaseSubqueryExec, InSubqueryExec, 
ProjectedBroadcastValueSubqueryExec, QueryExecution, SparkPlan, 
SubqueryBroadcastExec, SubqueryExec}
 import org.apache.spark.sql.execution.exchange.BroadcastExchangeExec
 import org.apache.spark.sql.execution.joins._
 import org.apache.spark.sql.internal.SQLConf
@@ -46,47 +46,89 @@ case class PlanDynamicPruningFilters(sparkSession: 
SparkSession) extends Rule[Sp
     HashedRelationBroadcastMode(packedKeys)
   }
 
+  private def reusableBroadcast(
+      plan: SparkPlan,
+      name: String,
+      indices: Seq[Int],
+      buildPlan: LogicalPlan,
+      buildKeys: Seq[Expression],
+      projection: Option[BroadcastValueProjection]): Option[BaseSubqueryExec] 
= {
+    if (!conf.exchangeReuseEnabled || buildKeys.isEmpty) {
+      return None
+    }
+
+    val sparkPlan = QueryExecution.createSparkPlan(
+      sparkSession.sessionState.planner, buildPlan)
+    val requiredMode = broadcastMode(buildKeys, sparkPlan.output)
+    val canReuseExchange = plan.exists {
+      case join: BroadcastHashJoinExec =>
+        val (candidateKeys, candidatePlan) = join.buildSide match {
+          case BuildLeft => (join.leftKeys, join.left)
+          case BuildRight => (join.rightKeys, join.right)
+        }
+        // Preserve existing direct reuse; only a projected domain requires 
the exact hash mode.
+        candidatePlan.sameResult(sparkPlan) &&
+          (projection.isEmpty ||
+            (!join.isNullAwareAntiJoin &&
+              broadcastMode(candidateKeys, candidatePlan.output) == 
requiredMode))
+      case _ => false
+    }
+
+    if (canReuseExchange) {
+      val executedPlan = QueryExecution.prepareExecutedPlan(sparkSession, 
sparkPlan)
+      val exchange = BroadcastExchangeExec(
+        broadcastMode(buildKeys, executedPlan.output), executedPlan)
+      Some(projection match {
+        case Some(valueProjection) =>
+          ProjectedBroadcastValueSubqueryExec(
+            name, valueProjection.valueExpression, exchange)
+        case None =>
+          SubqueryBroadcastExec(name, indices, buildKeys, exchange)
+      })
+    } else {
+      None
+    }
+  }
+
   override def apply(plan: SparkPlan): SparkPlan = {
     if (!conf.dynamicPartitionPruningEnabled) {
       return plan
     }
 
     
plan.transformAllExpressionsWithPruning(_.containsPattern(DYNAMIC_PRUNING_SUBQUERY))
 {
-      case DynamicPruningSubquery(
+      case pruning @ DynamicPruningSubquery(
           value, buildPlan, buildKeys, broadcastKeyIndices, onlyInBroadcast, 
exprId, _) =>
-        val sparkPlan = 
QueryExecution.createSparkPlan(sparkSession.sessionState.planner, buildPlan)
         val name = s"dynamicpruning#${exprId.id}"
-        // Using `sparkPlan` is a little hacky as it is based on the 
assumption that this rule is
-        // the first to be applied (apart from `InsertAdaptiveSparkPlan`).
-        val canReuseExchange = conf.exchangeReuseEnabled && buildKeys.nonEmpty 
&&
-          plan.exists {
-            case BroadcastHashJoinExec(_, _, _, BuildLeft, _, left, _, _, _) =>
-              left.sameResult(sparkPlan)
-            case BroadcastHashJoinExec(_, _, _, BuildRight, _, _, right, _, _) 
=>
-              right.sameResult(sparkPlan)
-            case _ => false
+        val directBroadcast = reusableBroadcast(
+          plan, name, broadcastKeyIndices, buildPlan, buildKeys, None)
+        val reusedBroadcast = directBroadcast.orElse {
+          if (onlyInBroadcast) {
+            pruning.usableBroadcastValueProjection.flatMap { projection =>
+              reusableBroadcast(
+                plan,
+                name,
+                Seq(0),
+                projection.sourcePlan,
+                projection.sourceHashKeys,
+                Some(projection))
+            }
+          } else {
+            None
           }
+        }
 
-        if (canReuseExchange) {
-          val executedPlan = QueryExecution.prepareExecutedPlan(sparkSession, 
sparkPlan)
-          val mode = broadcastMode(buildKeys, executedPlan.output)
-          // plan a broadcast exchange of the build side of the join
-          val exchange = BroadcastExchangeExec(mode, executedPlan)
-          // place the broadcast adaptor for reusing the broadcast results on 
the probe side
-          val broadcastValues =
-            SubqueryBroadcastExec(name, broadcastKeyIndices, buildKeys, 
exchange)
-          DynamicPruningExpression(InSubqueryExec(value, broadcastValues, 
exprId))
-        } else if (onlyInBroadcast) {
-          // it is not worthwhile to execute the query, so we fall-back to a 
true literal
-          DynamicPruningExpression(Literal.TrueLiteral)
-        } else {
-          // we need to apply an aggregate on the buildPlan in order to be 
column pruned
-          val aliases = broadcastKeyIndices.map(idx =>
-            Alias(buildKeys(idx), buildKeys(idx).toString)())
-          val aggregate = Aggregate(aliases, aliases, buildPlan)
-          val sparkPlan = QueryExecution.prepareExecutedPlan(sparkSession, 
aggregate)
-          val values = SubqueryExec(name, sparkPlan)
-          DynamicPruningExpression(InSubqueryExec(value, values, exprId))
+        reusedBroadcast match {
+          case Some(broadcastValues) =>
+            DynamicPruningExpression(InSubqueryExec(value, broadcastValues, 
exprId))
+          case None if onlyInBroadcast =>
+            DynamicPruningExpression(Literal.TrueLiteral)
+          case None =>
+            val aliases = broadcastKeyIndices.map(idx =>
+              Alias(buildKeys(idx), buildKeys(idx).toString)())
+            val aggregate = Aggregate(aliases, aliases, buildPlan)
+            val sparkPlan = QueryExecution.prepareExecutedPlan(sparkSession, 
aggregate)
+            val values = SubqueryExec(name, sparkPlan)
+            DynamicPruningExpression(InSubqueryExec(value, values, exprId))
         }
     }
   }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala
index 245e5204104c..c2b11c977676 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/metric/SQLLastAttemptAccumulator.scala
@@ -23,7 +23,7 @@ import scala.util.control.NonFatal
 import org.apache.spark.SparkContext
 import org.apache.spark.internal.{LogEntry, Logging}
 import org.apache.spark.sql.Dataset
-import org.apache.spark.sql.execution.{BaseSubqueryExec, QueryExecution, 
ReusedSubqueryExec, SparkPlan, SubqueryAdaptiveBroadcastExec, 
SubqueryBroadcastExec, SubqueryExec, WholeStageCodegenExec}
+import org.apache.spark.sql.execution.{BaseSubqueryExec, 
ProjectedBroadcastValueSubqueryExec, QueryExecution, ReusedSubqueryExec, 
SparkPlan, SubqueryAdaptiveBroadcastExec, SubqueryBroadcastExec, SubqueryExec, 
WholeStageCodegenExec}
 import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
 import org.apache.spark.sql.execution.exchange.{BroadcastExchangeExec, 
BroadcastExchangeLike, ReusedExchangeExec, ShuffleExchangeExec, 
ShuffleExchangeLike}
 import org.apache.spark.util.{AccumulatorV2, LastAttemptAccumulator}
@@ -368,7 +368,7 @@ object SQLLastAttemptAccumulator extends Logging {
           // ```
           // will launch stages in scope of child.
           scopeIds(s.child)
-        case _: SubqueryBroadcastExec =>
+        case _: SubqueryBroadcastExec | _: ProjectedBroadcastValueSubqueryExec 
=>
           // Used by DPP filter only, not part of main flow of query execution.
           Nil
         case _: SubqueryAdaptiveBroadcastExec =>
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala
index d5f258a8084b..25b87f7aab6f 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/subquery.scala
@@ -127,23 +127,36 @@ case class InSubqueryExec(
 
   override def nullable: Boolean = child.nullable
   override def toString: String = s"$child IN ${plan.name}"
-  override def withNewPlan(plan: BaseSubqueryExec): InSubqueryExec = copy(plan 
= plan)
+  override def withNewPlan(plan: BaseSubqueryExec): InSubqueryExec =
+    copy(plan = plan, result = null)
   final override def nodePatternsInternal(): Seq[TreePattern] = 
Seq(IN_SUBQUERY_EXEC)
 
   def updateResult(): Unit = {
-    val rows = plan.executeCollect()
-    result = if (plan.output.length > 1) {
+    val (rows, unavailable) = 
ProjectedBroadcastValueSubqueryExec.resultOf(plan) match {
+      case Some(BroadcastValueResult.Available(values)) => (values, false)
+      case Some(BroadcastValueResult.Unavailable) => 
(Array.empty[InternalRow], true)
+      case None => (plan.executeCollect(), false)
+    }
+    result = if (unavailable) {
+      assert(isDynamicPruning,
+        "An unavailable projected broadcast value domain is only supported for 
" +
+          "dynamic partition pruning.")
+      InSubqueryExecResultState.unavailableResult
+    } else if (plan.output.length > 1) {
       rows.asInstanceOf[Array[Any]]
     } else {
       rows.map(_.get(0, child.dataType))
     }
-    if (!isDynamicPruning) {
+    if (!isDynamicPruning && !isResultUnavailable) {
       resultBroadcast = plan.session.sparkContext.broadcast(result)
     }
   }
 
   // This is used only by DPP where we don't need broadcast the result.
-  def values(): Option[Array[Any]] = Option(result)
+  def values(): Option[Array[Any]] = if (isResultUnavailable) None else 
Option(result)
+
+  private[sql] def isResultUnavailable: Boolean =
+    InSubqueryExecResultState.isUnavailable(result)
 
   private def prepareResult(): Unit = {
     require(result != null || resultBroadcast != null, s"$this has not 
finished")
@@ -154,12 +167,12 @@ case class InSubqueryExec(
 
   override def eval(input: InternalRow): Any = {
     prepareResult()
-    inSet.eval(input)
+    if (isResultUnavailable) true else inSet.eval(input)
   }
 
   override def doGenCode(ctx: CodegenContext, ev: ExprCode): ExprCode = {
     prepareResult()
-    inSet.doGenCode(ctx, ev)
+    if (isResultUnavailable) Literal.TrueLiteral.doGenCode(ctx, ev) else 
inSet.doGenCode(ctx, ev)
   }
 
   override lazy val canonicalized: InSubqueryExec = {
@@ -175,6 +188,17 @@ case class InSubqueryExec(
     copy(child = newChild)
 }
 
+private[execution] object InSubqueryExecResultState {
+  private case object UnavailableMarker
+
+  def unavailableResult: Array[Any] = Array(UnavailableMarker)
+
+  def isUnavailable(result: Array[Any]): Boolean = {
+    result != null && result.length == 1 &&
+      (result(0).asInstanceOf[AnyRef] eq UnavailableMarker)
+  }
+}
+
 /**
  * Plans subqueries that are present in the given [[SparkPlan]].
  */
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
index 4aa4a8870b77..77754ef5952b 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/DynamicPartitionPruningSuite.scala
@@ -17,22 +17,30 @@
 
 package org.apache.spark.sql
 
+import java.lang.reflect.Modifier
+import java.sql.Date
 import java.util.concurrent.atomic.AtomicInteger
 
 import scala.collection.concurrent.TrieMap
+import scala.collection.mutable.ArrayBuffer
 
 import org.scalatest.GivenWhenThen
 
-import org.apache.spark.sql.catalyst.expressions.{DynamicPruningExpression, 
Expression}
+import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.expressions.{BroadcastValueProjection,
+  DynamicPruningExpression, DynamicPruningSubquery, Expression}
 import org.apache.spark.sql.catalyst.expressions.CodegenObjectFactoryMode._
-import org.apache.spark.sql.catalyst.plans.ExistenceJoin
+import org.apache.spark.sql.catalyst.optimizer.BuildRight
+import org.apache.spark.sql.catalyst.plans.{ExistenceJoin, Inner, LeftAnti}
 import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
 import org.apache.spark.sql.connector.catalog.{InMemoryTableCatalog, 
InMemoryTableWithV2FilterCatalog}
 import org.apache.spark.sql.execution._
 import org.apache.spark.sql.execution.adaptive._
 import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
+import org.apache.spark.sql.execution.dynamicpruning.PlanDynamicPruningFilters
 import org.apache.spark.sql.execution.exchange.{BroadcastExchangeLike, 
ReusedExchangeExec}
 import org.apache.spark.sql.execution.joins.BroadcastHashJoinExec
+import org.apache.spark.sql.execution.metric.SQLLastAttemptMetrics
 import org.apache.spark.sql.execution.streaming.runtime.{MemoryStream, 
StreamingQueryWrapper}
 import org.apache.spark.sql.functions._
 import org.apache.spark.sql.internal.SQLConf
@@ -195,8 +203,9 @@ abstract class DynamicPartitionPruningSuiteBase
       case InSubqueryExec(_, _: SubqueryExec, _, _, _, _) => true
       case _ => false
     }
-    val subqueryBroadcast = dpExprs.collect {
+    val subqueryBroadcast: Seq[BaseSubqueryExec with UnaryExecNode] = 
dpExprs.collect {
       case InSubqueryExec(_, b: SubqueryBroadcastExec, _, _, _, _) => b
+      case InSubqueryExec(_, b: ProjectedBroadcastValueSubqueryExec, _, _, _, 
_) => b
     }
 
     val hasFilter = if (withSubquery) "Should" else "Shouldn't"
@@ -255,6 +264,348 @@ abstract class DynamicPartitionPruningSuiteBase
     assert(buf.distinct.size == n)
   }
 
+  test("reuse ancestor broadcast value rows for composite-key date partition 
pruning") {
+    withTable("dpp_projection_dates", "dpp_projection_users", 
"dpp_projection_target") {
+      Seq(
+        (Date.valueOf("2024-01-01"), "2024-01-01", "us"),
+        (Date.valueOf("2024-01-01"), "2024-01-01", "us"),
+        (Date.valueOf("2024-01-08"), "2024-01-08", "eu"),
+        (Date.valueOf("2024-01-15"), null, "us"),
+        (null.asInstanceOf[Date], "2024-01-22", "apac"),
+        (Date.valueOf("2024-01-29"), "2024-01-29", "latam"))
+        .toDF("cohort_date", "cohort_ds", "region")
+        .write.format(tableFormat).saveAsTable("dpp_projection_dates")
+
+      Seq(
+        ("2024-01-01", "us", "u1", true),
+        ("2024-01-08", "eu", "u2", true),
+        (null, "us", "null-key", true),
+        ("2024-01-22", "apac", "null-value", true),
+        ("2024-01-08", "us", "inactive", false))
+        .toDF("cohort_ds", "region", "user_id", "active")
+        .write.format(tableFormat).saveAsTable("dpp_projection_users")
+
+      Seq(
+        ("u1", "2024-01-07", "p1"),
+        ("u2", "2024-01-14", "p2"),
+        ("ghost", "2024-02-04", "extra-domain"),
+        ("u1", "2024-02-01", "unselected"))
+        .toDF("user_id", "ds", "payload")
+        
.write.partitionBy("ds").format(tableFormat).saveAsTable("dpp_projection_target")
+
+      val query =
+        """
+          |WITH cohorts AS (
+          |  SELECT /*+ BROADCAST(d) */ d.cohort_date, u.user_id
+          |  FROM dpp_projection_dates d
+          |  JOIN dpp_projection_users u
+          |    ON d.cohort_ds = u.cohort_ds AND d.region = u.region
+          |  WHERE u.active AND u.user_id <> 'filtered'
+          |)
+          |SELECT c.user_id, w.payload
+          |FROM cohorts c LEFT JOIN dpp_projection_target w
+          |  ON w.user_id = c.user_id
+          | AND w.ds = date_format(date_add(c.cohort_date, 6), 'yyyy-MM-dd')
+          |ORDER BY c.user_id, w.payload
+          |""".stripMargin
+      val expected = Seq(
+        Row("null-value", null),
+        Row("u1", "p1"),
+        Row("u1", "p1"),
+        Row("u2", "p2"))
+
+      withSQLConf(
+          SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED.key 
-> "false",
+          SQLConf.EXCHANGE_REUSE_ENABLED.key -> "true",
+          SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+        val df = sql(query)
+        checkAnswer(df, expected)
+        val projected = collectWithSubqueries(df.queryExecution.executedPlan) {
+          case subquery: ProjectedBroadcastValueSubqueryExec => subquery
+        }
+        assert(projected.isEmpty, df.queryExecution.executedPlan)
+      }
+
+      Seq("true", "false").foreach { ansiEnabled =>
+        withSQLConf(
+            SQLConf.ANSI_ENABLED.key -> ansiEnabled,
+            SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+            SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> 
"true",
+            SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED.key 
-> "true",
+            SQLConf.EXCHANGE_REUSE_ENABLED.key -> "true",
+            SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+          val df = sql(query)
+          val candidates = ArrayBuffer.empty[DynamicPruningSubquery]
+          
df.queryExecution.optimizedPlan.transformAllExpressionsWithSubqueries {
+            case candidate: DynamicPruningSubquery =>
+              candidates += candidate
+              candidate
+          }
+          val projections = candidates.flatMap(_.broadcastValueProjection)
+          assert(projections.size === 1, df.queryExecution.optimizedPlan)
+          assert(projections.head.sourceHashKeys.map(_.references.head.name) 
===
+            Seq("cohort_ds", "region"))
+
+          checkAnswer(df, expected)
+
+          val projected = 
collectWithSubqueries(df.queryExecution.executedPlan) {
+            case subquery: ProjectedBroadcastValueSubqueryExec => subquery
+          }
+          assert(projected.size === 1, df.queryExecution.executedPlan)
+          val lastAttemptMetric = SQLLastAttemptMetrics.createMetric(
+            spark.sparkContext, "projected broadcast pruning")
+          assert(lastAttemptMetric.lastAttemptValueForDataset(df) === Some(0L))
+          val pruning = 
collectDynamicPruningExpressions(df.queryExecution.executedPlan)
+            .collectFirst { case in: InSubqueryExec => in }.get
+          assert(pruning.values().get.map(String.valueOf).toSet ===
+            Set("2024-01-07", "2024-01-14", "2024-02-04"))
+          val projectedMetrics = projected.head.metrics.map {
+            case (name, metric) => name -> metric.value
+          }
+          assert(projected.head.metrics("numInputRows").value === 5, 
projectedMetrics)
+          assert(projected.head.metrics("numOutputRows").value === 3, 
projectedMetrics)
+        }
+      }
+
+      Seq(
+        SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_ROWS.key,
+        SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_BYTES.key,
+        
SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_SOURCE_BYTES.key
+      ).foreach { limitKey =>
+        withSQLConf(
+            SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+            SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> 
"true",
+            SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED.key 
-> "true",
+            limitKey -> "0",
+            SQLConf.EXCHANGE_REUSE_ENABLED.key -> "true",
+            SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+          val df = sql(query)
+          checkAnswer(df, expected)
+
+          val projected = 
collectWithSubqueries(df.queryExecution.executedPlan) {
+            case subquery: ProjectedBroadcastValueSubqueryExec => subquery
+          }
+          assert(projected.size === 1, df.queryExecution.executedPlan)
+          assert(projected.head.metrics("projectionDisabled").value === 1)
+
+          val pruning = 
collectDynamicPruningExpressions(df.queryExecution.executedPlan)
+            .collectFirst { case in: InSubqueryExec => in }.get
+          assert(pruning.isResultUnavailable)
+          assert(pruning.values().isEmpty)
+
+          val collectError = intercept[SparkException] {
+            projected.head.executeCollect()
+          }
+          assert(collectError.getCondition === "INTERNAL_ERROR")
+          assert(collectError.getMessage.contains("executeCollectResult"))
+
+          val ordinaryIn = pruning.copy(isDynamicPruning = false)
+          val ordinaryInError = intercept[AssertionError] {
+            ordinaryIn.updateResult()
+          }
+          assert(ordinaryInError.getMessage.contains("dynamic partition 
pruning"))
+        }
+      }
+    }
+  }
+
+  test("reuse ancestor broadcast string values with a residual join 
predicate") {
+    withTable("dpp_projection_products", "dpp_projection_promotions",
+        "dpp_projection_sales") {
+      Seq(
+        (101, "books", 10),
+        (202, "toys", 20),
+        (303, "clothing", 30),
+        (404, null.asInstanceOf[String], 40))
+        .toDF("product_id", "category", "price")
+        .write.format(tableFormat).saveAsTable("dpp_projection_products")
+
+      Seq((101, true, 15), (202, true, 25), (303, false, 35), (404, true, 45))
+        .toDF("product_id", "active", "max_price")
+        .write.format(tableFormat).saveAsTable("dpp_projection_promotions")
+
+      Seq(
+        (101, "books"),
+        (202, "toys"),
+        (303, "clothing"),
+        (909, "electronics"))
+        .toDF("product_id", "category")
+        .write.partitionBy("category").format(tableFormat)
+        .saveAsTable("dpp_projection_sales")
+
+      withSQLConf(
+          SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED.key 
-> "true",
+          SQLConf.EXCHANGE_REUSE_ENABLED.key -> "true",
+          SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+        val df = sql(
+          """
+            |WITH eligible AS (
+            |  SELECT /*+ BROADCAST(p) */ p.product_id, p.category
+            |  FROM dpp_projection_products p
+            |  JOIN dpp_projection_promotions promo
+            |    ON p.product_id = promo.product_id
+            |   AND p.price <= promo.max_price
+            |  WHERE promo.active AND promo.product_id > 0
+            |)
+            |SELECT s.product_id, s.category
+            |FROM dpp_projection_sales s
+            |JOIN eligible e
+            |  ON s.product_id = e.product_id
+            | AND s.category = e.category
+            |ORDER BY s.product_id
+            |""".stripMargin)
+
+        checkAnswer(df, Seq(Row(101, "books"), Row(202, "toys")))
+
+        val projected = collectWithSubqueries(df.queryExecution.executedPlan) {
+          case subquery: ProjectedBroadcastValueSubqueryExec => subquery
+        }
+        assert(projected.size === 1, df.queryExecution.executedPlan)
+
+        val pruning = 
collectDynamicPruningExpressions(df.queryExecution.executedPlan)
+          .collectFirst { case in: InSubqueryExec => in }.get
+        assert(pruning.values().get.map(String.valueOf).toSet ===
+          Set("books", "toys", "clothing"))
+      }
+    }
+  }
+
+  test("bind reordered broadcast value columns for partition pruning") {
+    withTable("dpp_reordered_dates", "dpp_reordered_users", 
"dpp_reordered_target") {
+      Seq(
+        (Date.valueOf("2024-01-01"), "us", "2024-01-01"),
+        (Date.valueOf("2024-01-08"), "eu", "2024-01-08"),
+        (Date.valueOf("2024-01-15"), "us", "2024-01-15"))
+        .toDF("cohort_date", "region", "cohort_ds")
+        .write.format(tableFormat).saveAsTable("dpp_reordered_dates")
+
+      Seq(
+        ("2024-01-01", "us", "u1", true),
+        ("2024-01-08", "eu", "u2", true),
+        ("2024-01-15", "us", "inactive", false))
+        .toDF("cohort_ds", "region", "user_id", "active")
+        .write.format(tableFormat).saveAsTable("dpp_reordered_users")
+
+      Seq(
+        ("u1", "20240107", "p1"),
+        ("u2", "20240114", "p2"),
+        ("inactive", "20240121", "extra-domain"))
+        .toDF("user_id", "ds", "payload")
+        
.write.partitionBy("ds").format(tableFormat).saveAsTable("dpp_reordered_target")
+
+      withSQLConf(
+          SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED.key 
-> "true",
+          SQLConf.EXCHANGE_REUSE_ENABLED.key -> "true",
+          SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+        val df = sql(
+          """
+            |WITH reordered_dates AS (
+            |  SELECT cohort_date, region, cohort_ds
+            |  FROM dpp_reordered_dates
+            |), cohorts AS (
+            |  SELECT /*+ BROADCAST(d) */ u.user_id, d.cohort_date
+            |  FROM reordered_dates d
+            |  JOIN dpp_reordered_users u
+            |    ON d.cohort_ds = u.cohort_ds AND d.region = u.region
+            |  WHERE u.active AND u.user_id <> 'filtered'
+            |)
+            |SELECT c.user_id, w.payload
+            |FROM cohorts c
+            |LEFT JOIN dpp_reordered_target w
+            |  ON w.user_id = c.user_id
+            | AND w.ds = date_format(date_add(c.cohort_date, 6), 'yyyyMMdd')
+            |ORDER BY c.user_id, w.payload
+            |""".stripMargin)
+
+        checkAnswer(df, Seq(Row("u1", "p1"), Row("u2", "p2")))
+
+        val projected = collectWithSubqueries(df.queryExecution.executedPlan) {
+          case subquery: ProjectedBroadcastValueSubqueryExec => subquery
+        }
+        assert(projected.size === 1, df.queryExecution.executedPlan)
+        assert(projected.head.child.output.map(_.name) ===
+          Seq("cohort_date", "region", "cohort_ds"))
+
+        val pruning = 
collectDynamicPruningExpressions(df.queryExecution.executedPlan)
+          .collectFirst { case in: InSubqueryExec => in }.get
+        assert(pruning.values().get.map(String.valueOf).toSet ===
+          Set("20240107", "20240114", "20240121"))
+      }
+    }
+  }
+
+  test("project every value row from dense and sparse long-key hash 
broadcasts") {
+    Seq(2L, 1000000000L).foreach { otherKey =>
+      withTable("dpp_long_dates", "dpp_long_users", "dpp_long_target") {
+        Seq(
+          (1L, Date.valueOf("2024-01-01")),
+          (1L, Date.valueOf("2024-01-08")),
+          (otherKey, Date.valueOf("2024-01-15")))
+          .toDF("cohort_id", "cohort_date")
+          .write.format(tableFormat).saveAsTable("dpp_long_dates")
+
+        Seq((1L, "u1", true), (otherKey, "u2", true))
+          .toDF("cohort_id", "user_id", "active")
+          .write.format(tableFormat).saveAsTable("dpp_long_users")
+
+        Seq(
+          ("u1", "2024-01-07", "p1"),
+          ("u1", "2024-01-14", "p2"),
+          ("u2", "2024-01-21", "p3"))
+          .toDF("user_id", "ds", "payload")
+          
.write.partitionBy("ds").format(tableFormat).saveAsTable("dpp_long_target")
+
+        withSQLConf(
+            SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+            SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> 
"true",
+            SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED.key 
-> "true",
+            SQLConf.EXCHANGE_REUSE_ENABLED.key -> "true",
+            SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+          val df = sql(
+            """
+              |WITH cohorts AS (
+              |  SELECT /*+ BROADCAST(d) */ d.cohort_date, u.user_id
+              |  FROM dpp_long_dates d
+              |  JOIN dpp_long_users u ON d.cohort_id = u.cohort_id
+              |  WHERE u.active AND u.user_id <> 'filtered'
+              |)
+              |SELECT c.user_id, w.payload
+              |FROM cohorts c
+              |LEFT JOIN dpp_long_target w
+              |  ON w.user_id = c.user_id
+              | AND w.ds = date_format(date_add(c.cohort_date, 6), 
'yyyy-MM-dd')
+              |ORDER BY c.user_id, w.payload
+              |""".stripMargin)
+
+          checkAnswer(df, Seq(
+            Row("u1", "p1"),
+            Row("u1", "p2"),
+            Row("u2", "p3")))
+
+          val projected = 
collectWithSubqueries(df.queryExecution.executedPlan) {
+            case subquery: ProjectedBroadcastValueSubqueryExec => subquery
+          }
+          assert(projected.size === 1, df.queryExecution.executedPlan)
+          val pruning = 
collectDynamicPruningExpressions(df.queryExecution.executedPlan)
+            .collectFirst { case in: InSubqueryExec => in }.get
+          assert(pruning.values().get.map(String.valueOf).toSet ===
+            Set("2024-01-07", "2024-01-14", "2024-01-21"))
+          val projectedMetrics = projected.head.metrics.map {
+            case (name, metric) => name -> metric.value
+          }
+          assert(projected.head.metrics("numInputRows").value === 3, 
projectedMetrics)
+          assert(projected.head.metrics("numOutputRows").value === 3, 
projectedMetrics)
+        }
+      }
+    }
+  }
+
   /**
    * Collect the children of all correctly pushed down dynamic pruning 
expressions in a spark plan.
    */
@@ -2181,11 +2532,95 @@ abstract class DynamicPartitionPruningV1Suite extends 
DynamicPartitionPruningDat
 }
 
 class DynamicPartitionPruningV1SuiteAEOff extends 
DynamicPartitionPruningV1Suite
-  with DisableAdaptiveExecutionSuite
+  with DisableAdaptiveExecutionSuite {
+
+  test("preserve direct broadcast pruning for existing hash and null-aware 
modes") {
+    withSQLConf(
+        SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true",
+        SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED.key -> 
"false",
+        SQLConf.EXCHANGE_REUSE_ENABLED.key -> "true") {
+      val source = spark.table("product").queryExecution.optimizedPlan
+      val build = QueryExecution.createSparkPlan(spark.sessionState.planner, 
source)
+      val probe = spark.range(4).queryExecution.sparkPlan
+      val mismatchedHashJoin = BroadcastHashJoinExec(
+        Seq(probe.output.head),
+        Seq(build.output(1)),
+        Inner,
+        BuildRight,
+        None,
+        probe,
+        build)
+      
assert(!mismatchedHashJoin.rightKeys.head.semanticEquals(source.output.head))
+
+      val nullAwareJoin = BroadcastHashJoinExec(
+        Seq(probe.output.head),
+        Seq(build.output.head),
+        LeftAnti,
+        BuildRight,
+        None,
+        probe,
+        build,
+        isNullAwareAntiJoin = true)
+      assert(nullAwareJoin.isNullAwareAntiJoin)
+
+      Seq(
+        "mismatched hash mode" -> mismatchedHashJoin,
+        "null-aware anti join" -> nullAwareJoin).foreach { case (name, 
existingJoin) =>
+        val pruning = DynamicPruningSubquery(
+          probe.output.head,
+          source,
+          Seq(source.output.head),
+          Seq(0),
+          onlyInBroadcast = true)()
+        val plan = FilterExec(DynamicPruningExpression(pruning), existingJoin)
+        val rewritten = PlanDynamicPruningFilters(spark).apply(plan)
+        val directFilters = rewritten.expressions.flatMap(_.collect {
+          case in: InSubqueryExec => in
+        })
+
+        assert(directFilters.size === 1, s"$name:\n$rewritten")
+        assert(directFilters.head.plan.isInstanceOf[SubqueryBroadcastExec],
+          s"$name:\n$rewritten")
+      }
+    }
+  }
+}
 
 class DynamicPartitionPruningV1SuiteAEOn extends DynamicPartitionPruningV1Suite
   with EnableAdaptiveExecutionSuite {
 
+  test("adaptive broadcast value projection survives Catalyst copies") {
+    val source = spark.table("product").queryExecution.optimizedPlan
+    val key = source.output.head
+    val child = QueryExecution.createSparkPlan(spark.sessionState.planner, 
source)
+    val projection = BroadcastValueProjection(source, Seq(key), key)
+    val adaptive = SubqueryAdaptiveBroadcastExec(
+      "dynamicpruning", Seq(0), true, source, Seq(key), 
child)(Some(projection))
+
+    assert(adaptive.productArity === 6)
+    
assert(SubqueryAdaptiveBroadcastExec.unapply(adaptive).exists(_.productArity == 
6))
+    assert(Modifier.isTransient(
+      
classOf[SubqueryAdaptiveBroadcastExec].getDeclaredField("broadcastValueProjection")
+        .getModifiers))
+
+    Seq(
+      adaptive.copy()(adaptive.broadcastValueProjection),
+      adaptive.withNewChildren(Seq(ProjectExec(child.output, child)))
+        .asInstanceOf[SubqueryAdaptiveBroadcastExec],
+      
adaptive.makeCopy(adaptive.productIterator.map(_.asInstanceOf[AnyRef]).toArray)
+        .asInstanceOf[SubqueryAdaptiveBroadcastExec]
+    ).foreach { rewritten =>
+      assert(rewritten.broadcastValueProjection.contains(projection))
+    }
+
+    val unprojected = adaptive.copy()(None)
+    assert(adaptive === unprojected)
+    assert(adaptive.canonicalized.asInstanceOf[SubqueryAdaptiveBroadcastExec]
+      .broadcastValueProjection.isEmpty)
+    assert(adaptive.canonicalized === unprojected.canonicalized)
+  }
+
   test("SPARK-39447: Avoid AssertionError in 
AdaptiveSparkPlanExec.doExecuteBroadcast") {
     val df = sql(
       """
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala
index 6cd49948630d..120679f66113 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/PlanStabilitySuite.scala
@@ -189,6 +189,8 @@ trait PlanStabilitySuite extends 
DisableAdaptiveExecutionSuite {
         subqueriesMap.getOrElseUpdate(subquery.id, subqueriesMap.size + 1)
       case subquery: SubqueryBroadcastExec =>
         subqueriesMap.getOrElseUpdate(subquery.id, subqueriesMap.size + 1)
+      case subquery: ProjectedBroadcastValueSubqueryExec =>
+        subqueriesMap.getOrElseUpdate(subquery.id, subqueriesMap.size + 1)
       case ReusedSubqueryExec(subquery) =>
         subqueriesMap.getOrElseUpdate(subquery.id, subqueriesMap.size + 1)
       case _ => -1
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
index d8d885c9b927..d5cc3e6c0c00 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
@@ -20,7 +20,8 @@ package org.apache.spark.sql
 import scala.collection.mutable.ArrayBuffer
 
 import org.apache.spark.SparkRuntimeException
-import org.apache.spark.sql.catalyst.expressions.{EqualTo, NamedExpression, 
OuterReference, SubqueryExpression}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{EqualTo, Literal, 
NamedExpression, OuterReference, SubqueryExpression}
 import org.apache.spark.sql.catalyst.plans.{LeftAnti, LeftSemi}
 import org.apache.spark.sql.catalyst.plans.logical.{Aggregate, Filter, Join, 
LogicalPlan, Project, Sort, Union}
 import org.apache.spark.sql.execution._
@@ -264,6 +265,17 @@ class SubquerySuite extends SharedSparkSession
       Row(3, 3.0) :: Nil)
   }
 
+  test("IN predicate subquery preserves its broadcast when replacing its 
plan") {
+    val subqueryPlan = SubqueryExec("subquery", 
spark.range(3).queryExecution.executedPlan)
+    val subquery = InSubqueryExec(
+      Literal(1L), subqueryPlan, NamedExpression.newExprId, isDynamicPruning = 
false)
+    subquery.updateResult()
+
+    val updatedSubquery = subquery.withNewPlan(subqueryPlan)
+    assert(updatedSubquery.values().isEmpty)
+    assert(updatedSubquery.eval(InternalRow.empty) == true)
+  }
+
   test("NOT IN predicate subquery") {
     checkAnswer(
       sql("select * from l where a not in (select c from r)"),
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2EnhancedRuntimePartitionFilterSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2EnhancedRuntimePartitionFilterSuite.scala
index 5c5a6210f29d..07a214c24fbc 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2EnhancedRuntimePartitionFilterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2EnhancedRuntimePartitionFilterSuite.scala
@@ -20,11 +20,11 @@ package org.apache.spark.sql.connector
 import org.scalatest.BeforeAndAfter
 
 import org.apache.spark.sql.{DataFrame, Row}
-import org.apache.spark.sql.catalyst.expressions.{DynamicPruning, 
DynamicPruningExpression}
+import org.apache.spark.sql.catalyst.expressions.{DynamicPruning, 
DynamicPruningExpression, Literal}
 import org.apache.spark.sql.connector.catalog.{BufferedRows, 
InMemoryEnhancedRuntimePartitionFilterTable, 
InMemoryTableEnhancedRuntimePartitionFilterCatalog}
 import org.apache.spark.sql.connector.expressions.PartitionFieldReference
 import org.apache.spark.sql.connector.expressions.filter.PartitionPredicate
-import org.apache.spark.sql.execution.ExplainUtils.stripAQEPlan
+import org.apache.spark.sql.execution.{ExplainUtils, 
ProjectedBroadcastValueSubqueryExec}
 import org.apache.spark.sql.execution.datasources.v2.BatchScanExec
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
@@ -75,6 +75,93 @@ class DataSourceV2EnhancedRuntimePartitionFilterSuite
       SQLConf.DYNAMIC_PARTITION_PRUNING_FALLBACK_FILTER_RATIO.key -> "10")(f)
   }
 
+  private def withProjectedBroadcastRuntimeFiltering(
+      maxRows: Int,
+      projectionEnabled: Boolean = true)(f: DataFrame => Unit): Unit = {
+    val fact = s"$catalogName.projection_fact"
+    val codes = "broadcast_projection_codes"
+    val selectors = "broadcast_projection_selectors"
+
+    withTable(fact, codes, selectors) {
+      sql(s"CREATE TABLE $fact (id INT, part INT) USING $v2Source PARTITIONED 
BY (part)")
+      for (part <- 0 until 5) {
+        sql(s"INSERT INTO $fact VALUES ($part, $part)")
+      }
+
+      sql(s"CREATE TABLE $codes (join_id INT, part INT) USING parquet")
+      sql(s"INSERT INTO $codes VALUES (1, 2), (2, 3), (3, 4)")
+      sql(s"CREATE TABLE $selectors (join_id INT, keep BOOLEAN) USING parquet")
+      sql(s"INSERT INTO $selectors VALUES (1, true), (2, true), (3, false)")
+
+      withSQLConf(
+          SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> "false",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_ENABLED.key -> "true",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_REUSE_BROADCAST_ONLY.key -> "true",
+          SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_ENABLED.key ->
+            projectionEnabled.toString,
+          SQLConf.DYNAMIC_PARTITION_PRUNING_BROADCAST_PROJECTION_MAX_ROWS.key 
->
+            maxRows.toString,
+          SQLConf.EXCHANGE_REUSE_ENABLED.key -> "true",
+          SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+        val df = sql(
+          s"""
+             |WITH selected AS (
+             |  SELECT /*+ BROADCAST(c) */ c.part
+             |  FROM $codes c
+             |  JOIN $selectors x ON c.join_id = x.join_id
+             |  WHERE x.keep AND x.join_id > 0
+             |)
+             |SELECT /*+ MERGE(f, s) */ f.id, f.part
+             |FROM $fact f
+             |JOIN selected s ON f.part = s.part
+             |ORDER BY f.id
+             |""".stripMargin)
+        f(df)
+      }
+    }
+  }
+
+  test("disabled broadcast projection preserves existing no-op iterative V2 
filtering") {
+    withProjectedBroadcastRuntimeFiltering(maxRows = 10, projectionEnabled = 
false) { df =>
+      checkAnswer(df, Seq(Row(2, 2), Row(3, 3)))
+      assertDPPRuntimeFilters(df)
+      assert(collectBatchScan(df).runtimeFilters.contains(
+        DynamicPruningExpression(Literal.TrueLiteral)))
+      assertPushedPartitionPredicates(df, expectedCount = 1)
+      assert(getPushedPartitionPredicates(df).head.references().isEmpty)
+      assertScanReturnsPartitionKeys(df, Set("0", "1", "2", "3", "4"))
+    }
+  }
+
+  test("unavailable broadcast values do not reach iterative V2 partition 
filtering") {
+    withProjectedBroadcastRuntimeFiltering(maxRows = 0) { df =>
+      checkAnswer(df, Seq(Row(2, 2), Row(3, 3)))
+      val projected = 
ExplainUtils.collectWithSubqueries(df.queryExecution.executedPlan) {
+        case subquery: ProjectedBroadcastValueSubqueryExec => subquery
+      }
+      assert(projected.size === 1, df.queryExecution.executedPlan)
+      assert(projected.head.metrics("projectionDisabled").value === 1)
+      assertDPPRuntimeFilters(df)
+      assertPushedPartitionPredicates(df, expectedCount = 0)
+      assertScanReturnsPartitionKeys(df, Set("0", "1", "2", "3", "4"))
+    }
+  }
+
+  test("projected broadcast values prune iterative V2 partitions with a safe 
superset") {
+    withProjectedBroadcastRuntimeFiltering(maxRows = 10) { df =>
+      checkAnswer(df, Seq(Row(2, 2), Row(3, 3)))
+      val projected = 
ExplainUtils.collectWithSubqueries(df.queryExecution.executedPlan) {
+        case subquery: ProjectedBroadcastValueSubqueryExec => subquery
+      }
+      assert(projected.size === 1, df.queryExecution.executedPlan)
+      assert(projected.head.metrics("numInputRows").value === 3)
+      assert(projected.head.metrics("numOutputRows").value === 3)
+      assertDPPRuntimeFilters(df)
+      assertPushedPartitionPredicates(df, expectedCount = 1)
+      assertScanReturnsPartitionKeys(df, Set("2", "3", "4"))
+    }
+  }
+
   // 
---------------------------------------------------------------------------
   // PartitionPredicate IS created
   // 
---------------------------------------------------------------------------
@@ -520,7 +607,7 @@ class DataSourceV2EnhancedRuntimePartitionFilterSuite
   }
 
   private def collectBatchScan(df: DataFrame): BatchScanExec = {
-    stripAQEPlan(df.queryExecution.executedPlan).collectFirst {
+    ExplainUtils.collectFirst(df.queryExecution.executedPlan) {
       case b: BatchScanExec => b
     }.getOrElse(fail("Expected BatchScanExec in plan"))
   }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to