This is an automated email from the ASF dual-hosted git repository.

szehon-ho pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new 72771ac6e97f [SPARK-60486][4.2][SQL] Preserve Python UDF partition 
filters after V2 pushdown
72771ac6e97f is described below

commit 72771ac6e97fb0f7c9641daf7eb6128ac8863e87
Author: 小哥不太逍遥 <[email protected]>
AuthorDate: Thu Oct 29 13:52:12 2026 -0700

    [SPARK-60486][4.2][SQL] Preserve Python UDF partition filters after V2 
pushdown
    
    ### What changes were proposed in this pull request?
    
    Backport #59732 to branch-4.2, as requested in the original PR. Return 
partition predicates rejected by pushdown for post-scan evaluation, preserve 
this branch's data-filter translation loop, and include the shared Python UDF 
regression and 4.2.2 migration note.
    
    ### Why are the changes needed?
    
    A deterministic Python UDF on a partition column can be dropped from both 
the pushed and residual filters in a V2 file scan. In the regression, a filter 
that should select two rows returns all six.
    
    ### Does this PR introduce any user-facing change?
    
    Yes. Affected Data Source V2 file scans now exclude rows that do not 
satisfy these predicates. Built-in file sources use V1 by default; that path 
already evaluates the filters correctly. The migration note explains that data 
derived from affected results may need to be recomputed.
    
    ### How was this patch tested?
    
    - Built the original branch with Hive support, then rebuilt the patched SQL 
reactor. `PruneFileSourcePartitionsSuite`: 9/9 passed.
    - The two shared regressions in `UDFTests` and `ArrowPythonUDFTests` both 
failed with the original production code and both passed with this patch. 
Runtime checks verified that the 4.2 PySpark sources and newly compiled 
`FileScanBuilder` class were loaded.
    - These local runs used a Windows fixture-only adapter to close the 
temporary file handle before unlinking it, as described in #59732.
    - Branch-pinned Ruff 0.14.8 and scalafmt checks passed. [Fork CI run 
40824048618](https://github.com/xgbtxy/spark/actions/runs/40824048618) passed 
on commit `e03c793`: 30 jobs succeeded and 1 was skipped. Its test report 
records 51,734 passed, 1,069 skipped, and no failures.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    Closes #59903 from xgbtxy/backport/spark-60486-4.2.
    
    Authored-by: 小哥不太逍遥 <[email protected]>
    Signed-off-by: Szehon Ho <[email protected]>
---
 docs/sql-migration-guide.md                         |  4 ++++
 python/pyspark/sql/tests/test_udf.py                | 21 +++++++++++++++++++--
 .../execution/datasources/v2/FileScanBuilder.scala  |  7 ++++---
 3 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index d70a63fd2cac..cc43967caaf1 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -22,6 +22,10 @@ license: |
 * Table of contents
 {:toc}
 
+## Upgrading from Spark SQL 4.2.1 to 4.2.2
+
+- Since Spark 4.2.2, Data Source V2 file scans correctly evaluate 
deterministic Python UDF filter predicates that reference only partition 
columns. Previously, these predicates could be dropped, returning rows that did 
not satisfy the filter (SPARK-60486). Data derived from affected query results 
may need to be recomputed.
+
 ## Upgrading from Spark SQL 4.2.0 to 4.2.1
 
 - Spark 4.2.0 contains a correctness bug in the Data Source V2 
`PartitionPredicate`. For scans, `PartitionPredicate` is the additional 
predicate type Spark sends during the second pass of iterative pushdown. When a 
data source opts into iterative pushdown by returning `true` from 
`supportsIterativePushdown()`, Spark may call its pushdown API again with a 
`PartitionPredicate` that evaluates a filter against the full partition key 
described by `Table.partitioning()`. Spark also uses `Parti [...]
diff --git a/python/pyspark/sql/tests/test_udf.py 
b/python/pyspark/sql/tests/test_udf.py
index ce7a42469e18..72a26d813c30 100644
--- a/python/pyspark/sql/tests/test_udf.py
+++ b/python/pyspark/sql/tests/test_udf.py
@@ -889,12 +889,29 @@ class BaseUDFTestsMixin:
 
         try:
             with self.sql_conf({"spark.sql.sources.useV1SourceList": ""}):
-                
self.spark.range(1).write.mode("overwrite").format("parquet").save(path)
-                df = self.spark.read.parquet(path).toDF("i")
+                (
+                    self.spark.range(6)
+                    .selectExpr("id AS i", "id % 3 AS p")
+                    .write.mode("overwrite")
+                    .partitionBy("p")
+                    .parquet(path)
+                )
+                df = self.spark.read.parquet(path)
                 f = udf(lambda x: False, "boolean")(lit(1))
                 result = df.filter(f)
                 self.assertEqual(0, result.count())
 
+                # SPARK-60486: partition predicates that cannot be pushed must 
still be evaluated.
+                matches_partition = udf(lambda x: x == 1, "boolean")(col("p"))
+                expected = [Row(i=1, p=1), Row(i=4, p=1)]
+                assertDataFrameEqual(df.filter(col("p") == 1), expected)
+                assertDataFrameEqual(df.filter(matches_partition), expected)
+                assertDataFrameEqual(df.filter(matches_partition & (col("p") > 
0)), expected)
+                assertDataFrameEqual(
+                    df.filter(matches_partition | (col("p") == 0)),
+                    [Row(i=0, p=0), Row(i=1, p=1), Row(i=3, p=0), Row(i=4, 
p=1)],
+                )
+
         finally:
             shutil.rmtree(path)
 
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
index 7e0bc25a9a1e..14f6192431f0 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
@@ -31,7 +31,7 @@ abstract class FileScanBuilder(
     sparkSession: SparkSession,
     fileIndex: PartitioningAwareFileIndex,
     dataSchema: StructType)
-  extends ScanBuilder
+    extends ScanBuilder
     with SupportsPushDownRequiredColumns
     with SupportsPushDownCatalystFilters {
   private val partitionSchema = fileIndex.partitionSchema
@@ -73,10 +73,11 @@ abstract class FileScanBuilder(
     val (deterministicFilters, nonDeterminsticFilters) = 
filters.partition(_.deterministic)
     val (partitionFilters, dataFilters) =
       DataSourceUtils.getPartitionFiltersAndDataFilters(partitionSchema, 
deterministicFilters)
-    this.partitionFilters = partitionFilters.filter { f =>
+    val (partitionFiltersToPush, postScanPartitionFilters) = 
partitionFilters.partition { f =>
       // Python UDFs might exist because this rule is applied before 
``ExtractPythonUDFs``.
       !SubqueryExpression.hasSubquery(f) && 
!f.exists(_.isInstanceOf[PythonUDF])
     }
+    this.partitionFilters = partitionFiltersToPush
     this.dataFilters = dataFilters
     val translatedFilters = mutable.ArrayBuffer.empty[sources.Filter]
     for (filterExpr <- dataFilters) {
@@ -86,7 +87,7 @@ abstract class FileScanBuilder(
       }
     }
     pushedDataFilters = pushDataFilters(translatedFilters.toArray)
-    dataFilters ++ nonDeterminsticFilters
+    dataFilters ++ postScanPartitionFilters ++ nonDeterminsticFilters
   }
 
   override def pushedFilters: Array[Predicate] = pushedDataFilters.map(_.toV2)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to