This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 72771ac6e97f [SPARK-60486][4.2][SQL] Preserve Python UDF partition
filters after V2 pushdown
72771ac6e97f is described below
commit 72771ac6e97fb0f7c9641daf7eb6128ac8863e87
Author: 小哥不太逍遥 <[email protected]>
AuthorDate: Thu Oct 29 13:52:12 2026 -0700
[SPARK-60486][4.2][SQL] Preserve Python UDF partition filters after V2
pushdown
### What changes were proposed in this pull request?
Backport #59732 to branch-4.2, as requested in the original PR. Return
partition predicates rejected by pushdown for post-scan evaluation, preserve
this branch's data-filter translation loop, and include the shared Python UDF
regression and 4.2.2 migration note.
### Why are the changes needed?
A deterministic Python UDF on a partition column can be dropped from both
the pushed and residual filters in a V2 file scan. In the regression, a filter
that should select two rows returns all six.
### Does this PR introduce any user-facing change?
Yes. Affected Data Source V2 file scans now exclude rows that do not
satisfy these predicates. Built-in file sources use V1 by default; that path
already evaluates the filters correctly. The migration note explains that data
derived from affected results may need to be recomputed.
### How was this patch tested?
- Built the original branch with Hive support, then rebuilt the patched SQL
reactor. `PruneFileSourcePartitionsSuite`: 9/9 passed.
- The two shared regressions in `UDFTests` and `ArrowPythonUDFTests` both
failed with the original production code and both passed with this patch.
Runtime checks verified that the 4.2 PySpark sources and newly compiled
`FileScanBuilder` class were loaded.
- These local runs used a Windows fixture-only adapter to close the
temporary file handle before unlinking it, as described in #59732.
- Branch-pinned Ruff 0.14.8 and scalafmt checks passed. [Fork CI run
40824048618](https://github.com/xgbtxy/spark/actions/runs/40824048618) passed
on commit `e03c793`: 30 jobs succeeded and 1 was skipped. Its test report
records 51,734 passed, 1,069 skipped, and no failures.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59903 from xgbtxy/backport/spark-60486-4.2.
Authored-by: 小哥不太逍遥 <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
---
docs/sql-migration-guide.md | 4 ++++
python/pyspark/sql/tests/test_udf.py | 21 +++++++++++++++++++--
.../execution/datasources/v2/FileScanBuilder.scala | 7 ++++---
3 files changed, 27 insertions(+), 5 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index d70a63fd2cac..cc43967caaf1 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -22,6 +22,10 @@ license: |
* Table of contents
{:toc}
+## Upgrading from Spark SQL 4.2.1 to 4.2.2
+
+- Since Spark 4.2.2, Data Source V2 file scans correctly evaluate
deterministic Python UDF filter predicates that reference only partition
columns. Previously, these predicates could be dropped, returning rows that did
not satisfy the filter (SPARK-60486). Data derived from affected query results
may need to be recomputed.
+
## Upgrading from Spark SQL 4.2.0 to 4.2.1
- Spark 4.2.0 contains a correctness bug in the Data Source V2
`PartitionPredicate`. For scans, `PartitionPredicate` is the additional
predicate type Spark sends during the second pass of iterative pushdown. When a
data source opts into iterative pushdown by returning `true` from
`supportsIterativePushdown()`, Spark may call its pushdown API again with a
`PartitionPredicate` that evaluates a filter against the full partition key
described by `Table.partitioning()`. Spark also uses `Parti [...]
diff --git a/python/pyspark/sql/tests/test_udf.py
b/python/pyspark/sql/tests/test_udf.py
index ce7a42469e18..72a26d813c30 100644
--- a/python/pyspark/sql/tests/test_udf.py
+++ b/python/pyspark/sql/tests/test_udf.py
@@ -889,12 +889,29 @@ class BaseUDFTestsMixin:
try:
with self.sql_conf({"spark.sql.sources.useV1SourceList": ""}):
-
self.spark.range(1).write.mode("overwrite").format("parquet").save(path)
- df = self.spark.read.parquet(path).toDF("i")
+ (
+ self.spark.range(6)
+ .selectExpr("id AS i", "id % 3 AS p")
+ .write.mode("overwrite")
+ .partitionBy("p")
+ .parquet(path)
+ )
+ df = self.spark.read.parquet(path)
f = udf(lambda x: False, "boolean")(lit(1))
result = df.filter(f)
self.assertEqual(0, result.count())
+ # SPARK-60486: partition predicates that cannot be pushed must
still be evaluated.
+ matches_partition = udf(lambda x: x == 1, "boolean")(col("p"))
+ expected = [Row(i=1, p=1), Row(i=4, p=1)]
+ assertDataFrameEqual(df.filter(col("p") == 1), expected)
+ assertDataFrameEqual(df.filter(matches_partition), expected)
+ assertDataFrameEqual(df.filter(matches_partition & (col("p") >
0)), expected)
+ assertDataFrameEqual(
+ df.filter(matches_partition | (col("p") == 0)),
+ [Row(i=0, p=0), Row(i=1, p=1), Row(i=3, p=0), Row(i=4,
p=1)],
+ )
+
finally:
shutil.rmtree(path)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
index 7e0bc25a9a1e..14f6192431f0 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileScanBuilder.scala
@@ -31,7 +31,7 @@ abstract class FileScanBuilder(
sparkSession: SparkSession,
fileIndex: PartitioningAwareFileIndex,
dataSchema: StructType)
- extends ScanBuilder
+ extends ScanBuilder
with SupportsPushDownRequiredColumns
with SupportsPushDownCatalystFilters {
private val partitionSchema = fileIndex.partitionSchema
@@ -73,10 +73,11 @@ abstract class FileScanBuilder(
val (deterministicFilters, nonDeterminsticFilters) =
filters.partition(_.deterministic)
val (partitionFilters, dataFilters) =
DataSourceUtils.getPartitionFiltersAndDataFilters(partitionSchema,
deterministicFilters)
- this.partitionFilters = partitionFilters.filter { f =>
+ val (partitionFiltersToPush, postScanPartitionFilters) =
partitionFilters.partition { f =>
// Python UDFs might exist because this rule is applied before
``ExtractPythonUDFs``.
!SubqueryExpression.hasSubquery(f) &&
!f.exists(_.isInstanceOf[PythonUDF])
}
+ this.partitionFilters = partitionFiltersToPush
this.dataFilters = dataFilters
val translatedFilters = mutable.ArrayBuffer.empty[sources.Filter]
for (filterExpr <- dataFilters) {
@@ -86,7 +87,7 @@ abstract class FileScanBuilder(
}
}
pushedDataFilters = pushDataFilters(translatedFilters.toArray)
- dataFilters ++ nonDeterminsticFilters
+ dataFilters ++ postScanPartitionFilters ++ nonDeterminsticFilters
}
override def pushedFilters: Array[Predicate] = pushedDataFilters.map(_.toV2)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]