Sumit Singh created HUDI-9502:
---------------------------------
Summary: Partition Pruning Fails for Non-Deterministic Filters in
Spark 3.3.x
Key: HUDI-9502
URL: https://issues.apache.org/jira/browse/HUDI-9502
Project: Apache Hudi
Issue Type: Bug
Components: spark-sql
Reporter: Sumit Singh
h4. *Summary*
Partition pruning in Hudi fails when a query includes *non-deterministic
expressions* (e.g. {{{}rand(){}}}) in the filter clause, even when a
deterministic *partition filter* is present. This leads to *full partition
scans* instead of optimized reads, significantly impacting performance.
h4. *Observed Behavior*
In {{{}HoodiePruneFileSourcePartitions.scala{}}}, the pruning rule uses:
{code:java}
override def apply(plan: LogicalPlan): LogicalPlan = plan transformDown {
case op @ PhysicalOperation(projects, filters, lr @
LogicalRelation(HoodieRelationMatcher(fileIndex), _, _, _))
if !fileIndex.hasPredicatesPushedDown => {code}
This uses {{{}PhysicalOperation{}}}, which in turn extends
{{{}OperationHelper{}}}. The {{collectProjectsAndFilters}} method inside
{{OperationHelper}} checks:
{code:java}
!legacyMode || condition.deterministic {code}
Since {{legacyMode = true}} for {{{}PhysicalOperation{}}}, and if the condition
is {*}not deterministic{*}, the filter is not collected — leading to
*{{filters}} being empty* and partition pruning being skipped.
h4. *Comparison with Spark's Native Pruning*
Spark uses {{PruneFileSourcePartitions}} for native Parquet/Hive tables. Its
code pattern is nearly identical, but it relies on {{{}ScanOperation{}}}, which
overrides:
{code:java}
override protected def legacyMode: Boolean = false {code}
This allows Spark to {*}collect deterministic partition filters even when mixed
with non-deterministic ones{*}, and successfully apply partition pruning in the
*physical plan only* (not logical).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
