szehon-ho commented on code in PR #58370:
URL: https://github.com/apache/spark/pull/58370#discussion_r3883195173
##########
sql/core/src/test/scala/org/apache/spark/sql/connector/DataSourceV2SQLSuite.scala:
##########
@@ -5440,11 +5440,49 @@ class DataSourceV2SQLSuiteV1Filter
}
class DataSourceV2SQLSuiteV2Filter extends DataSourceV2SQLSuite {
Review Comment:
Fixed in 6019cecc278. The legacy InMemoryBatchScan now retains resolvable
nested references, and the nested DPP pruning test runs through the shared base
suite for both SupportsRuntimeFiltering and SupportsRuntimeV2Filtering. Both
focused V1 and V2 cases pass.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -223,21 +223,11 @@ case class DataSourceV2ScanRelation(
case s: SupportsRuntimeCatalystFiltering =>
s.fullyPushedFilterAttributes()
case _ => Array.empty[NamedReference]
}
- resolveTopLevelFilterAttrs(filterAttrs)
+ resolveFilterAttrs(filterAttrs)
Review Comment:
Fixed in 6019cecc278. Nested fully-pushed references are rejected before
they can be collapsed to the root AttributeReference in AttributeSet. The new
sibling-field regression verifies that a predicate on s.other is retained when
s.part is the declared nested filter attribute. Ordinary nested
filterAttributes remain supported for V1, V2, and Catalyst compatibility.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala:
##########
@@ -223,21 +223,11 @@ case class DataSourceV2ScanRelation(
case s: SupportsRuntimeCatalystFiltering =>
s.fullyPushedFilterAttributes()
case _ => Array.empty[NamedReference]
}
- resolveTopLevelFilterAttrs(filterAttrs)
+ resolveFilterAttrs(filterAttrs)
Review Comment:
Fixed in 6019cecc278. Ordinary nested filterAttributes remain supported, but
fullyPushedFilterAttributes now rejects nested references before conversion to
AttributeSet. I also added exact-path rejection and a sibling-field regression
proving the residual predicate stays above the scan. The focused Catalyst suite
passes 19/19 tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
