cloud-fan commented on code in PR #56636:
URL: https://github.com/apache/spark/pull/56636#discussion_r3454400450
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkOptimizer.scala:
##########
@@ -112,7 +112,13 @@ class SparkOptimizer(
V2ScanRelationPushDown.ruleName,
V2ScanPartitioningAndOrdering.ruleName,
V2Writes.ruleName,
- ReplaceCTERefWithRepartition.ruleName)
+ ReplaceCTERefWithRepartition.ruleName,
+ // CleanupDynamicPruningFilters finalizes the DPP predicates inserted by
PartitionPruning --
+ // notably rewriting non-deterministic ones to `true` so they are not
re-evaluated. That is
+ // correctness behavior, not an optional optimization, so the rule must
not be excludable.
+ // Disabling DPP is done by excluding PartitionPruning (the inserter),
after which this rule
+ // is a no-op.
+ CleanupDynamicPruningFilters.ruleName)
Review Comment:
My agent actually added but I removed it. The non-excludable-rules framework
is not new in this PR, and I don't see we add one test for each rule added in
the list. We should trust this long-standing non-excludable-rules framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
