Thank you for the clarification.. Right , the supposition makes sense, for generalized pruning behaviour, I suppose that is all that can be done...
On Sun, Dec 7, 2025, 8:10 PM Wenchen Fan <[email protected]> wrote: > You can find the initial commit in the commit history: > https://github.com/apache/spark/commit/3db8ec258c4a8438bda73c26fc7b1eb6f9d51631 > > Yes, and it's reasonable as we can't know if the new node has the same > pruning behavior of the previous node. > > On Sat, Dec 6, 2025 at 5:33 AM Asif Shahid <[email protected]> wrote: > >> In the transform API : >> def transformUpWithPruning(cond: TreePatternBits => Boolean, >> ruleId: RuleId = UnknownRuleId)(rule: PartialFunction[BaseType, >> BaseType]) >> >> my understanding is that the information whether a TreeNode ( be Plan or >> Expression) keeps the state in a local var in the Plan, to track if the >> rule is effective or not, based on first time traversal results. >> >> Now if say a batch has two rules ( Rule A and rule B) , such that rule B >> changes the node ( Plan or Expression) , but rule A does not.. >> Am I right in understanding that in such case the effect of pruning will >> be lost for rule A, as rule B is creating new nodes and hence information >> will be lost? >> >> Regards >> Asif >> >
