rkkorlapati-db opened a new pull request, #41163: URL: https://github.com/apache/spark/pull/41163
### What changes were proposed in this pull request? Allow expression lineage through partial aggregation (AggregatePart) and Window operators. This enables optimizations such as DFP, bloom filters to trace down expression lineage and do necessary optimizations. Also, enhanced the logic to accept the following additional parameters to be able to customize lineage logic. noLineageOnNullPaddingSide: If true, lineage is not tracked down through the null padding side of the outer joins. stopAtSubPlan: If specified, lineage tracking stops at this specified operator. Code changes include delegating the core logic to findExpressionAndTrackLineageDownBase() and the current signature findExpressionAndTrackLineageDown() would become wrapper to avoid changes to the existing callers. ### Why are the changes needed? To support optimizations such as bloom filters, DFP, etc. which depend on the expression lineage. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tested -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
