mrutunjay-kinagi commented on PR #58976: URL: https://github.com/apache/spark/pull/58976#issuecomment-5791541101
Thanks for the detailed trace. The SPARK-55702 connection is useful context I had not pinned down, and it holds up: a filtered `First` could not reach `OptimizeWindowFunctions` before 4.2, because `WindowResolution` rejected `FILTER` on a window aggregate at analysis time. Both points addressed: **ignoreNulls + FILTER.** Added. The plan case now iterates over both rewritable frames and both `ignoreNulls` values, so `First(a, true)` with a filter is covered. **Affects Version.** Agreed, 4.2.0 rather than 5.0.0. SPARK-55702 carries fixVersion 4.2.0, so that is when the rewrite could first encounter a filtered aggregate. I will get the JIRA field corrected. On `isDistinct`, agreed it is safe to leave out of the guard. I had reasoned from `WindowResolution` rejecting a distinct window aggregate before the optimizer runs, which makes such a guard unreachable; the `isDuplicateAgnostic` and `EliminateDistinct` route you traced is the stronger argument. On the `window.sql` golden case, agreed it cannot catch this, since the first row in the sort order already satisfies the filter and both plans therefore agree. I was not able to run `SQLQueryTestSuite` locally, so I am relying on CI to confirm the golden output is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
