szehon-ho commented on PR #58412: URL: https://github.com/apache/spark/pull/58412#issuecomment-5530257020
I opened #58503 as the `master` follow-up requested by the review here. It addresses the upstream items before this 4.3 backport proceeds: - restricts the V1, V2, Catalyst, and row-level in-memory fixtures to identity partition transforms while preserving the actual partition-key ordinal; - validates that every `fullyPushedFilterAttributes()` entry is also an exact `filterAttributes()` entry, including the top-level struct-root case; - routes DPP, row-level group filtering, and iterative pushdown through the structured runtime-filter attribute validation; - changes the API wording to say Spark *currently* tracks eligibility by root attribute; and - adds regressions for transformed partition sources and invalid fully-pushed declarations. The focused Catalyst, V1/V2 DPP, and both row-level implementations pass, along with compilation, error-condition validation, scalastyle, and checkstyle. Once #58503 lands in `master` and `branch-4.x`, this backport can be updated and reconsidered. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
