szehon-ho commented on PR #58145: URL: https://github.com/apache/spark/pull/58145#issuecomment-5421029568
Follow-up: while validating the physical-removal fix, I found another path where an advisory filter could reach `FilterExec`. Advisory filters were materialized above residual filters; when a residual was non-deterministic, `PhysicalOperation` stopped at that barrier and `BasicOperators` could plan the outer advisory `Filter` before `DataSourceV2Strategy` removed it. Fixed in de3d0eac59d by keeping advisory filters closest to the scan. The change also rejects all `UserDefinedExpression` advisory predicates, including deterministic UDFs, because UDF extraction can introduce a similar barrier. Added an end-to-end non-deterministic residual regression and focused UDF coverage. The focused strategy and end-to-end suites pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
