pan3793 commented on PR #58851: URL: https://github.com/apache/spark/pull/58851#issuecomment-5707587835
@dongjoon-hyun both points are real, and both are now in the description and the migration guide. 1. Confirmed, and connector-dependent as you say. I measured your example: `dep <> 'hr' AND to_int(dep) = 1` still succeeds against the in-memory source, because its predicate loop short-circuits before reaching the UDF, while `data <> 'x' AND to_int(dep) = 1` throws, since the data-column conjunct stays post-scan and cannot shield the partition predicate. Spark defines no order here, so the guide says the outcome depends on the source. 2. Confirmed with one correction worth recording: `EXPLAIN` does not fail. It prints `Error occurred during query planning:` followed by the error in place of the plan. The empty-partition case follows from evaluating partition metadata rather than rows, and is noted too. @tdcmeehan added in 9b8d884. Since this targets 4.3.0 and 4.2.1, there are two entries: one under "Upgrading from Spark SQL 4.2 to 4.3" and one under a new "Upgrading from Spark SQL 4.2.0 to 4.2.1" section. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
