shivadarshan-devadiga opened a new pull request, #18992:
URL: https://github.com/apache/iceberg/pull/18992

   Closes #18949.
   
   [SPARK-59572](https://github.com/apache/spark/pull/58851) makes a 
`PartitionPredicate` that fails to evaluate during planning fail the query, 
instead of being reported as a partition match after the predicate has been 
fully pushed down. This adds `partitionPredicateEvaluationFailureIsPropagated` 
to `TestFilterPushDown` in `spark/v4.2`. It partitions a table by a string 
column, filters with a UDF that throws `NumberFormatException` for one 
partition value, and checks that planning the query fails with 
`FAILED_EXECUTE_UDF`, caused by that `NumberFormatException`. The test is the 
regression from the Spark 4.2.1 verification draft #18294 by @manuzhang, kept 
as a permanent test.
   
   It is added to Spark 4.2 only: `PartitionPredicate` pushdown is new in Spark 
4.2, and only `spark/v4.2` tests it (`partitionPredicatePushdownWithUDF`).
   
   Testing:
   - The new test passes in both planning modes on Spark 4.2.1, the version on 
main. The rest of `TestFilterPushDown` passes too.
   - With `spark42` set to 4.2.0, which doesn't have SPARK-59572, the new test 
fails in both planning modes: planning doesn't throw, so the failing predicate 
is treated as a match.
   - `spotlessCheck` passes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to