anuragmantri commented on PR #18615:
URL: https://github.com/apache/iceberg/pull/18615#issuecomment-6352200563
Thanks for the context @szehon-ho, that makes sense. I reproduced it to
confirm. The following test fails on this PR.
```Java
@TestTemplate
void partitionPredicateErrorIsNotSwallowed() {
sql("CREATE TABLE %s (id INT, dep STRING) USING iceberg PARTITIONED BY
(dep)", tableName);
configurePlanningMode(planningMode);
setSmallSplitSize();
sql("INSERT INTO %s VALUES (1, 'd1')", tableName);
sql("INSERT INTO %s VALUES (2, 'd2')", tableName);
sql("INSERT INTO %s VALUES (3, 'd3')", tableName);
spark
.udf()
.register(
"fails_on_d3",
(String value) -> {
if ("d3".equals(value)) {
throw new IllegalStateException("Cannot evaluate " + value);
}
return "d2".equals(value);
},
DataTypes.BooleanType);
assertThatThrownBy(() -> sql("SELECT * FROM %s WHERE fails_on_d3(dep)",
tableName))
.as("Spark must evaluate the predicate on rows of the partition where
pruning failed")
.hasMessageContaining("fails_on_d3");
}
```
I'm moving this PR to draft until the Spark fix is released. After that I'll
rebase and add the test here as a regression test. Do you have the Spark issue
number I can link here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
