shardulm94 opened a new issue #1952:
URL: https://github.com/apache/iceberg/issues/1952
```
scala> val df = spark.read.format("iceberg").load("/tmp/iceberg/logs")
df: org.apache.spark.sql.DataFrame = [level: string]
scala> df.filter(not($"level".startsWith("b"))).show()
java.lang.IllegalArgumentException: No negation for operation: STARTS_WITH
at
org.apache.iceberg.expressions.Expression$Operation.negate(Expression.java:72)
at
org.apache.iceberg.expressions.UnboundPredicate.negate(UnboundPredicate.java:69)
at org.apache.iceberg.expressions.RewriteNot.not(RewriteNot.java:44)
at org.apache.iceberg.expressions.RewriteNot.not(RewriteNot.java:22)
at
org.apache.iceberg.expressions.ExpressionVisitors.visit(ExpressionVisitors.java:293)
at
org.apache.iceberg.expressions.Projections$BaseProjectionEvaluator.project(Projections.java:153
```
There are multiple places in Iceberg which call `RewriteNot` on the user
provided filters
([1](https://github.com/apache/iceberg/blob/425a45f8acec0496d77e070c07fb209de92ab2c1/api/src/main/java/org/apache/iceberg/expressions/Projections.java#L153),
[2](https://github.com/apache/iceberg/blob/425a45f8acec0496d77e070c07fb209de92ab2c1/api/src/main/java/org/apache/iceberg/expressions/InclusiveMetricsEvaluator.java#L59),
[3](https://github.com/apache/iceberg/blob/425a45f8acec0496d77e070c07fb209de92ab2c1/api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java#L67)
and many more). However, the `STARTS_WITH` predicate does not support negation
and hence `RewriteNot` throws an exception. Should we implement a
`NOT_STARTS_WITH` operation to support this usecase?
Another approach would be to just keep `NOT` expression in `RewriteNot` for
predicates that do not support negation. However some evaluators in Iceberg
make the assumption that there are no `NOT`s in the tree and it may affect
their correctness.
https://github.com/apache/iceberg/blob/425a45f8acec0496d77e070c07fb209de92ab2c1/api/src/main/java/org/apache/iceberg/expressions/Projections.java#L148
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
