ludlows commented on PR #6760:
URL: https://github.com/apache/iceberg/pull/6760#issuecomment-1469811907
Hi @szehon-ho ,
either a deterministic true or false expression will lead to a None.
it seems that we should distinguish them. and the problem can be solved.
here I implement a function to do that.
```scala
def collectDeterministicSparkExpression(session: SparkSession,
tableName: String, where: String):
Boolean = {
// used only to check if a deterministic expression is true or false
val tableAttrs = session.table(tableName).queryExecution.analyzed.output
val firstColumnName = tableAttrs.head.name
val anotherWhere = s"$firstColumnName is not null and $where"
val unresolvedExpression =
session.sessionState.sqlParser.parseExpression(anotherWhere)
val filter = Filter(unresolvedExpression, DummyRelation(tableAttrs))
val optimizedLogicalPlan =
session.sessionState.executePlan(filter).optimizedPlan
val option = optimizedLogicalPlan.collectFirst {
case filter: Filter => Some(filter.condition)
}.getOrElse(Option.empty)
if (option.isDefined) true else false
}
```
following this idea, I adapt the code in `RewriteDataFilesProcedure` and add
more test cases.
how do you think about it?
many thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
