daijy commented on PR #16262: URL: https://github.com/apache/iceberg/pull/16262#issuecomment-4589817827
Thanks @szehon-ho @huaxingao! Yes, the new pushPredicates is what we need. The key advantage is that partition expression evaluation happens in Spark's own code (PartitionPredicate.eval), which means Spark can handle all possible expressions. The data source doesn't need to implement its own expression evaluator. I haven't seen Iceberg implement support for this yet, but the integration should be straightforward. Given this approach, we no longer need to introduce a new transform, which makes this PR unnecessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
