rdblue commented on a change in pull request #3945:
URL: https://github.com/apache/iceberg/pull/3945#discussion_r792221239
##########
File path:
core/src/test/java/org/apache/iceberg/TestOverwriteWithValidation.java
##########
@@ -333,6 +333,31 @@ public void
testOverwriteCompatibleAdditionStrictValidated() {
committedSnapshotId, table.currentSnapshot().snapshotId());
}
+ @Test
+ public void testOverwriteCompatibleAdditionStrictValidatedNoConflict() {
Review comment:
This is an area where we're really like for Spark to be able to support
transform expressions in pushdown. Iceberg already supports expressions like
`day(ts) = '2022-01-01'` and that would be really convenient for some use cases.
Until then, one thing that we can do is to start accepting predicates from
Spark for the `_partition` metadata column. In Spark 3.2, we've added a
metadata column that returns the partition for a given row. I think that we
could recognize predicates for those columns and convert them into expressions.
For example, we could take `_partition.id_bucket=55` and convert it to
`bucket(256, id) = 55` and push that down to cover the use case you're trying
to solve here.
Would you like to try implementing this, @coolderli?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
