ahmedabu98 commented on code in PR #39720:
URL: https://github.com/apache/beam/pull/39720#discussion_r3761414797
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/AddFiles.java:
##########
@@ -650,6 +650,24 @@ static String getPartitionFromMetrics(Metrics metrics,
InputFile inputFile, Tabl
"Min and max transformed values were not equal, for column: " +
field.name());
}
+ // Equal transformed bounds only cover the values in between for
+ // order-preserving transforms. For bucket, which hashes the value mod
N
+ // , min and max can land in the same bucket while intermediate values
+ // land in others. The void transform maps every value to null and
needs
+ // no check.
+ if (!transform.preservesOrder()
+ && !transform.isVoid()
+ && !Objects.deepEquals(
+ Conversions.fromByteBuffer(type, lowerBytes),
+ Conversions.fromByteBuffer(type, upperBytes))) {
Review Comment:
I don't think `max == min` is a good enough condition to condition our
behavior on. It will be false more often than not since bucket transform does
not group by identical values.
Another option is to assume safety (note: this is what Spark does), and add
a stricter "validate" option that would iterate through the whole
bucket-partitioned column to verify.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
