rdblue commented on code in PR #18604:
URL: https://github.com/apache/iceberg/pull/18604#discussion_r4462002659
##########
core/src/main/java/org/apache/iceberg/StatsUtil.java:
##########
@@ -325,4 +336,194 @@ private static boolean isScalar(Schema schema,
Map<Integer, Integer> parentIndex
return true;
}
+
+ /**
+ * Lower and upper bounds for source values in a partition from an
order-preserving transform.
+ *
+ * <p>Every source value that transforms to the partition is >= lower and <=
upper. Upper is null
+ * when unbounded. Both bounds are inclusive.
+ *
+ * <p>For most transforms, lower and upper are the values in the partition.
Exceptions:
+ *
+ * <ul>
+ * <li>For string and binary truncate, upper is the next prefix when it
exists, so upper can
+ * truncate to the next partition. That still covers every value that
truncates to the
+ * partition.
+ * <li>For time transforms, lower starts one unit earlier when the
partition value is <= 1970 to
+ * handle older writers that rounded pre-1970 values toward zero.
+ * </ul>
+ */
+ record PartitionSourceBounds<T>(T lower, T upper) {}
+
+ /**
+ * Returns the source column bounds for a partition value, or null for a NaN
identity value.
+ *
+ * @throws IllegalArgumentException if the transform does not preserve order
or the value is null
+ */
+ static PartitionSourceBounds<?> sourceBoundsFor(
+ Schema schema, PartitionField field, Object partitionValue) {
+ Preconditions.checkArgument(
+ field.transform().preservesOrder(),
+ "Cannot derive source column bounds from partition field that does not
preserve order: %s",
+ field);
+ Preconditions.checkArgument(
+ partitionValue != null, "Cannot derive source column bounds from null
partition value");
+ Type sourceType = schema.findType(field.sourceId());
+ Preconditions.checkArgument(
+ sourceType != null, "Cannot find source column for partition field:
%s", field);
+ return PartitionSpecVisitor.visit(
+ schema, field, new PartitionSourceBoundsVisitor(sourceType,
partitionValue));
+ }
+
+ private static class PartitionSourceBoundsVisitor
+ implements PartitionSpecVisitor<PartitionSourceBounds<?>> {
+ private final Type sourceType;
+ private final Object value;
+
+ private PartitionSourceBoundsVisitor(Type sourceType, Object value) {
+ this.sourceType = sourceType;
+ this.value = value;
+ }
+
+ @Override
+ public PartitionSourceBounds<?> identity(int fieldId, String sourceName,
int sourceId) {
+ // NaN cannot be used as a bound
+ if (NaNUtil.isNaN(value)) {
+ return null;
+ }
+
+ return new PartitionSourceBounds<>(
+ value, value instanceof ByteBuffer buffer ? buffer.duplicate() :
value);
Review Comment:
If we are duplicating buffers (which I'm not sure is even needed) then I
think we should duplicate them both. Either way, there should eventually be a
test for writing whatever this is into `FieldStats`. All of our writers should
not modify the data they are passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
