yyanyy commented on a change in pull request #1981:
URL: https://github.com/apache/iceberg/pull/1981#discussion_r548345460
##########
File path: api/src/main/java/org/apache/iceberg/transforms/Timestamps.java
##########
@@ -52,12 +52,19 @@ public Integer apply(Long timestampMicros) {
return null;
}
- // discards fractional seconds, not needed for calculation
- OffsetDateTime timestamp = Instant
- .ofEpochSecond(timestampMicros / 1_000_000)
- .atOffset(ZoneOffset.UTC);
-
- return (int) granularity.between(EPOCH, timestamp);
+ if (timestampMicros >= 0) {
+ OffsetDateTime timestamp = Instant
+ .ofEpochSecond(Math.floorDiv(timestampMicros, 1_000_000),
Math.floorMod(timestampMicros, 1_000_000))
+ .atOffset(ZoneOffset.UTC);
+ return (int) granularity.between(EPOCH, timestamp);
+ } else {
+ // add 1 micro to the value to account for the case where there is
exactly 1 unit between the timestamp and epoch
+ // because the result will always be decremented.
+ OffsetDateTime timestamp = Instant
+ .ofEpochSecond(Math.floorDiv(timestampMicros, 1_000_000),
Math.floorMod(timestampMicros + 1, 1_000_000))
Review comment:
I think the second argument in `Instant.ofEpochSecond` is nanosecond
instead of microsecond? Although this doesn't change the outcome of this method
since we don't accept nanosecond in the first place, and inputing microsecond
for nanosecond field just makes the field smaller than it should be, which is
already enough to address the case when `Math.floorMod()` == 0
##########
File path:
api/src/test/java/org/apache/iceberg/transforms/TestDatesProjection.java
##########
@@ -142,6 +161,24 @@ public void testMonthStrictUpperBound() {
assertProjectionStrictValue(spec, in("date", anotherDate, date),
Expression.Operation.FALSE);
}
+ @Test
+ public void testNegativeMonthStrictUpperBound() {
+ Integer date = (Integer) Literal.of("1969-12-31").to(TYPE).value();
+ PartitionSpec spec =
PartitionSpec.builderFor(SCHEMA).month("date").build();
+
+ assertProjectionStrictValue(spec, lessThan("date", date),
Expression.Operation.FALSE);
+ assertProjectionStrictValue(spec, lessThanOrEqual("date", date),
Expression.Operation.FALSE);
+ assertProjectionStrict(spec, greaterThan("date", date),
Expression.Operation.GT, "1969-12");
+ assertProjectionStrict(spec, greaterThanOrEqual("date", date),
Expression.Operation.GT, "1969-12");
+ assertProjectionStrict(spec, notEqual("date", date),
Expression.Operation.NOT_IN, "[1969-12, 1970-01]");
+ assertProjectionStrictValue(spec, equal("date", date),
Expression.Operation.FALSE);
+
+ Integer anotherDate = (Integer) Literal.of("1970-01-01").to(TYPE).value();
+ assertProjectionStrict(spec, notIn("date", date, anotherDate),
+ Expression.Operation.NOT_IN, "[1969-12, 1970-01]");
Review comment:
I think this is testing the same thing for `NOT_IN` in
`testNegativeMonthStrictLowerBound`, do we want to change 1970-01-01 to an
earlier time?
##########
File path: api/src/main/java/org/apache/iceberg/transforms/ProjectionUtil.java
##########
@@ -254,4 +257,124 @@ private ProjectionUtil() {
return predicate(predicate.op(), fieldName,
Iterables.transform(predicate.asSetPredicate().literalSet(),
transform::apply));
}
+
+ static UnboundPredicate<Integer>
fixInclusiveTimeProjection(UnboundPredicate<Integer> projected) {
+ if (projected == null) {
+ return projected;
+ }
+
+ // adjust the predicate for values that were 1 larger than the correct
transformed value
Review comment:
I think the mention of "correct" and "incorrect" in these comments might
not be obvious to understand outside the context of this PR, as the reader has
no prior knowledge of the issue we are fixing here from looking at the code
base itself. Do we want to add some more explanation/link to the issue?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
