JackieTien97 commented on code in PR #19267:
URL: https://github.com/apache/iotdb/pull/19267#discussion_r4581307365
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/unary/scalar/DateBinFunctionColumnTransformer.java:
##########
@@ -90,19 +90,19 @@ private static LocalDateTime convertToLocalDateTime(long
timestamp, ZoneId zoneI
}
private static long convertToTimestamp(LocalDateTime dateTime, ZoneId
zoneId) {
- // Converting LocalDateTime to Seconds since epoch
- long epochMilliSecond = dateTime.atZone(zoneId).toInstant().toEpochMilli();
- // Get the nanoseconds section
- long nanoAdjustment = dateTime.getNano();
+ // getEpochSecond() gives the whole seconds and getNano() the nanos inside
the second, so the
+ // milliseconds inside the second are counted exactly once
+ Instant instant = dateTime.atZone(zoneId).toInstant();
+ long epochSecond = instant.getEpochSecond();
+ long nanoAdjustment = instant.getNano();
switch
(CommonDescriptor.getInstance().getConfig().getTimestampPrecision()) {
case "ms":
- return epochMilliSecond + nanoAdjustment / NANOSECONDS_IN_MILLISECOND;
+ return TimeUnit.SECONDS.toMillis(epochSecond) + nanoAdjustment /
NANOSECONDS_IN_MILLISECOND;
case "us":
- return TimeUnit.MILLISECONDS.toMicros(epochMilliSecond)
- + nanoAdjustment / NANOSECONDS_IN_MICROSECOND;
+ return TimeUnit.SECONDS.toMicros(epochSecond) + nanoAdjustment /
NANOSECONDS_IN_MICROSECOND;
case "ns":
- return TimeUnit.MILLISECONDS.toNanos(epochMilliSecond) +
nanoAdjustment;
+ return TimeUnit.SECONDS.toNanos(epochSecond) + nanoAdjustment;
Review Comment:
Edge case in the rewritten lines:
`TimeUnit.SECONDS.toMillis/toMicros/toNanos` saturates at `Long.MAX_VALUE`, but
the `+ nanoAdjustment / ...` after it is an unchecked add. So when a bin
boundary lies past the end of the `long` range and the origin has a sub-second
part, the result wraps to a large negative value.
A probe test on this branch (UTC, `1mo`) shows:
- `ms`, origin `500`: `dateBinStartEnd(Long.MAX_VALUE - 1000)` returns
`[9223372035446400500, -9223372036854775309]`, and `nextDateBin(1, UTC,
9223372035446400500)` returns `-9223372036854775309`. Before this PR,
`toEpochMilli()` threw `ArithmeticException: long overflow` for the same input.
- `ns`, origin `1500000`: `dateBinStartEnd(Long.MAX_VALUE - 10)` (a time in
2262-04, which `ns` precision accepts by default) returns an end of
`-9223372036853275809`. This one already happened before the PR.
- With a whole-second origin the value only saturates to `Long.MAX_VALUE`.
The gap-fill operators look sensitive to this. From reading the code (not
run): when `writeCurrentRow` calls `nextTime()` and `currentTime` wraps,
`hasRemainingGapInPreviousGroup()` (`currentTime < endTime`) stays true, so
`fillGaps` starts emitting rows from near `Long.MIN_VALUE` up to `endTime`.
`saturatedAdd` is already statically imported here, and it would keep this
path consistent with the saturating behavior from #17893. With the change
below, the probe cases return `Long.MAX_VALUE` as the bin end, and
`DateBinFunctionTest` still passes:
```suggestion
case "ms":
return saturatedAdd(
TimeUnit.SECONDS.toMillis(epochSecond), nanoAdjustment /
NANOSECONDS_IN_MILLISECOND);
case "us":
return saturatedAdd(
TimeUnit.SECONDS.toMicros(epochSecond), nanoAdjustment /
NANOSECONDS_IN_MICROSECOND);
case "ns":
return saturatedAdd(TimeUnit.SECONDS.toNanos(epochSecond),
nanoAdjustment);
```
(If you'd rather keep the old throw-on-overflow behavior for `ms`,
`instant.toEpochMilli()` does that.)
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/transformation/dag/column/unary/scalar/DateBinFunctionTest.java:
##########
@@ -126,4 +133,53 @@ public void testSourceBeforeUnixEpoch() throws
ParseException {
zoneId);
assertEquals(getTimestamp("1969-12-31 00:00:00.000"), result);
}
+
+ @Test
+ public void testMonthIntervalWithSubSecondOrigin() throws ParseException {
Review Comment:
Could you add a pre-1970 case as well? That is where the new code relies on
`getEpochSecond()` rounding down (with `getNano()` always non-negative), and
where the old code was furthest off: `dateBin(-500, -500, 1, 0, zoneId)`
returned `0` under `ms`, and `dateBin(-1500, -1500, 1, 0, zoneId)` returned
`996500` under `us`. Both are correct on this branch, for example:
```java
assertEquals(-500L, dateBin(-500L, -500L, 1, 0, zoneId));
assertEquals(-2678400500L, dateBin(-2678399500L, -500L, 1, 0, zoneId));
setTimestampPrecision("us");
assertEquals(-1500L, dateBin(-1500L, -1500L, 1, 0, zoneId));
```
An IT covering `date_bin`, `date_bin_gapfill` and `GROUP BY date_bin` with a
sub-second origin would be a nice extra, but I don't think it should block this
PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]