JackieTien97 commented on code in PR #18589:
URL: https://github.com/apache/iotdb/pull/18589#discussion_r3963872089
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/fill/filter/FixedIntervalFillFilter.java:
##########
@@ -34,6 +36,7 @@ public FixedIntervalFillFilter(long timeInterval) {
public boolean needFill(long time, long previousTime) {
// the reason that we use Math.abs is that we may use order by time desc
which will cause
// previousTime is larger than time
- return Math.abs(time - previousTime) <= timeInterval;
+ return timeDifferenceAsDouble(time, previousTime) <= timeInterval
+ && timeDifferenceAsDouble(previousTime, time) <= timeInterval;
Review Comment:
**[P2] Keep the fill interval comparison exact**
Using a double for this integer threshold comparison accepts gaps that
exceed the configured interval once the distance is above `2^53`. For example,
at nanosecond precision, set `timeInterval=15552000000000000L` (180 days),
`previousTime=1700000000000000000L`, and `time=previousTime+timeInterval+1`.
`needFill` now returns `true`, whereas the previous implementation returns
`false`: the 180-day-plus-1-ns gap rounds down to the same double as the
threshold. This fills a value outside the requested range. Please compare the
integer distance exactly with overflow handling; the unsigned-distance
comparison added in `DownSamplingTimeUtils` is one applicable approach.
##########
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/udf/builtin/UDTFEqualSizeBucketOutlierSample.java:
##########
@@ -354,14 +356,12 @@ public void outlierSampleLong(RowWindow rowWindow,
PointCollector collector)
currentValue = rowWindow.getRow(i).getLong(0);
nextValue = rowWindow.getRow(i + 1).getLong(0);
- x1 = currentTime - lastTime;
- x2 = nextTime - currentTime;
- y1 = currentValue - lastValue;
- y2 = nextValue - currentValue;
+ x1 = timeDifferenceAsDouble(currentTime, lastTime);
+ x2 = timeDifferenceAsDouble(nextTime, currentTime);
+ y1 = (double) currentValue - (double) lastValue;
+ y2 = (double) nextValue - (double) currentValue;
Review Comment:
**[P2] Preserve the integer difference before converting INT64 values to
double**
Casting each operand before subtraction loses small differences between
large INT64 values. With timestamps `[0, 1, 2, 3, 4]`, values `B + [0, 1, 10,
1, 0]` where `B = 1L << 60`, and `number=1, proportion=0.2`, both `cos` and
`prenextdis` select timestamp `1` on this commit instead of the unique central
spike at timestamp `2`. All five values round to the same double, so the value
differences become zero. The previous implementations select timestamp `2` for
both modes. Please use an overflow-safe integer difference before converting to
double, as `timeDifferenceAsDouble` already does for timestamps, and apply the
same correction to the INT64 `prenextdis` path.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/DateTimeUtils.java:
##########
@@ -591,32 +592,40 @@ public static long convertDurationStrToLong(
*/
public static long convertDurationStrToLong(
long currentTime, String duration, String timestampPrecision, boolean
convertYearToMonth) {
- long total = 0;
- long temp = 0;
- for (int i = 0; i < duration.length(); i++) {
- char ch = duration.charAt(i);
- if (Character.isDigit(ch)) {
- temp *= 10;
- temp += (ch - '0');
- } else {
- String unit = String.valueOf(duration.charAt(i));
- // This is to identify units with two letters.
- if (i + 1 < duration.length() && !Character.isDigit(duration.charAt(i
+ 1))) {
- i++;
- unit += duration.charAt(i);
- }
- unit = unit.toLowerCase();
- if (convertYearToMonth && unit.equals("y")) {
- temp *= 12;
- unit = "mo";
+ try {
+ long total = 0;
+ long temp = 0;
+ for (int i = 0; i < duration.length(); i++) {
+ char ch = duration.charAt(i);
+ if (Character.isDigit(ch)) {
+ temp = Math.addExact(Math.multiplyExact(temp, 10),
Character.digit(ch, 10));
+ } else {
+ String unit = String.valueOf(duration.charAt(i));
+ // This is to identify units with two letters.
+ if (i + 1 < duration.length() &&
!Character.isDigit(duration.charAt(i + 1))) {
+ i++;
+ unit += duration.charAt(i);
+ }
+ unit = unit.toLowerCase();
+ if (convertYearToMonth && unit.equals("y")) {
+ temp = Math.multiplyExact(temp, 12);
+ unit = "mo";
+ }
+ total =
+ Math.addExact(
+ total,
+ DateTimeUtils.convertDurationStrToLong(
+ currentTime == -1 ? -1 : Math.addExact(currentTime,
total),
Review Comment:
**[P2] Do not reject a valid subtraction because an unused addition
overflows**
This method is also called when parsing date subtraction, but every
component of a duration now checks `currentTime + total`. At nanosecond
precision with UTC, `select s1 from root.sg.d1 where time >
2262-04-11T23:47:16.854775806 - 1s1ns` throws `Time duration is out of range.`
when parsing the second component. The final timestamp is the valid value
`9223372035854775805`, and the equivalent expression using `- 1000000001ns`
succeeds. The previous implementation accepts both. For fixed-duration units
this reference timestamp is not needed; please avoid the unconditional addition
and let the caller check the final addition/subtraction, while handling any
calendar-dependent reference calculation according to the actual operation.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/process/ai/InferenceOperator.java:
##########
@@ -374,4 +378,27 @@ public long ramBytesUsed() {
? 0
:
targetColumnNames.stream().mapToLong(RamUsageEstimator::sizeOf).sum());
}
+
+ static long calculateGeneratedTimeInterval(long minTimestamp, long
maxTimestamp, long totalRow) {
+ try {
+ BigInteger interval =
+ BigInteger.valueOf(maxTimestamp)
+ .subtract(BigInteger.valueOf(minTimestamp))
+ .divide(BigInteger.valueOf(totalRow));
+ if (interval.compareTo(BigInteger.valueOf(Long.MAX_VALUE)) > 0) {
+ throw new ArithmeticException();
+ }
+ return interval.longValue();
+ } catch (ArithmeticException e) {
+ throw new
ModelInferenceProcessException(GENERATED_TIME_COLUMN_OUT_OF_RANGE_MESSAGE);
+ }
+ }
+
+ static long calculateGeneratedTime(long maxTimestamp, long interval, long
currentRowIndex) {
+ try {
+ return Math.addExact(maxTimestamp, Math.multiplyExact(interval,
currentRowIndex));
Review Comment:
**[P2] Check the complete generated timestamp rather than the intermediate
product**
A negative `maxTimestamp` can bring an intermediate product larger than
`Long.MAX_VALUE` back into the valid timestamp range. For two input timestamps
`-6000000000000000000L` and `-4000000000000000000L`, the interval calculation
yields `1000000000000000000L`. If the model returns 11 rows, index `10` should
receive timestamp `6000000000000000000L`, and every generated timestamp is
representable. The new `multiplyExact` instead throws before adding the
negative maximum timestamp, causing `fillTimeColumn` to fail with `Generated
time column is out of range.`; the previous implementation completes this case.
Please evaluate the entire multiply-and-add expression with `BigInteger` and
range-check its final result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]