Karman580 opened a new pull request, #19211:
URL: https://github.com/apache/iotdb/pull/19211
Fixes #19210
## Summary
Subtracting natural months in a date expression lands on the wrong day.
`<time> - Nmo` computed the length of the N months after `<time>` and
subtracted it, so the result depended on how long the following month is:
- `2024-03-15 - 1mo` gave `2024-02-13`.
- `2024-03-31 - 1mo` gave `2024-03-01`.
- `2024-02-15 + 1mo - 1mo` gave `2024-02-13`.
With this change, `- Nmo` goes back N calendar months from the original
time, the same way `+ Nmo` already goes forward: it keeps the time of day in
the session time zone and clamps to the end of the month.
## Root cause
All three date-expression parsers do the same thing:
- `ASTVisitor.parseDateExpression(ctx, String precision)` and
`ASTVisitor.parseDateExpression(ctx, long currentTime)` (tree)
- `AstBuilder.parseDateExpression` (table)
They call `DataNodeDateTimeUtils.convertDurationStrToLong(time, duration,
false)` and then apply `Math.subtractExact(time, duration)` for `-`. For `mo`,
that conversion runs `calendar.add(Calendar.MONTH, +N)` from `time`. It always
measures forward, even when the operator is `-`.
## Changes
- `DataNodeDateTimeUtils`: added `convertDurationStrToLong(..., boolean
negative)` overloads. When `negative` is true, each unit in the duration string
is converted with a negated value, so the month step becomes `Calendar.MONTH,
-N` from the original time, and the returned offset is negative. The existing
overloads delegate with `negative = false`, so GROUP BY, CQ, the `currentTime
== -1` path and every other caller behave exactly as before.
- The three parsers pass `"-".equals(op)` and then do `Math.addExact(time,
offset)`. Addition produces the same values as before.
- Mixed durations are still applied left to right (`2024-03-31 - 1mo1d` =
`2024-02-29 - 1d` = `2024-02-28`). Fixed-length units (`d`, `w`, `h`, ...) give
the same result as before, and the precision conversion for `ms`/`us`/`ns`
units is symmetric for negative values.
Before and after, standalone server, `timestamp_precision=ms`, server and
CLI in UTC. The tree and table models returned the same rows in both builds:
| Query | Before | After |
|---|---|---|
| `time <= 2024-03-15T00:00:00+00:00 - 1mo` | 2024-02-12, 02-13 |
2024-02-12, 02-13, 02-14, 02-15 |
| `time >= 2024-03-31T00:00:00+00:00 - 1mo` | 2024-03-01 | 2024-02-29, 03-01
|
| `time >= 2024-02-15T00:00:00+00:00 + 1mo - 1mo AND time <=
2024-02-20T00:00:00+00:00` | 2024-02-13 ... 02-16 | 2024-02-15, 02-16 |
Output after the fix (tree):
```
>>> SELECT s1 FROM root.test.d1 WHERE time <= 2024-03-15T00:00:00+00:00 - 1mo
|2024-02-12T00:00:00.000Z| 12|
|2024-02-13T00:00:00.000Z| 13|
|2024-02-14T00:00:00.000Z| 14|
|2024-02-15T00:00:00.000Z| 15|
Total line number = 4
>>> SELECT s1 FROM root.test.d1 WHERE time >= 2024-03-31T00:00:00+00:00 - 1mo
|2024-02-29T00:00:00.000Z| 29|
|2024-03-01T00:00:00.000Z| 301|
Total line number = 2
```
## Tests
New tests in `DateTimeUtilsTest`. Each case is parsed by the real tree
parser (`DataNodeDateTimeUtils.parseDateTimeExpressionToLong`) and the real
table parser (`SqlParser.createExpression`) with a registered session, and the
resulting timestamp is checked:
- `dateExpressionSubtractNaturalMonthTest` (UTC) covers:
- `-1mo` / `-2mo` across different month lengths
- month-end clamping: `2024-03-31`, `2025-03-31`, `2024-05-31`
- mixed durations: `1mo1d` and `1w2d` (tree only, because the table
grammar has no compound durations), plus `- 1mo - 1d`
- a pre-1970 timestamp
- the `+ 1mo - 1mo` round trip
- unchanged `+ 1mo` cases
- `dateExpressionSubtractNaturalMonthWithTimeZoneTest`: `America/New_York`
across the DST start, and `Asia/Kolkata`.
Both new tests fail on current master (for example, `tree:
2024-03-15T00:00:00+00:00 - 1mo expected:<1707955200000> but
was:<1707782400000>`) and pass with this change.
Local runs (JDK 25, macOS):
```
TZ=UTC ./mvnw test -pl iotdb-core/datanode -Dtest=DateTimeUtilsTest
Tests run: 15, Failures: 0, Errors: 0, Skipped: 1
TZ=UTC ./mvnw test -pl iotdb-core/datanode
-Dsurefire.failIfNoSpecifiedTests=false \
-Dtest='*Parser*Test,*ASTVisitor*Test,*StatementGenerator*Test,*AstBuilder*Test,*DateTime*Test,*TimeRange*Test,*GroupBy*Test,*Fill*Test,*Analyze*Test,*Predicate*Test,*Statement*Test,*TimePredicate*Test,*Filter*Test,*Precision*Test'
Tests run: 522, Failures: 0, Errors: 0, Skipped: 5
./mvnw spotless:check -pl iotdb-core/datanode
-> BUILD SUCCESS
./mvnw test-compile -pl iotdb-core/datanode -am -DskipTests
-> BUILD SUCCESS
./mvnw test-compile -P with-zh-locale -pl iotdb-core/datanode -am
-DskipTests -> BUILD SUCCESS
```
The skipped test is the existing `@Ignore`d
`getConvertDurationIncludingMonthUnit`. Note that
`convertDatetimeStrToLongTest1` in the same class fails in a `+05:30` local
time zone on master as well, independent of this change, so the runs above use
`TZ=UTC`.
## Known limitation
With `timestamp_precision=us` or `ns`, `mo` arithmetic is still wrong in
both directions. In those precisions the precision-scaled timestamp is passed
to `Calendar.setTimeInMillis`, which is a separate problem and also affects `+
1mo`. This PR doesn't address it; I'll report it separately.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for code coverage.
- [x] been tested in a test IoTDB cluster (standalone, tree and table SQL).
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `DataNodeDateTimeUtils`
- `ASTVisitor`
- `AstBuilder`
- `DateTimeUtilsTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]