Karman580 opened a new pull request, #19211:
URL: https://github.com/apache/iotdb/pull/19211

   Fixes #19210
   
   ## Summary
   
   Subtracting natural months in a date expression lands on the wrong day. 
`<time> - Nmo` computed the length of the N months after `<time>` and 
subtracted it, so the result depended on how long the following month is:
   
   - `2024-03-15 - 1mo` gave `2024-02-13`.
   - `2024-03-31 - 1mo` gave `2024-03-01`.
   - `2024-02-15 + 1mo - 1mo` gave `2024-02-13`.
   
   With this change, `- Nmo` goes back N calendar months from the original 
time, the same way `+ Nmo` already goes forward: it keeps the time of day in 
the session time zone and clamps to the end of the month.
   
   ## Root cause
   
   All three date-expression parsers do the same thing:
   
   - `ASTVisitor.parseDateExpression(ctx, String precision)` and 
`ASTVisitor.parseDateExpression(ctx, long currentTime)` (tree)
   - `AstBuilder.parseDateExpression` (table)
   
   They call `DataNodeDateTimeUtils.convertDurationStrToLong(time, duration, 
false)` and then apply `Math.subtractExact(time, duration)` for `-`. For `mo`, 
that conversion runs `calendar.add(Calendar.MONTH, +N)` from `time`. It always 
measures forward, even when the operator is `-`.
   
   ## Changes
   
   - `DataNodeDateTimeUtils`: added `convertDurationStrToLong(..., boolean 
negative)` overloads. When `negative` is true, each unit in the duration string 
is converted with a negated value, so the month step becomes `Calendar.MONTH, 
-N` from the original time, and the returned offset is negative. The existing 
overloads delegate with `negative = false`, so GROUP BY, CQ, the `currentTime 
== -1` path and every other caller behave exactly as before.
   - The three parsers pass `"-".equals(op)` and then do `Math.addExact(time, 
offset)`. Addition produces the same values as before.
   - Mixed durations are still applied left to right (`2024-03-31 - 1mo1d` = 
`2024-02-29 - 1d` = `2024-02-28`). Fixed-length units (`d`, `w`, `h`, ...) give 
the same result as before, and the precision conversion for `ms`/`us`/`ns` 
units is symmetric for negative values.
   
   Before and after, standalone server, `timestamp_precision=ms`, server and 
CLI in UTC. The tree and table models returned the same rows in both builds:
   
   | Query | Before | After |
   |---|---|---|
   | `time <= 2024-03-15T00:00:00+00:00 - 1mo` | 2024-02-12, 02-13 | 
2024-02-12, 02-13, 02-14, 02-15 |
   | `time >= 2024-03-31T00:00:00+00:00 - 1mo` | 2024-03-01 | 2024-02-29, 03-01 
|
   | `time >= 2024-02-15T00:00:00+00:00 + 1mo - 1mo AND time <= 
2024-02-20T00:00:00+00:00` | 2024-02-13 ... 02-16 | 2024-02-15, 02-16 |
   
   Output after the fix (tree):
   
   ```
   >>> SELECT s1 FROM root.test.d1 WHERE time <= 2024-03-15T00:00:00+00:00 - 1mo
   |2024-02-12T00:00:00.000Z|             12|
   |2024-02-13T00:00:00.000Z|             13|
   |2024-02-14T00:00:00.000Z|             14|
   |2024-02-15T00:00:00.000Z|             15|
   Total line number = 4
   >>> SELECT s1 FROM root.test.d1 WHERE time >= 2024-03-31T00:00:00+00:00 - 1mo
   |2024-02-29T00:00:00.000Z|             29|
   |2024-03-01T00:00:00.000Z|            301|
   Total line number = 2
   ```
   
   ## Tests
   
   New tests in `DateTimeUtilsTest`. Each case is parsed by the real tree 
parser (`DataNodeDateTimeUtils.parseDateTimeExpressionToLong`) and the real 
table parser (`SqlParser.createExpression`) with a registered session, and the 
resulting timestamp is checked:
   
   - `dateExpressionSubtractNaturalMonthTest` (UTC) covers:
     - `-1mo` / `-2mo` across different month lengths
     - month-end clamping: `2024-03-31`, `2025-03-31`, `2024-05-31`
     - mixed durations: `1mo1d` and `1w2d` (tree only, because the table 
grammar has no compound durations), plus `- 1mo - 1d`
     - a pre-1970 timestamp
     - the `+ 1mo - 1mo` round trip
     - unchanged `+ 1mo` cases
   - `dateExpressionSubtractNaturalMonthWithTimeZoneTest`: `America/New_York` 
across the DST start, and `Asia/Kolkata`.
   
   Both new tests fail on current master (for example, `tree: 
2024-03-15T00:00:00+00:00 - 1mo expected:<1707955200000> but 
was:<1707782400000>`) and pass with this change.
   
   Local runs (JDK 25, macOS):
   
   ```
   TZ=UTC ./mvnw test -pl iotdb-core/datanode -Dtest=DateTimeUtilsTest
   Tests run: 15, Failures: 0, Errors: 0, Skipped: 1
   
   TZ=UTC ./mvnw test -pl iotdb-core/datanode 
-Dsurefire.failIfNoSpecifiedTests=false \
     
-Dtest='*Parser*Test,*ASTVisitor*Test,*StatementGenerator*Test,*AstBuilder*Test,*DateTime*Test,*TimeRange*Test,*GroupBy*Test,*Fill*Test,*Analyze*Test,*Predicate*Test,*Statement*Test,*TimePredicate*Test,*Filter*Test,*Precision*Test'
   Tests run: 522, Failures: 0, Errors: 0, Skipped: 5
   
   ./mvnw spotless:check -pl iotdb-core/datanode                              
-> BUILD SUCCESS
   ./mvnw test-compile -pl iotdb-core/datanode -am -DskipTests                
-> BUILD SUCCESS
   ./mvnw test-compile -P with-zh-locale -pl iotdb-core/datanode -am 
-DskipTests -> BUILD SUCCESS
   ```
   
   The skipped test is the existing `@Ignore`d 
`getConvertDurationIncludingMonthUnit`. Note that 
`convertDatetimeStrToLongTest1` in the same class fails in a `+05:30` local 
time zone on master as well, independent of this change, so the runs above use 
`TZ=UTC`.
   
   ## Known limitation
   
   With `timestamp_precision=us` or `ns`, `mo` arithmetic is still wrong in 
both directions. In those precisions the precision-scaled timestamp is passed 
to `Calendar.setTimeInMillis`, which is a separate problem and also affects `+ 
1mo`. This PR doesn't address it; I'll report it separately.
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests or modified existing tests to cover new code paths, 
ensuring the threshold for code coverage.
   - [x] been tested in a test IoTDB cluster (standalone, tree and table SQL).
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   - `DataNodeDateTimeUtils`
   - `ASTVisitor`
   - `AstBuilder`
   - `DateTimeUtilsTest`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to