DaZuiZui commented on issue #18428:
URL: https://github.com/apache/iotdb/issues/18428#issuecomment-5235636695
## Proposed functional definition and implementation direction
After reviewing the latest master, I think calendar duration must be treated
as first-class CQ metadata throughout parsing, RPC, persistence, recovery, and
scheduling. Changing only parseResampleClause or converting a month against the
creation time would still drift later.
### 1. User-visible syntax and supported values
The existing CQ syntax remains unchanged:
CREATE [CONTINUOUS QUERY | CQ] <cq_id>
[RESAMPLE
[EVERY <duration>]
[BOUNDARY <time_value>]
[RANGE <start_offset> [, <end_offset>]]
]
[TIMEOUT POLICY {BLOCKED | DISCARD}]
BEGIN
<select_into_statement>
END
Proposed duration units:
- Calendar units: y/year and mo/month
- Existing fixed units: w, d, h, m, s, ms, us, ns
- Compound values remain supported, for example 1y2mo3d
- 1y is normalized to 12 calendar months
- d and w remain fixed 24-hour/7-day durations; only mo/y have calendar
semantics
- This feature does not change the data types, aggregations, or SELECT INTO
rules supported by the CQ query body
Logically, a duration is represented by an integer calendar-month part plus
a fixed-tick part in the configured timestamp precision.
If EVERY is omitted, it inherits the complete GROUP BY TIME duration.
Therefore, GROUP BY(1mo) must yield a calendar-month EVERY, not 30 days. If
RANGE is omitted, its defaults remain startOffset = EVERY and endOffset = 0.
### 2. Calendar scheduling semantics
All calendar arithmetic must use the session zone captured when the CQ is
created and persisted with the CQ.
For boundary B, interval E, CQ zone Z, and occurrence index n:
executionTime(n) = calendarApply(B, n * E, Z)
Each occurrence must be calculated from the original boundary. It must not
repeatedly call previousExecutionTime.plusMonths(1), because that would drift:
Wrong: Jan-31 -> Feb-29 -> Mar-29
Correct: Jan-31 -> Feb-29 -> Mar-31
Month-end uses standard last-valid-day clamping:
2023-01-31 + 1mo = 2023-02-28
2024-01-31 + 1mo = 2024-02-29
2020-02-29 + 1y = 2021-02-28
2020-02-29 + 4y = 2024-02-29
To preserve the currently documented boundary formula and avoid
non-invertible month subtraction, I recommend calculating RANGE endpoints in
duration-vector space:
startTime(n) = calendarApply(B, n * E - startOffset, Z)
endTime(n) = calendarApply(B, n * E - endOffset, Z)
This makes EVERY 1mo RANGE 1mo produce contiguous windows even for a Jan-31
or Feb-29 anchor.
For timeout policies:
- BLOCKED executes every occurrence in order, even when late. Its query
window is based on the scheduled occurrence time, not the actual wall-clock
start time.
- DISCARD skips missed occurrences and jumps directly to the first valid
occurrence not earlier than the current time.
- Calendar lookup should use an estimate plus correction or binary search,
not a linear loop from 1970.
### 3. Validation and compatibility
The existing positive-value rules remain: EVERY > 0, startOffset > 0,
endOffset >= 0, startOffset > endOffset, and the current startOffset >= EVERY
constraint.
Calendar and fixed durations do not have a total ordering: 1mo may be
shorter or longer than 30d. Such comparisons must not flatten months to 30
days. Ambiguous mixed comparisons should either use conservative min/max bounds
or be rejected with a clear semantic error.
For compatibility, I suggest:
- Add optional structured duration fields to TCreateCQReq; keep the existing
i64 fields for legacy readers
- New ConfigNodes prefer the structured values; old requests become
fixed-only durations
- Version the CQInfo snapshot format and load old snapshots as
fixed-duration CQs
- Do not silently migrate existing persisted 1mo/1y CQs by reparsing their
saved SQL; users can recreate them to opt into calendar semantics
- Enable creation of calendar CQs only after all ConfigNodes are upgraded
The DataNode execution RPC can continue receiving concrete
startTime/endTime; the main execution engine does not need calendar-duration
awareness. Its timeout should use the actual distance to the next occurrence
rather than a 30-day approximation.
### 4. Two semantics to confirm before implementation
1. When a calendar EVERY omits BOUNDARY, should it align to local calendar
boundaries? My recommendation is to use 1970-01-01 00:00:00 in the persisted CQ
zone, while an explicit BOUNDARY 0 continues to mean the Unix epoch instant.
This requires preserving whether BOUNDARY was explicitly specified.
2. Can RANGE use the anchored vector formula above? It preserves the
existing documented formula and makes EVERY == RANGE contiguous at month-end,
while direct subtraction from a clamped execution timestamp does not.
Suggested tests should cover explicit and inherited 1mo/1y, Jan-31, leap
day, DST zones, ms/us/ns precision, BLOCKED/DISCARD catch-up, leader recovery,
procedure/plan serialization, and legacy CQ snapshots.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
