CoollZzz opened a new pull request, #18334: URL: https://github.com/apache/iotdb/pull/18334
## Description Related issue: #17976 This PR adds four built-in aggregation functions to the IoTDB table model: `rate()`, `increase()`, `irate()`, and `delta()`. Their calculation semantics are based on the corresponding Prometheus range-vector functions, while window boundaries follow IoTDB's `[window_start, window_end)` convention. ### Function semantics | Function | Signature | Description | | --- | --- | --- | | `rate` | `rate(value, time, window_start, window_end)` | Calculates the extrapolated average per-second growth rate of a counter and compensates for counter resets. | | `increase` | `increase(value, time, window_start, window_end)` | Calculates the extrapolated total increase of a counter and compensates for counter resets. | | `irate` | `irate(value, time)` | Calculates the instantaneous per-second counter rate using the last two valid samples. | | `delta` | `delta(value, time, window_start, window_end)` | Calculates the extrapolated difference of a gauge without counter-reset compensation. | The supported value types are `INT32`, `INT64`, `FLOAT`, and `DOUBLE`. Time and window arguments accept `TIMESTAMP` and `INT64`, with `INT64` interpreted using the current `timestamp_precision`. All four functions return `DOUBLE`. `rate()`, `increase()`, and `delta()` use all valid samples in the aggregation group and extrapolate the result to the window boundaries. Extrapolation uses the average sampling interval, a `1.1` threshold for distant boundaries, and counter zero-point protection where applicable. `irate()` only uses the last two valid samples and does not perform boundary extrapolation. ### Accumulator and planner design Both `TableAccumulator` and `GroupedAccumulator` execution paths are supported. Each function has its own abstract accumulator class for shared function-specific validation, window handling, and result calculation. Ordered and naive accumulators are implemented as separate concrete classes because their state structures and lifecycle behavior differ significantly. - Ordered accumulators are selected only for the `SINGLE` aggregation step when the physical planner proves that the function's `time` argument is ascending within every SQL aggregation group. They consume samples incrementally without buffering and sorting all input rows. - Naive accumulators buffer samples and sort them by time before evaluation. They support unordered input and all multi-stage aggregation steps. - Multi-stage aggregation uses a versioned `BLOB` intermediate state containing the complete sample state and window boundaries. - Naive table and grouped accumulators report retained buffer memory through the existing memory reservation framework. - The four functions are not pushed into aggregation table scans because their explicit time argument may differ from the physical table time column and their calculations require raw samples. The ordered-input property is propagated through `AggregationNode` serialization so that distributed execution selects the same accumulator implementation as the coordinator plan. ### Validation and edge cases The implementation handles the following cases: - `NULL` values are ignored. - Fewer than two valid samples return `NULL`. - Counter resets are compensated for `rate()`, `increase()`, and `irate()`. - Negative values are rejected for counter functions but are supported by `delta()`. - Non-finite values, including `NaN` and positive or negative infinity, are rejected. - Required time or window arguments cannot be `NULL` for a valid sample. - Window boundaries must be consistent within an aggregation group and satisfy `window_start < window_end`. - Every valid sample must satisfy `window_start <= time < window_end`. - Duplicate timestamps within the same aggregation group are rejected. - Non-finite intermediate or extrapolated results are rejected. ### Tests Integration tests were added to `IoTDBTableAggregationIT` for: - Normal global, grouped, ordered, unordered, distributed, and windowed aggregation. - Counter resets, multiple resets, zero-point protection, extrapolation thresholds, null values, missing samples, and gap-fill scenarios. - All supported numeric and time argument types. - Invalid argument counts and types, non-finite and negative counter values, invalid or inconsistent windows, out-of-window samples, null time arguments, and duplicate timestamps. The four target test methods were verified with both the `TableSimpleIT` and `TableClusterIT` profiles. The full reactor test compilation also passed for both the default and Chinese locales. <hr> This PR has: - [x] been self-reviewed. - [x] added integration tests. - [x] been tested in a test IoTDB cluster. <hr> ##### Key changed/added classes (or packages if there are too many classes) in this PR - `org.apache.iotdb.calc.execution.operator.source.relational.aggregation.rate` - `org.apache.iotdb.calc.execution.operator.source.relational.aggregation.grouped.rate` - `AccumulatorFactory` - `TableMetadataImpl` - `TableDistributedPlanGenerator` - `AggregationNode` - `PushAggregationIntoTableScan` - `IoTDBTableAggregationIT` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
