JackieTien97 opened a new issue, #17798: URL: https://github.com/apache/iotdb/issues/17798
### Search before asking - [x] I searched in the [issues](https://github.com/apache/iotdb/issues) and found nothing similar. ### Motivation ## Summary The table model (relational SQL dialect) currently supports three fill methods in the `FILL` clause: `PREVIOUS`, `LINEAR`, and `CONSTANT`. It does **not** support **next/backward value fill** — i.e. filling a null with the nearest *later* non-null value along the time axis. This issue proposes adding a `NEXT` fill method to the table model, mirroring the existing `PREVIOUS` fill in syntax, and reusing the forward look-ahead machinery that `LINEAR` fill already implements. This is a good issue to pick up if you want to get familiar with the IoTDB table-model query engine (grammar → analyzer → planner → operator). Comment below to claim it; happy to help with pointers and review. ## Motivation `PREVIOUS` fill carries the last known value *forward* in time. The symmetric and equally common need is to carry the next known value *backward* in time — useful when a reading is only valid up to the moment the next sample arrives, or when you want to back-fill leading gaps that `PREVIOUS` cannot fill (there is no earlier value to carry). `NEXT` fill closes this gap and makes the table model's fill capabilities symmetric. ## Current behavior Only `PREVIOUS` / `LINEAR` / `CONSTANT` are accepted. From the table-model grammar `iotdb-core/relational-grammar/src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4`: ```antlr fillClause : FILL METHOD fillMethod ; fillMethod : LINEAR timeColumnClause? fillGroupClause? #linearFill | PREVIOUS timeBoundClause? timeColumnClause? fillGroupClause? #previousFill | CONSTANT literalExpression #valueFill ; ``` Note: the `NEXT` keyword token **already exists** in this grammar (`NEXT: 'NEXT';`) and is already in the non-reserved keyword list, so no new reserved word is introduced. ## Proposed syntax Mirror `PREVIOUS` exactly — `TIME_BOUND`, `TIME_COLUMN`, and `FILL_GROUP` are all optional and carry the same meaning: ```sql -- basic SELECT time, device_id, temperature FROM tbl FILL METHOD NEXT; -- only fill when the next non-null value is within 2ms of the current row SELECT time, device_id, temperature FROM tbl FILL METHOD NEXT TIME_BOUND 2ms; -- explicit time column (1-based ordinal) and fill-group columns SELECT time, device_id, temperature FROM tbl FILL METHOD NEXT TIME_COLUMN 1 FILL_GROUP 2; ``` New grammar alternative to add: ```antlr | NEXT timeBoundClause? timeColumnClause? fillGroupClause? #nextFill ``` ## Semantics For each null cell, `NEXT` fill replaces it with the value of the nearest **later** (greater timestamp) non-null cell, within the same fill group, ordered by the time column. - **Tail behavior:** rows after the last non-null value have no "next" value and remain null. - **`TIME_BOUND`:** if the next non-null value's timestamp is farther than the bound from the current row, leave the cell null. - **`TIME_COLUMN`:** which timestamp column defines ordering (defaults to the single `TIMESTAMP` column; error if it cannot be inferred — same rule as `PREVIOUS`/`LINEAR`). - **`FILL_GROUP`:** fill does not cross group boundaries. - **Data types:** all types are supported (boolean / text / blob / numeric), because `NEXT` simply copies a value — unlike `LINEAR`, which is numeric-only. ### Example Input: | time | temperature | |-----:|------------:| | 1 | null | | 2 | 10.0 | | 3 | null | | 4 | null | | 5 | 20.0 | | 6 | null | Result under each method: | time | PREVIOUS | **NEXT** | LINEAR | |-----:|---------:|---------:|-------:| | 1 | null | **10.0** | null | | 2 | 10.0 | **10.0** | 10.0 | | 3 | 10.0 | **20.0** | 13.33 | | 4 | 10.0 | **20.0** | 16.67 | | 5 | 20.0 | **20.0** | 20.0 | | 6 | 20.0 | **null** | null | ## Why build on top of `LINEAR` fill `PREVIOUS` fill only needs to remember the last seen value, so it streams block-by-block trivially. `NEXT` fill must look **ahead** to a value it hasn't read yet, possibly in a later `TsBlock`. That forward look-ahead / cross-block buffering is exactly what `LINEAR` fill already implements (it needs both the previous and the next non-null value to interpolate). So the recommended split is: - **Syntax + data-type coverage + node shape** → copy from `PREVIOUS` fill. - **Operator mechanics (look-ahead, cross-`TsBlock` buffering)** → copy from `LINEAR` fill, but instead of interpolating, just take the next non-null value (and apply `TIME_BOUND` if present). The reusable look-ahead lives in `iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/fill/linear/LinearFill.java` (`fill(Column timeColumn, Column valueColumn, long startRowIndex)` plus the `nextRowIndex` / `nextRowIndexInCurrentColumn` forward scan). ## Implementation guide (table model only) All paths are relative to the repo root and verified against current `master`. | Layer | File | Change | |-------|------|--------| | Grammar | `iotdb-core/relational-grammar/.../sql/RelationalSql.g4` | Add `#nextFill` alternative to `fillMethod` (`NEXT` token already exists) | | Fill method enum | `iotdb-core/node-commons/.../plan/statement/component/FillPolicy.java` | Add `NEXT((byte) 3)` | | AST | `iotdb-core/node-commons/.../relational/sql/ast/Fill.java` + the `AstBuilder` that builds it | Handle the `nextFill` rule (same fields as `previousFill`: `timeBound`, `timeColumnIndex`, `groupingElements`) | | Logical/physical node | `iotdb-core/node-commons/.../relational/planner/node/` (`FillNode.java`, `PreviousFillNode.java`) | Add `NextFillNode extends FillNode` mirroring `PreviousFillNode` (`timeBound`, `helperColumn`, `groupingKeys`); register its serialization type alongside the existing `TABLE_PREVIOUS_FILL_NODE` / `TABLE_LINEAR_FILL_NODE` in `PlanNodeType`, and add the `PlanVisitor.visitNextFill` hook | | Analyzer | `iotdb-core/datanode/.../relational/analyzer/StatementAnalyzer.java` (`analyzeFill`, `analyzeFillGroup`) | Add a `FillPolicy.NEXT` branch + a `NextFillAnalysis` (mirror `PreviousFillAnalysis`: `TIME_COLUMN` inferred like `PREVIOUS`/`LINEAR`, `TIME_BOUND` optional) | | Logical planner | `iotdb-core/datanode/.../relational/planner/QueryPlanner.java` (`visitFill`) | Build a `NextFillNode` from `NextFillAnalysis` | | Operator generator | `iotdb-core/calc-commons/.../plan/planner/TableOperatorGenerator.java` (`visitPreviousFill`, `visitLinearFill`, `visitValueFill`) | Add `visitNextFill`; wire to the new operator (use the linear-fill operators as the template) | | Operators | `iotdb-core/calc-commons/.../execution/operator/process/` (`AbstractLinearFillOperator.java`, `TableLinearFillOperator.java`, `TableLinearFillWithGroupOperator.java`) | Add `Table[…]NextFillOperator` + grouped variant, modeled on the linear-fill operators (they already do look-ahead / cross-block buffering) | | Per-datatype fill (codegen) | `iotdb-core/calc-commons/src/main/codegen/templates/` (`previousFill.ftl`, `previousFillWithTimeDuration.ftl`, `linearFill.ftl`) | Add `nextFill.ftl` (and `nextFillWithTimeDuration.ftl` for `TIME_BOUND`) generating `IntNextFill`, `LongNextFill`, `FloatNextFill`, `DoubleNextFill`, `BooleanNextFill`, `BinaryNextFill` — all types, like `previousFill`; register the new template in `codegen/config.fmpp` | ### Suggested step-by-step 1. Grammar: add `#nextFill`; regenerate parser; confirm `FILL METHOD NEXT ...` parses. 2. `FillPolicy.NEXT`; AST + `AstBuilder` handling. 3. `NextFillNode` (+ `PlanNodeType` registration, serde, `PlanVisitor` hook). 4. `StatementAnalyzer` `NEXT` branch + `NextFillAnalysis`. 5. `QueryPlanner.visitFill` → `NextFillNode`. 6. `nextFill.ftl` / `nextFillWithTimeDuration.ftl` per-datatype classes (mirror `LinearFill`'s look-ahead, copy the next value instead of interpolating). 7. `TableNextFillOperator` (+ grouped variant) modeled on the linear-fill operators. 8. `TableOperatorGenerator.visitNextFill` wiring. 9. Tests + docs. ## Tests - **Operator unit tests:** mirror the existing previous/linear fill operator tests (multi-`TsBlock` inputs, group boundaries, `TIME_BOUND`, the tail-stays-null case, all data types). - **Analyzer tests:** `NEXT` accepted; `TIME_COLUMN` inference / error path; `FILL_GROUP` validation. - **Integration tests:** extend the table-model fill IT (e.g. `IoTDBFillTableIT` / the relational fill IT) with `NEXT` cases including leading gap, trailing gap, grouping, and `TIME_BOUND`. ## Acceptance criteria - [ ] `FILL METHOD NEXT [TIME_BOUND ...] [TIME_COLUMN ...] [FILL_GROUP ...]` parses and executes in the table model. - [ ] Semantics match the example table above, including tail-null and `TIME_BOUND` behavior. - [ ] Works for all data types and across multiple `TsBlock`s. - [ ] `FILL_GROUP` does not leak values across groups. - [ ] Plan node serialization round-trips (distributed plan). - [ ] Unit + integration tests added; docs updated. ## Out of scope - Tree-model `NEXT` fill (the tree model also lacks it; can be a follow-up). - Changes to `LINEAR` / `PREVIOUS` / `CONSTANT` behavior. ## References - Grammar: `iotdb-core/relational-grammar/.../sql/RelationalSql.g4` (`fillMethod`) - Linear-fill look-ahead to reuse: `iotdb-core/calc-commons/.../execution/operator/process/fill/linear/LinearFill.java` - Previous-fill operator/templates to mirror for syntax & data-type coverage: `previousFill.ftl`, `previousFillWithTimeDuration.ftl`, `PreviousFillWithGroupOperator.java` ### Solution _No response_ ### Alternatives _No response_ ### Are you willing to submit a PR? - [ ] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
