ColinLeeo opened a new pull request, #18889: URL: https://github.com/apache/iotdb/pull/18889
A CASE can leave shared transformer columns cached after one TsBlock or device batch is consumed. Reusing the DAG for the next input can then return stale values or use a column with the wrong length. This change resets the collected nodes before each input and rebuilds missing schema UPDATE common expressions only for devices that passed the filter. This is a narrow extraction from #18826. It keeps the existing selection evaluation and evaluation-time cache clearing mechanism. ### Changes - Collect each phase's DAG once, deduplicating nodes by identity; reset each node's own cache at the start of a new input. - Cover FilterAndProjectOperator, DevicePredicateHandler, and DeviceAttributeUpdater, preserving the original schema constructor descriptors. - Apply the final filter mask when an UPDATE common expression must be recomputed, including guarded CAST inputs such as 'n/a'. Query common-expression evaluation retains its existing behavior. - Read the selected MultiColumnTransformer input length from the retained column, because consuming the cache can release its stored length. - Add cross-block/batch, original aggregation, effective upper(src) common-expression, and mixed-device guarded UPDATE tests. The pending-sibling IT is a guard against changing existing within-block behavior, not the original bug reproduction. ### Draft: unresolved merge blocker Keeping the old within-block cache mechanism leaves the [shared DIFF problem identified in the review of #18826](https://github.com/apache/iotdb/pull/18826#discussion_r4258223689) unresolved. The boundary reset changes the observed failure mode: an isolated operator probe with inputs [1, 3, 6] and [10, 11], a CASE filter on diff(x) > 1, and diff(x) as the projected common expression throws ArrayIndexOutOfBoundsException on the second block on master, but this branch returns -1 where the correct result is 4. Both probe runs are explicitly recorded as failures. This draft must not be merged with that silent wrong-value behavior. Selection-result isolation, per-input DIFF result retention, phase-end cache release, and the broader UDF changes from #18826 are intentionally deferred rather than reintroduced into this extraction. ### Validation - 18 calc unit tests and 10 DataNode unit tests passed, including the four existing BlobLength Mockito tests. - Six table-model integration tests passed: the original three aggregation query variants, the pending-sibling guard, repeated device batches/projections, upper(src) common materialization, and mixed 'n/a'/'50' guarded UPDATE. - Separate baseline/new runs of six schema and six operator probes: eleven failures on master 673434b333, all twelve passing on this branch. The upper(src) test reproduces the null common Column on master. - Full reactor test compilation passed in both English and Chinese (53 modules each). - Spotless and reactor license checks passed. ### Performance The new-input path clears a precomputed flat array in O(V) per evaluation phase, without a per-block graph walk or visited-set allocation. Schema UPDATE allocates a boolean selection mask and scans its rows only when common expressions exist. The original CASE/common recomputation cost remains. Local exploratory JMH comparison against master 673434b333 (JDK 17, 512 MiB/G1, 3 x 1 s warmup, 4 x 1 s measurement, two runs per version in baseline/new/new/baseline order): | 1000 rows per block | master | this branch | | --- | ---: | ---: | | Addition projection | 3.536 us | 3.503 us | | 16 projections sharing an expression | 40.932 us | 41.549 us | | CASE with a common expression | 22.749 us | 26.916 us | All ten benchmark shapes (five scenarios x 1/1000 rows) validated every result across 16 blocks before measurement. Timings were noisy on the local machine, especially the wide filter scenario; these numbers are not evidence of a general speedup or absence of regressions. The CASE/common allocation increase was stable: +192 bytes per one-row block and +9176 bytes per 1000-row block. A deterministic counting probe explains the extra work: shared x + 1 was computed 17 times across 16 blocks on master and 32 times here. Master partly benefits from a leftover cache whose predicate happens to remain true for these benchmark inputs; the reset removes that stale-cache reuse, while CASE still clears the result before common materialization. Expensive expressions can therefore amplify this cost. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
