ColinLeeo opened a new pull request, #18889:
URL: https://github.com/apache/iotdb/pull/18889

   A CASE can leave shared transformer columns cached after one TsBlock or 
device batch is consumed. Reusing the DAG for the next input can then return 
stale values or use a column with the wrong length. This change resets the 
collected nodes before each input and rebuilds missing schema UPDATE common 
expressions only for devices that passed the filter.
   
   This is a narrow extraction from #18826. It keeps the existing selection 
evaluation and evaluation-time cache clearing mechanism.
   
   ### Changes
   
   - Collect each phase's DAG once, deduplicating nodes by identity; reset each 
node's own cache at the start of a new input.
   - Cover FilterAndProjectOperator, DevicePredicateHandler, and 
DeviceAttributeUpdater, preserving the original schema constructor descriptors.
   - Apply the final filter mask when an UPDATE common expression must be 
recomputed, including guarded CAST inputs such as 'n/a'. Query 
common-expression evaluation retains its existing behavior.
   - Read the selected MultiColumnTransformer input length from the retained 
column, because consuming the cache can release its stored length.
   - Add cross-block/batch, original aggregation, effective upper(src) 
common-expression, and mixed-device guarded UPDATE tests. The pending-sibling 
IT is a guard against changing existing within-block behavior, not the original 
bug reproduction.
   
   ### Draft: unresolved merge blocker
   
   Keeping the old within-block cache mechanism leaves the [shared DIFF problem 
identified in the review of 
#18826](https://github.com/apache/iotdb/pull/18826#discussion_r4258223689) 
unresolved. The boundary reset changes the observed failure mode: an isolated 
operator probe with inputs [1, 3, 6] and [10, 11], a CASE filter on diff(x) > 
1, and diff(x) as the projected common expression throws 
ArrayIndexOutOfBoundsException on the second block on master, but this branch 
returns -1 where the correct result is 4. Both probe runs are explicitly 
recorded as failures.
   
   This draft must not be merged with that silent wrong-value behavior. 
Selection-result isolation, per-input DIFF result retention, phase-end cache 
release, and the broader UDF changes from #18826 are intentionally deferred 
rather than reintroduced into this extraction.
   
   ### Validation
   
   - 18 calc unit tests and 10 DataNode unit tests passed, including the four 
existing BlobLength Mockito tests.
   - Six table-model integration tests passed: the original three aggregation 
query variants, the pending-sibling guard, repeated device batches/projections, 
upper(src) common materialization, and mixed 'n/a'/'50' guarded UPDATE.
   - Separate baseline/new runs of six schema and six operator probes: eleven 
failures on master 673434b333, all twelve passing on this branch. The 
upper(src) test reproduces the null common Column on master.
   - Full reactor test compilation passed in both English and Chinese (53 
modules each).
   - Spotless and reactor license checks passed.
   
   ### Performance
   
   The new-input path clears a precomputed flat array in O(V) per evaluation 
phase, without a per-block graph walk or visited-set allocation. Schema UPDATE 
allocates a boolean selection mask and scans its rows only when common 
expressions exist. The original CASE/common recomputation cost remains.
   
   Local exploratory JMH comparison against master 673434b333 (JDK 17, 512 
MiB/G1, 3 x 1 s warmup, 4 x 1 s measurement, two runs per version in 
baseline/new/new/baseline order):
   
   | 1000 rows per block | master | this branch |
   | --- | ---: | ---: |
   | Addition projection | 3.536 us | 3.503 us |
   | 16 projections sharing an expression | 40.932 us | 41.549 us |
   | CASE with a common expression | 22.749 us | 26.916 us |
   
   All ten benchmark shapes (five scenarios x 1/1000 rows) validated every 
result across 16 blocks before measurement. Timings were noisy on the local 
machine, especially the wide filter scenario; these numbers are not evidence of 
a general speedup or absence of regressions.
   
   The CASE/common allocation increase was stable: +192 bytes per one-row block 
and +9176 bytes per 1000-row block. A deterministic counting probe explains the 
extra work: shared x + 1 was computed 17 times across 16 blocks on master and 
32 times here. Master partly benefits from a leftover cache whose predicate 
happens to remain true for these benchmark inputs; the reset removes that 
stale-cache reuse, while CASE still clears the result before common 
materialization. Expensive expressions can therefore amplify this cost.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to