JackieTien97 commented on PR #18826:
URL: https://github.com/apache/iotdb/pull/18826#issuecomment-6139897424

   There is a performance concern with the unconditional per-block cache 
traversal in `FilterAndProjectOperator` ([filter 
phase](https://github.com/apache/iotdb/blob/50c38c16ddf29032a6d1cf1c89b2bcf67627e957/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperator.java#L158-L165),
 [projection 
phase](https://github.com/apache/iotdb/blob/50c38c16ddf29032a6d1cf1c89b2bcf67627e957/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperator.java#L277-L281)).
 Each phase allocates a new identity-based visited set and traverses its 
transformer DAG. This also adds work to operators with no CASE expressions; an 
operator performing both filtering and projection does this twice per input 
block.
   
   I compared the actual operator and changed transformer classes from the 
merge base `fbd5fd22a6c9009dccd7f1a5b84bf7d61e5718e0` against PR head 
`50c38c16ddf29032a6d1cf1c89b2bcf67627e957`, using the same dependencies. The 
following scenarios contain **no CASE expressions**. Results below are means 
across two JVM forks, with **one row per input TsBlock**:
   
   | Operator scenario | Before (µs/block) | PR (µs/block) | Time increase | 
Additional allocation/block |
   | --- | ---: | ---: | ---: | ---: |
   | One addition projection, `x + 1` | 0.084 | 0.128 | ~52% | 400 B |
   | 16 projections sharing `s = x + 1`, with outputs `s + 2` through `s + 17` 
| 0.762 | 1.457 | ~91% | 1,440 B |
   | Filter `(x0 + 1) > 0`, then pass through 16 columns | 1.001 | 1.272 | ~27% 
| ~772 B |
   
   All three scenarios were slower in both forks. Method: JMH 1.37, Zulu JDK 
17.0.4.1, one benchmark thread, average-time mode, 3 × 1 s warmup and 4 × 1 s 
measurement per fork, two forks per revision/scenario with revision order 
reversed in the second round, a 512 MiB G1 heap, and the GC allocation 
profiler. The transformer graphs were constructed directly and consumed eight 
rotating, prebuilt input blocks; setup checked every output against expected 
values before timing, and each measured invocation returned its result to JMH.
   
   **These are operator microbenchmark results, not end-to-end SQL regression 
percentages.** They do not include SQL planning, storage scanning, aggregation, 
or network costs. The 1024-row variants and the separate 
16-independent-additions scenario had substantial variation between forks; a 
longer-warmup check also showed variation, so I am not drawing a stable, 
general query-latency regression percentage from those results.
   
   Suggested optimization: once the transformer DAG is fully constructed, 
**collect and identity-deduplicate the nodes once, and retain a flat list for 
each evaluation phase**. The filter-phase list should include the filter and 
common-expression roots; the projection-phase list should include all 
projection roots. Both should include CASE descendants. At each new-input 
boundary, iterate the corresponding list and clear **only each node's own 
cache** before initializing the leaves. Keep the existing evaluation-time CASE 
cleanup semantics unchanged; calling the recursive legacy `clearCache()` on 
every list entry would defeat the purpose and could change the CASE boundary 
behavior.
   
   This should preserve the cross-block correctness fix while removing repeated 
DAG traversal, identity-set allocation/resizing, and hash lookups from the 
per-block path. The same approach can be applied to the schema-batch handlers. 
I recommend addressing this overhead, particularly for workloads producing many 
small TsBlocks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to