JackieTien97 commented on PR #18826: URL: https://github.com/apache/iotdb/pull/18826#issuecomment-6139897424
There is a performance concern with the unconditional per-block cache traversal in `FilterAndProjectOperator` ([filter phase](https://github.com/apache/iotdb/blob/50c38c16ddf29032a6d1cf1c89b2bcf67627e957/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperator.java#L158-L165), [projection phase](https://github.com/apache/iotdb/blob/50c38c16ddf29032a6d1cf1c89b2bcf67627e957/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperator.java#L277-L281)). Each phase allocates a new identity-based visited set and traverses its transformer DAG. This also adds work to operators with no CASE expressions; an operator performing both filtering and projection does this twice per input block. I compared the actual operator and changed transformer classes from the merge base `fbd5fd22a6c9009dccd7f1a5b84bf7d61e5718e0` against PR head `50c38c16ddf29032a6d1cf1c89b2bcf67627e957`, using the same dependencies. The following scenarios contain **no CASE expressions**. Results below are means across two JVM forks, with **one row per input TsBlock**: | Operator scenario | Before (µs/block) | PR (µs/block) | Time increase | Additional allocation/block | | --- | ---: | ---: | ---: | ---: | | One addition projection, `x + 1` | 0.084 | 0.128 | ~52% | 400 B | | 16 projections sharing `s = x + 1`, with outputs `s + 2` through `s + 17` | 0.762 | 1.457 | ~91% | 1,440 B | | Filter `(x0 + 1) > 0`, then pass through 16 columns | 1.001 | 1.272 | ~27% | ~772 B | All three scenarios were slower in both forks. Method: JMH 1.37, Zulu JDK 17.0.4.1, one benchmark thread, average-time mode, 3 × 1 s warmup and 4 × 1 s measurement per fork, two forks per revision/scenario with revision order reversed in the second round, a 512 MiB G1 heap, and the GC allocation profiler. The transformer graphs were constructed directly and consumed eight rotating, prebuilt input blocks; setup checked every output against expected values before timing, and each measured invocation returned its result to JMH. **These are operator microbenchmark results, not end-to-end SQL regression percentages.** They do not include SQL planning, storage scanning, aggregation, or network costs. The 1024-row variants and the separate 16-independent-additions scenario had substantial variation between forks; a longer-warmup check also showed variation, so I am not drawing a stable, general query-latency regression percentage from those results. Suggested optimization: once the transformer DAG is fully constructed, **collect and identity-deduplicate the nodes once, and retain a flat list for each evaluation phase**. The filter-phase list should include the filter and common-expression roots; the projection-phase list should include all projection roots. Both should include CASE descendants. At each new-input boundary, iterate the corresponding list and clear **only each node's own cache** before initializing the leaves. Keep the existing evaluation-time CASE cleanup semantics unchanged; calling the recursive legacy `clearCache()` on every list entry would defeat the purpose and could change the CASE boundary behavior. This should preserve the cross-block correctness fix while removing repeated DAG traversal, identity-set allocation/resizing, and hash lookups from the per-block path. The same approach can be applied to the schema-batch handlers. I recommend addressing this overhead, particularly for workloads producing many small TsBlocks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
