zclllyybb commented on issue #65638: URL: https://github.com/apache/doris/issues/65638#issuecomment-4978260306
Breakwater-GitHub-Analysis-Slot: slot_45846e1babf3 This content is generated by AI for reference only. Initial triage: The attached profile supports the reporter's observation: this is dominated by FE planning, specifically the Nereids rewrite phase, not by BE execution. In the profile for `doris-4.1.1-rc01-b10073ad9ca`, total time is `8sec788ms`, with `Plan Time: 8sec390ms` and `Nereids Rewrite Time: 8sec266ms`. Other FE phases are much smaller (`Parse SQL Time: 23ms`, `Nereids Analysis Time: 71ms`, `Nereids Translate Time: 24ms`, `Schedule Time: 138ms`, `Wait and Fetch Result Time: 237ms`). The attached CSV is an explain output. The plan is very wide and complex: an outer `select *, count(*) over() ... order by ... limit 20` over a large aggregate result, many repeated `sum/if/cast` expressions, several joins and union/aggregate branches, 19 fragments, 937 instances, and the explain ends with `planned with unknown column statistics`. The reporter's note that planning worsens as selected fields increase is consistent with rewrite-time amplification over a very wide projection/aggregate plan. Code-path check on the 4.1.1 tag: `NereidsPlanner.rewrite()` runs `Rewriter.getWholeTreeRewriter(cascadesContext).execute()` and then records `Nereids Rewrite Time`. In this version, that whole-tree rewrite includes join rewrite, set-operation optimization, column pruning/predicate inference, project/join rewrites, aggregate pushdown/eager aggregation, and other heuristic rewrite jobs. Therefore the profile only proves that the RBO rewrite stage is expensive; it does not yet prove which specific rule is responsible. Current conclusion: This looks like a real 4.1.1 Nereids rewrite performance regression or at least a high-risk rewrite scalability case for wide aggregation/join/union plans. I would not recommend a blind configuration change yet. `nereids_timeout_second` would only avoid timeout, not reduce the 8s planning latency, and `enable_nereids_planner` is marked removed in 4.1.1. A targeted `disable_nereids_rules` workaround may be possible, but only after identifying the slow rule from trace evidence. Recommended next checks: 1. Ask the reporter for the complete original SQL text, the 2.1.8 profile/explain for the same SQL, and table DDL/statistics state (`SHOW CREATE TABLE`, relevant column stats or `ANALYZE` status). 2. Re-run on 4.1.1 with Nereids tracing enabled, for example `set enable_nereids_trace=true; set nereids_trace_event_mode='all';`, then attach the trace JSON from the FE `nereids_trace_log_dir` for the query id. This is needed to identify the exact rewrite rule or topic consuming the 8s. 3. As a mitigation validation, try reducing the projection width: avoid the outer `select *`, return only required columns, and test whether `count(*) over()` can be split into a separate count query. This should not be treated as the product fix, but it can confirm that wide projection/window rewrite is the main amplifier. 4. If trace shows a single dominant rewrite rule, test `disable_nereids_rules=<RULE_TYPE>` only for that rule as a temporary workaround and compare plan time plus result correctness. Maintainer-side follow-up should focus on reproducing this query shape on 4.1.1/upstream and profiling the Nereids whole-tree rewrite jobs over wide projections and aggregate/join/union branches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
