Doris-Breakwater commented on issue #70529: URL: https://github.com/apache/doris/issues/70529#issuecomment-6826626103
Breakwater-GitHub-Analysis-Slot: slot_d0d1bd8b21f6 **Initial assessment:** route to FE/Nereids planner maintainers, with ES catalog involvement. The reported plan violates the parent/child ExprId invariant: the project requires `yjsj#18`, while the scan supplies `yjsj#27`. The column exists; its identity differs. This is a planning validation failure, before query data execution. The original report already shows a successful direct query on **the same `yjsj` column**; the earlier request to test it instead of `fssj` is unnecessary. [Original report](https://github.com/apache/doris-website/issues/3828) **Code evidence:** I inspected upstream tag `4.1.4`, commit `ad35a140c7fd0b842f18c23300bac581f7d04326`, as a reference implementation—not as an identification of the reporter's version. There is a concrete slot-lifetime defect in that implementation: 1. A single-reference ordinary CTE is inlined under the reference defaults (`enable_cte_materialize=true`, `inline_cte_referenced_threshold=1`). `CTEInline.visitLogicalCTEConsumer()` deep-copies the producer subtree. [CTEInline](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/CTEInline.java#L93) 2. ES scans use the generic catalog-relation copier. It creates scan A with `withRelationId()`, reads A's output, and records the old-to-new ExprId mapping. It then calls `withVirtualColumns()`, **even when the virtual-column list is empty**. [LogicalPlanDeepCopier](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/copier/LogicalPlanDeepCopier.java#L107) 3. ES's `withVirtualColumns()` creates scan B with empty logical properties, discarding A's cached output identities. `LogicalCatalogRelation.computeOutput()` allocates new ExprIds for the base columns. The copied project still uses the mapping to A, but its child is B. ES's inherited `withOperativeSlots()` returns the scan unchanged and does not repair this mismatch. [LogicalEsScan](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalEsScan.java#L75), [output generation](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java#L135) 4. `CheckAfterRewrite` validates input ExprIds against child output ExprIds and emits the reported error. [Validation](https://github.com/apache/doris/blob/ad35a140c7fd0b842f18c23300bac581f7d04326/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAfterRewrite.java#L101) This is a strong, source-supported explanation for the symptom. **Its presence in the reporter's actual build remains unverified; no live reproduction or fix validation was performed.** **Related-fix boundary:** #63315 / its branch-4.0 backport #63708 correct the loop bound used to locate operative slots. They do not change the ES reconstruction described above. The inspected `4.1.4` reference already contains the corresponding branch-4.1 backport #63709, yet retains this reconstruction path. A last-position column alone therefore does not establish that those PRs fix this issue. Do not recommend cherry-picking them or promise a fixed release without checking the exact build and reproducing. [Backport diff](https://github.com/apache/doris/commit/f06836483e351f9b488a47c72daaa4edb49d7d9b) **Information needed to confirm and reproduce:** - Exact Doris version and FE build Git hash, including the FE handling the failed query; `SELECT VERSION()` plus build/startup version information. - Query ID, timestamp, full error/plan text, and the corresponding **server-side FE exception stack**. The original stack is predominantly JDBC/client frames. Attach `EXPLAIN VERBOSE` for both queries where possible; if explaining the CTE also fails, retain that error. A runtime Profile is not required for this planning failure. - The two CTE session-variable values; sanitized catalog DDL, `DESC` output preserving column order/types, ES version, and index mapping/alias structure. Remove credentials and private endpoints. **Next steps:** - Use the already successful direct query as the temporary workaround for this simple CTE. Unordered `LIMIT` does not guarantee the same ten rows. - If retaining the CTE is necessary, a **candidate diagnostic workaround**, supported by the reference `CTEInline` condition, is to test in a separate session with `enable_cte_materialize=true` and `inline_cte_referenced_threshold=0`. Capture the plan and result, then close that session. Confirm a producer/consumer plan; this is not yet validated on the reported build and materialization can add work. Setting materialization to false forces inlining in this path. - Once the build is known, reproduce against its source and check whether scan reconstruction invalidates the recorded mapping. A fix should preserve/remap base-column identities consistently with the final returned scan, including virtual columns. Add coverage for empty virtual columns, non-first projected columns, and repeated CTE references. Keep slot validation enabled; verify both original SQL statements before marking this resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
