Doris-Breakwater commented on issue #67133: URL: https://github.com/apache/doris/issues/67133#issuecomment-5411749845
Breakwater-GitHub-Analysis-Slot: slot_71200f479ac3 ## Initial triage **Verdict: confirmed source-level lifecycle regression, high confidence.** This blocks Cloud IVM refresh planning after #66739; it is not a BE or MetaService data-read failure. The issue is currently open with no labels, assignee, milestone, or linked development. `kind/bug` plus Cloud/Table Stream/IVM component labels would be appropriate. ### Verified facts - #66739 removed the generic `ResolveCloudTableStreamReadState` rewrite and moved the same collect → batch RPC → install orchestration into `CloudTableStreamReadStateHook.afterAnalyze()`. `NereidsPlanner` runs only hooks present in `StatementContext` after the analyzer completes. - `BindRelation.makeTableStreamScan()` is the only production registration site for that hook. An IVM definition binds ordinary `LogicalOlapScan` nodes, so this registration does not happen. - On the current IVM draft head ([#62606](https://github.com/apache/doris/pull/62606), `5ad279ae66a8cbaffa98c9b23a6d12a87925e043`), both late paths are present in the analyzer after relation binding: - `IvmFullRefreshMTMV` constructs a new `OlapTableStreamWrapper` and `LogicalOlapTableStreamScan`, using `RESET` or `SNAPSHOT` mode. - `IvmIncrRefreshMTMV` calls `IvmDeltaRewriteState.createDeltaScan()`, which constructs a new wrapper and incremental stream scan. Neither path installs cloud state or registers the hook. The reported COMPLETE path creates a `RESET` scan; `RESET` is non-snapshot, so `StreamConsumptionInfoExtractor` reaches the new precondition and fails exactly as reported. The incremental path has the same missing-state condition by inspection, although the issue only demonstrates COMPLETE at runtime. - The invariant should be retained. `installCloudReadStates()` also initializes the wrapper's output-update map; normalization uses the states for partition classification and cloud visible versions, and extraction uses them to build the expected/next stream-offset update. Silently skipping a missing state can omit atomic offset advancement and cause stale or replayed consumption. Relaxing only the extractor precondition would therefore mask an earlier lifecycle error rather than fix it. ### Recommended fix Prefer a plan-content-driven installation point after all analyzer rules have synthesized stream scans and before normal rewrite/normalization starts. Extract the hook's `resolve(Plan)` body into a public, generator-neutral installer (or make an equivalent public entry point), keep one batched `get_table_stream_offset` RPC for the whole statement, and have the existing hook delegate to it. The installer should preserve the current all-or-none installation, partition coverage, response validation, and idempotence checks. The smallest scoped repair is for `IvmFullRefreshMTMV` and `IvmIncrRefreshMTMV` to register `CloudTableStreamReadStateHook.INSTANCE` only when their rewritten plan actually contains a stream scan; `StatementContext` stores hooks in a set, so duplicate registration is harmless. A central post-analysis plan check is more maintainable because future scan generators cannot forget registration. In either design, do not invoke the hook on a no-delta/empty plan unless the installer treats “no stream scans” as a no-op. ### Tests needed 1. Cloud FE integration tests for late-synthesized COMPLETE/RESET and INCREMENTAL scans: mock one MetaService response, assert one batched RPC, assert every wrapper is installed before normalization, and assert `StreamConsumptionInfoExtractor` produces the expected update. 2. Cover multiple base tables/wrappers, empty selected partitions/no pending delta, snapshot plus delta scans, and an incomplete MetaService response. 3. Add a Cloud end-to-end regression based on this SQL: COMPLETE bootstrap, another INSERT, then INCREMENTAL refresh; verify MV rows and stream offsets after each successful transaction. Keep the shared-nothing case to guard unchanged behavior. EXPLAIN and incremental dry-run should also be checked because they synthesize the same plans even when they do not commit offsets. ### Missing/ambiguous information - Please provide the exact FE build commit/branch. As of this analysis, public `apache/master` at `c899ceada941807a5114e48a23a2c0a74bd664bd` contains #66739 but does not contain the IVM classes; those classes are in draft PR #62606. Thus “master branch” is not sufficient to identify the reproduced build. - Please attach the complete FE task stack trace/query ID and run the equivalent `REFRESH MATERIALIZED VIEW mv INCREMENTAL` case. These are not required to establish the COMPLETE source-level cause, but they would confirm that no second failure is hidden behind the precondition and validate the incremental runtime impact. No code or runtime profile is needed to diagnose the current assertion itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
