yujun777 opened a new pull request, #70785:
URL: https://github.com/apache/doris/pull/70785

   ### Background
   
   Incremental view maintenance stores a signature of the normalized query plan 
to check that a refresh can find and update the rows written by earlier 
refreshes. Creating a view normalizes a query with a result sink. Refreshing 
the view normalizes an INSERT with a table sink.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #62606 (introduced IVM layout signatures), #69890 (publishes the 
signature after a complete refresh)
   PR Type: Bugfix
   
   Problem Summary:
   
   A deferred aggregate IVM view can fail signature validation before its first 
complete refresh. One example groups a UNION ALL over a VARCHAR column and a 
shorter string literal. The union yields STRING, and the table sink truncates 
it when writing to the bounded VARCHAR column of the view. This PR makes a 
write-only truncation project transparent to the signature.
   
   ### Details
   
   - **Frequency:** Always when the table sink adds a write-only string 
truncation project to a query that otherwise has a matching IVM layout.
   - **Root cause:** IvmPlanSignatureGenerator treated the sink's substring 
projection as a change to the maintenance plan, although it only adapted a 
value after the row identifier and aggregate state had been computed.
   - **Why existing tests missed it:** IvmPlanSignatureGeneratorTest covered 
passthrough outputs and sink bookkeeping columns. The signature fallback 
regression tested an injected signature mismatch after a complete refresh. 
Neither covered write truncation before the first complete refresh.
   - **Fix:** Walk the contiguous projects under the OLAP table sink and check 
that each projected value is a passthrough or a truncation matching its target 
column. Keep the sink's existing hidden defaults transparent. Preserve the 
signature for truncation in the query and for changes to persisted maintenance 
columns.
   - **Other paths and branches:** The unique-key table sink uses the same 
canonicalization for full and incremental refreshes. Append-only views do not 
generate layout signatures. The signature generator is absent in the local 
upstream 3.1, 4.0, 4.1, and 4.2 branch refs.
   - **Behavior change:** A write-only string truncation no longer changes the 
IVM layout signature for a newly created view.
   
   ### Testing
   
   The original aggregate view definition fails EXPLAIN ANALYZED PLAN REFRESH 
MATERIALIZED VIEW INCREMENTAL WITH ALL STREAMS on the baseline before any 
refresh task. With the fix, the same EXPLAIN succeeds before any refresh task, 
and its write plan still contains the truncation. A subsequent complete refresh 
and strict incremental refresh both succeed. The four result groups match the 
original base-table query. The FE build and all 26 tests in 
IvmPlanSignatureGeneratorTest passed locally.
   
   ### Release note
   
   Deferred incremental materialized views no longer report a layout mismatch 
solely because the table sink truncates a visible string column at write time.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: No
   - Pre-submission status: Ready
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to