yujun777 opened a new pull request, #67853: URL: https://github.com/apache/doris/pull/67853
### What problem does this PR solve? An MTMV re-validates its schema against a fresh analysis of its own query whenever a base table changes (`MTMVPlanUtil.ensureMTMVQueryUsable` -> `checkColumnIfChange`). A table created with row binlog carries hidden columns that the query itself never produces (`__DORIS_COMMIT_TSO_COL__`, plus `__DORIS_ROW_LSN_COL__` for duplicate keys), and an MV ends up with them in its physical schema when its own properties enable row binlog. The analyzed column list did not contain them, so for those MVs the comparison was permanently short by a column and every refresh failed with `column length not equals, please check whether columns of base table have changed` -- including `COMPLETE`, which is the only way out of a stale baseline. The MV then became unrecoverable without dropping and recreating it. Cascade IVM always hits this, because a cascade source has to keep row binlog for its downstream level. Reported shape and trace issue: https://github.com/apache/doris/issues/65418 The root cause is that the hidden-column rules were split across two classes: `CreateTableInfo` owns the OLAP ones (delete sign / row store / version / skip bitmap), while the row-binlog ones are added by `InternalCatalog#createOlapTable` right before the table is built. The analyzed schema path only mirrored the first half. ### What changed - Extracted the row-binlog hidden-column rule into `CreateTableInfo.addRowBinlogHiddenColumns`, and let `createRowBinlogHiddenColumnsIfNecessary` delegate to it. The helper is idempotent. - Applied the same rule when the analyzed MTMV column list is built, so that list matches the physical schema of the table it produces. The binlog config is derived from the properties passed in: for a refresh those are the MV table's own properties, where `InternalCatalog` already stores the resolved (database-merged) binlog config. - The physical schema is unchanged: both paths append the same columns at the same position, so a table built from the analyzed list still ends up with exactly one of each. `checkColumnIfChange` compares types index by index, so the fix also pins the column order, not just the count. ### Test - `MTMVPlanUtilTest#testEnsureMTMVQueryUsableWithRowBinlogHiddenColumns` (new): IVM and duplicate-key MVs, with and without row binlog, assert the physical schema holds the hidden columns exactly once and that `ensureMTMVQueryUsable` accepts all four. - `MTMVPlanUtilTest`, `OlapTableRowBinlogSchemaTest`, `CreateTableTest`: 45 cases. - `test_ivm_row_binlog_schema_validation` (new suite): the four MV shapes above have to stay refreshable after an unrelated `DROP COLUMN`, and a cascade source has to recover through `COMPLETE` after a schema ABA on a referenced column. - Neighbouring regression suites: `test_ivm_mtmv_row_binlog`, `test_ivm_chained_mtmv_1`, `test_ivm_chained_mtmv_2`, `test_ivm_hidden_columns`, `test_ivm_dup_keys_lsn_rowid`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
